dogstatsd-ruby 5.0.1 → 5.3.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c77f82b5b9a858517a937a5b2db1a1f890a80c94ca63e60f12e256ab28f7192d
4
- data.tar.gz: e95ee401174c084edb117068f73d1c72f08e602ea9b51363e57e53c176072472
3
+ metadata.gz: 539ec3d4e02bd8784df4d032b0f8e96ebb2a582b510ff5e27c6e7cfff4515926
4
+ data.tar.gz: d8ecf4c6724e491aaea6d738cc255891b55119987b1ca5665087d506552135ee
5
5
  SHA512:
6
- metadata.gz: 88d54866c8693d2dab18a1370d6ba5951a42ec4ed62615c6194548dd5faa109731840187e2a7ddaf591f08d07064c8e173df74728f182fe5cb5593c896503e19
7
- data.tar.gz: d8021d0b6b21efe7ab14841665b01819c397d4419b247615d88b7685b332b1c8bb9a13d05fc22d21baf5d709d5452f9a9a3d932438addfbf1238e8ab5656a8e5
6
+ metadata.gz: d615efd205f7858c55b9864e1d001d607dcb39a47a26c87f5e27402b25834cf750fb3904d10d667c63a437587d38ba315d7a3c609b7709bc28a8bc8138d30eef
7
+ data.tar.gz: 254bf33695141f521d55d5f5b9c4d6b3abf2408b6b231263949094cbc2c9f531bf3ff2aea92ae0f11d505bbb8d25a789a2c0da76d9a065b713f9dc9b5b9d836d
data/README.md CHANGED
@@ -22,22 +22,83 @@ To instantiate a DogStatsd client:
22
22
  # Import the library
23
23
  require 'datadog/statsd'
24
24
 
25
- # Create a DogStatsD client instance.
25
+ # Create a DogStatsD client instance
26
26
  statsd = Datadog::Statsd.new('localhost', 8125)
27
+ # ...
28
+ # release resources used by the client instance
29
+ statsd.close()
27
30
  ```
28
31
  Or if you want to connect over Unix Domain Socket:
29
32
  ```ruby
30
33
  # Connection over Unix Domain Socket
31
34
  statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
35
+ # ...
36
+ # release resources used by the client instance
37
+ statsd.close()
32
38
  ```
33
39
 
34
- Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby#client-instantiation-parameters).
40
+ Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?code-lang=ruby#client-instantiation-parameters).
41
+
42
+ ### Migrating from v4.x to v5.x
43
+
44
+ If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
45
+ change concerning you is the new [threading model](#threading-model):
46
+
47
+ In practice, it means two things:
48
+
49
+ 1. Now that the client is buffering metrics before sending them, you have to call `Datadog::Statsd#flush(sync: true)` if you want synchronous behavior. In most cases, this is not needed, as the sender thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
50
+
51
+ 2. You have to make sure you are either:
52
+
53
+ * Using a singleton instance of the DogStatsD client instead of creating a new instance whenever you need one; this will let the buffering mechanism flush metrics regularly
54
+ * Or properly disposing of the DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close`
55
+
56
+ If you have issues with the sender thread or the buffering mode, you can instantiate a client that behaves exactly as in v4.x (i.e. no sender thread and flush on every metric submission):
57
+
58
+ ```ruby
59
+ # Create a DogStatsD client instance using UDP
60
+ statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true, buffer_max_pool_size: 1)
61
+ # ...
62
+ statsd.close()
63
+ ```
64
+
65
+ or
66
+
67
+ ```ruby
68
+ # Create a DogStatsD client instance using UDS
69
+ statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file', single_thread: true, buffer_max_pool_size: 1)
70
+ # ...
71
+ statsd.close()
72
+ ```
73
+
74
+ ### v5.x Common Pitfalls
75
+
76
+ Version v5.x of `dogstatsd-ruby` is using a sender thread for flushing. This provides better performance, but you need to consider the following pitfalls:
77
+
78
+ 1. Applications that use `fork` after having created the dogstatsd instance: the child process will automatically spawn a new sender thread to flush metrics.
79
+
80
+ 2. Applications that create multiple instances of the client without closing them: it is important to `#close` all instances to free the thread and the socket they are using otherwise you will leak those resources.
81
+
82
+ If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
83
+
84
+ If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
85
+
86
+ Applications that run into issues but can't apply these recommendations should use the `single_thread` mode which disables the use of the sender thread.
87
+ Here is how to instantiate a client in this mode:
88
+
89
+ ```ruby
90
+ statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true)
91
+ # ...
92
+ # release resources used by the client instance and flush last metrics
93
+ statsd.close()
94
+ ```
35
95
 
36
96
  ### Origin detection over UDP
37
97
 
38
- Origin detection is a method to detect which pod DogStatsD packets are coming from in order to add the pod's tags to the tag list.
98
+ Origin detection is a method to detect which pod DogStatsD packets are coming from, in order to add the pod's tags to the tag list.
99
+
100
+ To enable origin detection over UDP, add the following lines to your application manifest:
39
101
 
40
- To enable origin detection over UDP, add the following lines to your application manifest
41
102
  ```yaml
42
103
  env:
43
104
  - name: DD_ENTITY_ID
@@ -45,49 +106,109 @@ env:
45
106
  fieldRef:
46
107
  fieldPath: metadata.uid
47
108
  ```
109
+
48
110
  The DogStatsD client attaches an internal tag, `entity_id`. The value of this tag is the content of the `DD_ENTITY_ID` environment variable, which is the pod’s UID.
49
111
 
50
112
  ## Usage
51
113
 
52
- In order to use DogStatsD metrics, events, and Service Checks the Agent must be [running and available](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby).
114
+ In order to use DogStatsD metrics, events, and Service Checks the Datadog Agent must be [running and available](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby).
53
115
 
54
116
  ### Metrics
55
117
 
56
- After the client is created, you can start sending custom metrics to Datadog. See the dedicated [Metric Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby) to see how to submit all supported metric types to Datadog with working code examples:
118
+ After the client is created, you can start sending custom metrics to Datadog. See the dedicated [Metric Submission: DogStatsD documentation](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby) to see how to submit all supported metric types to Datadog with working code examples:
57
119
 
58
- * [Submit a COUNT metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#count).
59
- * [Submit a GAUGE metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#gauge).
60
- * [Submit a SET metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#set)
61
- * [Submit a HISTOGRAM metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#histogram)
62
- * [Submit a DISTRIBUTION metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#distribution)
120
+ * [Submit a COUNT metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#count).
121
+ * [Submit a GAUGE metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#gauge).
122
+ * [Submit a SET metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#set)
123
+ * [Submit a HISTOGRAM metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#histogram)
124
+ * [Submit a DISTRIBUTION metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#distribution)
63
125
 
64
- Some options are suppported when submitting metrics, like [applying a Sample Rate to your metrics](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-submission-options) or [tagging your metrics with your custom tags](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-tagging). Find all the available functions to report metrics in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
126
+ Some options are suppported when submitting metrics, like [applying a Sample Rate to your metrics](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-submission-options) or [tagging your metrics with your custom tags](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-tagging). Find all the available functions to report metrics in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
65
127
 
66
128
  ### Events
67
129
 
68
- After the client is created, you can start sending events to your Datadog Event Stream. See the dedicated [Event Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/events/dogstatsd/?tab=ruby) to see how to submit an event to Datadog your Event Stream.
130
+ After the client is created, you can start sending events to your Datadog Event Stream. See the dedicated [Event Submission: DogStatsD documentation](https://docs.datadoghq.com/events/guides/dogstatsd/?code-lang=ruby) to see how to submit an event to Datadog your Event Stream.
69
131
 
70
132
  ### Service Checks
71
133
 
72
134
  After the client is created, you can start sending Service Checks to Datadog. See the dedicated [Service Check Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/?tab=ruby) to see how to submit a Service Check to Datadog.
73
135
 
74
- ### Maximum packets size in high-throughput scenarios
136
+ ### Maximum packet size in high-throughput scenarios
75
137
 
76
138
  In order to have the most efficient use of this library in high-throughput scenarios,
77
- default values for the maximum packets size have already been set for both UDS (8192 bytes)
78
- and UDP (1432 bytes) in order to have the best usage of the underlying network.
79
- However, if you perfectly know your network and you know that a different value for the maximum packets
80
- size should be used, you can set it with the parameter `buffer_max_payload_size`. Example:
139
+ recommended values for the maximum packet size have already been set for both UDS (8192 bytes)
140
+ and UDP (1432 bytes).
141
+
142
+ However, if are in control of your network and want to use a different value for the maximum packet
143
+ size, you can do it by setting the `buffer_max_payload_size` parameter:
81
144
 
82
145
  ```ruby
83
- # Create a DogStatsD client instance.
84
146
  statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
147
+ # ...
148
+ statsd.close()
85
149
  ```
86
150
 
151
+ ## Threading model
152
+
153
+ Starting with version 5.0, `dogstatsd-ruby` employs a new threading model where one instance of `Datadog::Statsd` can be shared between threads and where data sending is non-blocking (asynchronous).
154
+
155
+ When you instantiate a `Datadog::Statsd`, a sender thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class. You can make use of `single_thread: true` to disable this behavior.
156
+
157
+ This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). Instantiating a lot of statsd clients without calling `#close` after they are not needed anymore will most likely lead to threads being leaked.
158
+
159
+ The sender thread has the following logic (from `Datadog::Statsd::Sender#send_loop`):
160
+
161
+ ```
162
+ while the sender message queue is not closed do
163
+ read message from sender message queue
164
+
165
+ if message is a Control message to flush
166
+ flush buffer in connection
167
+ else if message is a Control message to synchronize
168
+ synchronize with calling thread
169
+ else
170
+ add message to the buffer
171
+ end
172
+ end while
173
+ ```
174
+
175
+ There are three different kinds of messages:
176
+
177
+ 1. a control message to flush the buffer in the connection
178
+ 2. a control message to synchronize any thread with the sender thread
179
+ 3. a message to append to the buffer
180
+
181
+ There is also an implicit message which closes the queue which will cause the sender thread to finish processing and exit.
182
+
183
+ ### Usual workflow
184
+
185
+ You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
186
+
187
+ ### Flushing
188
+
189
+ When calling `Datadog::Statsd#flush`, a specific control message (`:flush`) is sent to the sender thread. When the sender thread receives it, it flushes its internal buffer into the connection.
190
+
191
+ ### Rendez-vous
192
+
193
+ It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronous flush using `Datadog::Statsd#flush(sync: true)`.
194
+
195
+ Doing so means the caller thread is blocked and waiting until the data has been flushed by the sender thread.
196
+
197
+ This is useful when preparing to exit the application or when checking unit tests.
198
+
199
+ ### Thread-safety
200
+
201
+ By default, instances of `Datadog::Statsd` are thread-safe and we recommend that a single instance be reused by all application threads (even in applications that employ forking). The sole exception is the `#close` method — this method is not yet thread safe (work in progress here [#209](https://github.com/DataDog/dogstatsd-ruby/pull/209)).
202
+
203
+ When using the `single_thread: true` mode, instances of `Datadog::Statsd` are still thread-safe, but you may run into contention on heavily-threaded applications, so we don’t recommend (for performance reasons) reusing these instances.
204
+
205
+ ## Versioning
206
+
207
+ This Ruby gem is using [Semantic Versioning](https://guides.rubygems.org/patterns/#semantic-versioning) but please note that supported Ruby versions can change in a minor release of this library. As much as possible, we will add a "future deprecation" message in the minor release preceding the one dropping the support.
208
+
87
209
  ## Credits
88
210
 
89
- dogstatsd-ruby is forked from Rein Henrichs [original Statsd
90
- client](https://github.com/reinh/statsd).
211
+ dogstatsd-ruby is forked from Rein Henrichs' [original Statsd client](https://github.com/reinh/statsd).
91
212
 
92
213
  Copyright (c) 2011 Rein Henrichs. See LICENSE.txt for
93
214
  further details.
@@ -8,16 +8,11 @@ module Datadog
8
8
  @logger = logger
9
9
  end
10
10
 
11
- # Close the underlying socket
12
- def close
13
- begin
14
- @socket && @socket.close if instance_variable_defined?(:@socket)
15
- rescue StandardError => boom
16
- logger.error { "Statsd: #{boom.class} #{boom}" } if logger
17
- end
18
- @socket = nil
11
+ def reset_telemetry
12
+ telemetry.reset
19
13
  end
20
14
 
15
+ # not thread safe: `Sender` instances that use this are required to properly synchronize or sequence calls to this method
21
16
  def write(payload)
22
17
  logger.debug { "Statsd: #{payload}" } if logger
23
18
 
@@ -36,6 +31,7 @@ module Datadog
36
31
  retries += 1
37
32
  begin
38
33
  close
34
+ connect
39
35
  retry
40
36
  rescue StandardError => e
41
37
  boom = e
@@ -48,11 +44,16 @@ module Datadog
48
44
  end
49
45
 
50
46
  private
47
+
51
48
  attr_reader :telemetry
52
49
  attr_reader :logger
53
50
 
54
- def socket
55
- @socket ||= connect
51
+ def connect
52
+ raise 'Should be implemented by subclass'
53
+ end
54
+
55
+ def close
56
+ raise 'Should be implemented by subclass'
56
57
  end
57
58
  end
58
59
  end
@@ -18,6 +18,8 @@ module Datadog
18
18
  telemetry_flush_interval: nil,
19
19
  global_tags: [],
20
20
 
21
+ single_thread: false,
22
+
21
23
  logger: nil
22
24
  )
23
25
  @transport_type = socket_path.nil? ? :udp : :uds
@@ -47,13 +49,12 @@ module Datadog
47
49
  raise ArgumentError, "buffer_max_payload_size is not high enough to use telemetry (tags=(#{global_tags.inspect}))"
48
50
  end
49
51
 
50
- @buffer = MessageBuffer.new(@connection,
52
+ buffer = MessageBuffer.new(@connection,
51
53
  max_payload_size: buffer_max_payload_size,
52
54
  max_pool_size: buffer_max_pool_size || DEFAULT_BUFFER_POOL_SIZE,
53
55
  overflowing_stategy: buffer_overflowing_stategy,
54
56
  )
55
-
56
- @sender = Sender.new(buffer)
57
+ @sender = (single_thread ? SingleThreadSender : Sender).new(buffer, logger: logger)
57
58
  @sender.start
58
59
  end
59
60
 
@@ -97,7 +98,6 @@ module Datadog
97
98
  end
98
99
 
99
100
  private
100
- attr_reader :buffer
101
101
  attr_reader :sender
102
102
  attr_reader :connection
103
103
 
@@ -19,7 +19,7 @@ module Datadog
19
19
  @overflowing_stategy = overflowing_stategy
20
20
 
21
21
  @buffer = String.new
22
- @message_count = 0
22
+ clear_buffer
23
23
  end
24
24
 
25
25
  def add(message)
@@ -42,16 +42,20 @@ module Datadog
42
42
  true
43
43
  end
44
44
 
45
+ def reset
46
+ clear_buffer
47
+ connection.reset_telemetry
48
+ end
49
+
45
50
  def flush
46
51
  return if buffer.empty?
47
52
 
48
53
  connection.write(buffer)
49
-
50
- buffer.clear
51
- @message_count = 0
54
+ clear_buffer
52
55
  end
53
56
 
54
57
  private
58
+
55
59
  attr :max_payload_size
56
60
  attr :max_pool_size
57
61
 
@@ -66,6 +70,11 @@ module Datadog
66
70
  false
67
71
  end
68
72
 
73
+ def clear_buffer
74
+ buffer.clear
75
+ @message_count = 0
76
+ end
77
+
69
78
  def preemptive_flush?
70
79
  @message_count == max_pool_size || buffer.bytesize > bytesize_threshold
71
80
  end
@@ -2,22 +2,45 @@
2
2
 
3
3
  module Datadog
4
4
  class Statsd
5
+ # Sender is using a companion thread to flush and pack messages
6
+ # in a `MessageBuffer`.
7
+ # The communication with this thread is done using a `Queue`.
8
+ # If the thread is dead, it is starting a new one to avoid having a blocked
9
+ # Sender with no companion thread to communicate with (most of the time, having
10
+ # a dead companion thread means that a fork just happened and that we are
11
+ # running in the child process).
5
12
  class Sender
6
13
  CLOSEABLE_QUEUES = Queue.instance_methods.include?(:close)
7
14
 
8
- def initialize(message_buffer)
15
+ def initialize(message_buffer, logger: nil)
9
16
  @message_buffer = message_buffer
17
+ @logger = logger
18
+ @mx = Mutex.new
10
19
  end
11
20
 
12
21
  def flush(sync: false)
13
- raise ArgumentError, 'Start sender first' unless message_queue
14
-
15
- message_queue.push(:flush)
22
+ # keep a copy around in case another thread is calling #stop while this method is running
23
+ current_message_queue = message_queue
24
+
25
+ # don't try to flush if there is no message_queue instantiated or
26
+ # no companion thread running
27
+ if !current_message_queue
28
+ @logger.debug { "Statsd: can't flush: no message queue ready" } if @logger
29
+ return
30
+ end
31
+ if !sender_thread.alive?
32
+ @logger.debug { "Statsd: can't flush: no sender_thread alive" } if @logger
33
+ return
34
+ end
16
35
 
36
+ current_message_queue.push(:flush)
17
37
  rendez_vous if sync
18
38
  end
19
39
 
20
40
  def rendez_vous
41
+ # could happen if #start hasn't be called
42
+ return unless message_queue
43
+
21
44
  # Initialize and get the thread's sync queue
22
45
  queue = (Thread.current[:statsd_sync_queue] ||= Queue.new)
23
46
  # tell sender-thread to notify us in the current
@@ -31,19 +54,39 @@ module Datadog
31
54
  def add(message)
32
55
  raise ArgumentError, 'Start sender first' unless message_queue
33
56
 
57
+ # if the thread does not exist, we assume we are running in a forked process,
58
+ # empty the message queue and message buffers (these messages belong to
59
+ # the parent process) and spawn a new companion thread.
60
+ if !sender_thread.alive?
61
+ @mx.synchronize {
62
+ # a call from another thread has already re-created
63
+ # the companion thread before this one acquired the lock
64
+ break if sender_thread.alive?
65
+ @logger.debug { "Statsd: companion thread is dead, re-creating one" } if @logger
66
+
67
+ message_queue.close if CLOSEABLE_QUEUES
68
+ @message_queue = nil
69
+ message_buffer.reset
70
+ start
71
+ }
72
+ end
73
+
34
74
  message_queue << message
35
75
  end
36
76
 
37
77
  def start
38
78
  raise ArgumentError, 'Sender already started' if message_queue
39
79
 
40
- # initialize message queue for background thread
80
+ # initialize a new message queue for the background thread
41
81
  @message_queue = Queue.new
42
82
  # start background thread
43
83
  @sender_thread = Thread.new(&method(:send_loop))
44
84
  end
45
85
 
46
86
  if CLOSEABLE_QUEUES
87
+ # when calling stop, make sure that no other threads is trying
88
+ # to close the sender nor trying to continue to `#add` more message
89
+ # into the sender.
47
90
  def stop(join_worker: true)
48
91
  message_queue = @message_queue
49
92
  message_queue.close if message_queue
@@ -52,6 +95,9 @@ module Datadog
52
95
  sender_thread.join if sender_thread && join_worker
53
96
  end
54
97
  else
98
+ # when calling stop, make sure that no other threads is trying
99
+ # to close the sender nor trying to continue to `#add` more message
100
+ # into the sender.
55
101
  def stop(join_worker: true)
56
102
  message_queue = @message_queue
57
103
  message_queue << :close if message_queue
@@ -64,7 +110,6 @@ module Datadog
64
110
  private
65
111
 
66
112
  attr_reader :message_buffer
67
-
68
113
  attr_reader :message_queue
69
114
  attr_reader :sender_thread
70
115
 
@@ -0,0 +1,62 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Datadog
4
+ class Statsd
5
+ # The SingleThreadSender is a sender synchronously buffering messages
6
+ # in a `MessageBuffer`.
7
+ # It is using current Process.PID to check it is the result of a recent fork
8
+ # and it is reseting the MessageBuffer if that's the case.
9
+ class SingleThreadSender
10
+ def initialize(message_buffer, logger: nil)
11
+ @message_buffer = message_buffer
12
+ @logger = logger
13
+ @mx = Mutex.new
14
+ # store the pid for which this sender has been created
15
+ update_fork_pid
16
+ end
17
+
18
+ def add(message)
19
+ @mx.synchronize {
20
+ # we have just forked, meaning we have messages in the buffer that we should
21
+ # not send, they belong to the parent process, let's clear the buffer.
22
+ if forked?
23
+ @message_buffer.reset
24
+ update_fork_pid
25
+ end
26
+ @message_buffer.add(message)
27
+ }
28
+ end
29
+
30
+ def flush(*)
31
+ @mx.synchronize {
32
+ @message_buffer.flush()
33
+ }
34
+ end
35
+
36
+ # Compatibility with `Sender`
37
+ def start()
38
+ end
39
+
40
+ # Compatibility with `Sender`
41
+ def stop()
42
+ end
43
+
44
+ # Compatibility with `Sender`
45
+ def rendez_vous()
46
+ end
47
+
48
+ private
49
+
50
+ # below are "fork management" methods to be able to clean the MessageBuffer
51
+ # if it detects that it is running in a unknown PID.
52
+
53
+ def forked?
54
+ Process.pid != @fork_pid
55
+ end
56
+
57
+ def update_fork_pid
58
+ @fork_pid = Process.pid
59
+ end
60
+ end
61
+ end
62
+ end
@@ -0,0 +1,132 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Datadog
4
+ class FlushQueue < Queue
5
+ end
6
+ class CloseQueue < Queue
7
+ end
8
+ class Statsd
9
+ # Sender is using a background thread to flush and pack messages
10
+ # in a `MessageBuffer`.
11
+ # The communication with this thread is done using a `Queue`.
12
+ # If the thread is dead, it is starting a new one to avoid having a blocked
13
+ # Sender with no background thread to communicate with (most of the time,
14
+ # having a dead background thread means that a fork just happened and that we
15
+ # are running in the child process).
16
+ class Sender
17
+ CLOSEABLE_QUEUES = Queue.instance_methods.include?(:close)
18
+
19
+ def initialize(message_buffer, logger: nil)
20
+ @message_buffer = message_buffer
21
+ @logger = logger
22
+
23
+ # communication and synchronization with the background thread
24
+ # @mux is also used to not having multiple threads fighting for
25
+ # closing the Sender or creating a new background thread
26
+ @channel = Queue.new
27
+ @mux = Mutex.new
28
+
29
+ @is_closed = false
30
+
31
+ # start background thread immediately
32
+ @sender_thread = Thread.new(&method(:send_loop))
33
+ end
34
+
35
+ def flush(sync: false)
36
+ @mux.synchronize {
37
+ # we don't want to send a flush action to the bg thread if:
38
+ # - there is no bg thread running
39
+ # - the sender has been closed
40
+ return if !sender_thread.alive? || @is_closed
41
+
42
+ if sync
43
+ # blocking flush
44
+ blocking_queue = FlushQueue.new
45
+ channel << blocking_queue
46
+ blocking_queue.pop # wait for the bg thread to finish its work
47
+ blocking_queue.close if CLOSEABLE_QUEUES
48
+ else
49
+ # asynchronous flush
50
+ channel << :flush
51
+ end
52
+ }
53
+ end
54
+
55
+ def add(message)
56
+ return if @is_closed # don't send a message to the bg thread if the sender has been closed
57
+
58
+ # the bg thread is not running anymore, this is happening if the main process has forked and
59
+ # we are running in the child, we will spawn a bg thread and reset buffers (containing parents' messages)
60
+ if !sender_thread.alive?
61
+ @mux.synchronize {
62
+ return if @is_closed
63
+ # test if a call from another thread has already re-created
64
+ # the background thread before this one acquired the lock
65
+ break if sender_thread.alive?
66
+
67
+ # re-create the channel of communication since we will spawn a new bg thread
68
+ channel.close if CLOSEABLE_QUEUES
69
+ @channel = Queue.new
70
+ message_buffer.reset # don't use messages appended by another fork
71
+ @sender_thread = Thread.new(&method(:send_loop))
72
+ }
73
+ end
74
+
75
+ channel << message
76
+ end
77
+
78
+ # Compatibility with `Sender`
79
+ def start()
80
+ end
81
+
82
+ def stop()
83
+ return if @is_closed
84
+ # use this lock to both: not having another thread stopping this instance nor
85
+ # having a #add call creating a new thread
86
+ @mux.synchronize {
87
+ @is_closed = true
88
+ if sender_thread.alive? # no reasons to stop the bg thread is none is running already
89
+ blocking_queue = CloseQueue.new
90
+ channel << blocking_queue
91
+ blocking_queue.pop # wait for the bg thread to finish its work
92
+ blocking_queue.close if CLOSEABLE_QUEUES
93
+ sender_thread.join(3) # wait for completion, timeout after 3 seconds
94
+ # TODO(remy): should I close `channel` here?
95
+ end
96
+ }
97
+ end
98
+
99
+ private
100
+
101
+ attr_reader :message_buffer
102
+ attr_reader :channel
103
+ attr_reader :mux
104
+ attr_reader :sender_thread
105
+
106
+ def send_loop
107
+ until (message = channel.pop).nil? && (CLOSEABLE_QUEUES && channel.closed?)
108
+ # skip if message is nil, e.g. when the channel is empty and closed
109
+ next unless message
110
+
111
+ case message
112
+ # if a FlushQueue is received, the background thread has to flush the message
113
+ # buffer and to send an :unblock to let the caller know that it has finished
114
+ when FlushQueue
115
+ message_buffer.flush
116
+ message << :unblock
117
+ # if a :flush is received, the background thread has to flush asynchronously
118
+ when :flush
119
+ message_buffer.flush
120
+ # if a CloseQueue is received, the background thread has to do a last flush
121
+ # and to send an :unblock to let the caller know that it has finished
122
+ when CloseQueue
123
+ message << :unblock
124
+ return
125
+ else
126
+ message_buffer.add(message)
127
+ end
128
+ end
129
+ end
130
+ end
131
+ end
132
+ end
@@ -19,18 +19,30 @@ module Datadog
19
19
 
20
20
  @host = host || ENV.fetch('DD_AGENT_HOST', DEFAULT_HOST)
21
21
  @port = port || ENV.fetch('DD_DOGSTATSD_PORT', DEFAULT_PORT).to_i
22
+ @socket = nil
23
+ end
24
+
25
+ def close
26
+ @socket.close if @socket
27
+ @socket = nil
22
28
  end
23
29
 
24
30
  private
25
31
 
26
32
  def connect
27
- UDPSocket.new.tap do |socket|
28
- socket.connect(host, port)
29
- end
33
+ close if @socket
34
+
35
+ @socket = UDPSocket.new
36
+ @socket.connect(host, port)
30
37
  end
31
38
 
39
+ # send_message is writing the message in the socket, it may create the socket if nil
40
+ # It is not thread-safe but since it is called by either the Sender bg thread or the
41
+ # SingleThreadSender (which is using a mutex while Flushing), only one thread must call
42
+ # it at a time.
32
43
  def send_message(message)
33
- socket.send(message, 0)
44
+ connect unless @socket
45
+ @socket.send(message, 0)
34
46
  end
35
47
  end
36
48
  end
@@ -14,20 +14,31 @@ module Datadog
14
14
  super(**kwargs)
15
15
 
16
16
  @socket_path = socket_path
17
+ @socket = nil
18
+ end
19
+
20
+ def close
21
+ @socket.close if @socket
22
+ @socket = nil
17
23
  end
18
24
 
19
25
  private
20
26
 
21
27
  def connect
22
- socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM)
23
- socket.connect(Socket.pack_sockaddr_un(@socket_path))
24
- socket
28
+ close if @socket
29
+
30
+ @socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM)
31
+ @socket.connect(Socket.pack_sockaddr_un(@socket_path))
25
32
  end
26
33
 
34
+ # send_message is writing the message in the socket, it may create the socket if nil
35
+ # It is not thread-safe but since it is called by either the Sender bg thread or the
36
+ # SingleThreadSender (which is using a mutex while Flushing), only one thread must call
37
+ # it at a time.
27
38
  def send_message(message)
28
- socket.sendmsg_nonblock(message)
39
+ connect unless @socket
40
+ @socket.sendmsg_nonblock(message)
29
41
  rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::ENOENT => e
30
- @socket = nil
31
42
  # TODO: FIXME: This error should be considered as a retryable error in the
32
43
  # Connection class. An even better solution would be to make BadSocketError inherit
33
44
  # from a specific retryable error class in the Connection class.
@@ -4,6 +4,6 @@ require_relative 'connection'
4
4
 
5
5
  module Datadog
6
6
  class Statsd
7
- VERSION = '5.0.1'
7
+ VERSION = '5.3.1'
8
8
  end
9
9
  end
@@ -8,8 +8,12 @@ require_relative 'statsd/uds_connection'
8
8
  require_relative 'statsd/message_buffer'
9
9
  require_relative 'statsd/serialization'
10
10
  require_relative 'statsd/sender'
11
+ require_relative 'statsd/single_thread_sender'
11
12
  require_relative 'statsd/forwarder'
12
13
 
14
+ $deprecation_message_mutex = Mutex.new
15
+ $deprecation_message_done = false
16
+
13
17
  # = Datadog::Statsd: A DogStatsd client (https://www.datadoghq.com)
14
18
  #
15
19
  # @example Set up a global Statsd client for a server on localhost:8125
@@ -70,6 +74,7 @@ module Datadog
70
74
  # @option [Integer] buffer_max_pool_size max messages to buffer
71
75
  # @option [String] socket_path unix socket path
72
76
  # @option [Float] default sample rate if not overridden
77
+ # @option [Boolean] single_thread flushes the metrics on the main thread instead of in a companion thread
73
78
  def initialize(
74
79
  host = nil,
75
80
  port = nil,
@@ -85,6 +90,8 @@ module Datadog
85
90
 
86
91
  logger: nil,
87
92
 
93
+ single_thread: false,
94
+
88
95
  telemetry_enable: true,
89
96
  telemetry_flush_interval: DEFAULT_TELEMETRY_FLUSH_INTERVAL
90
97
  )
@@ -97,6 +104,19 @@ module Datadog
97
104
  @serializer = Serialization::Serializer.new(prefix: @prefix, global_tags: tags)
98
105
  @sample_rate = sample_rate
99
106
 
107
+ # deprecation message for ruby < 2.1.0 users as we will drop support for ruby 2.0
108
+ # in dogstatsd-ruby 5.4.0
109
+ # TODO(remy): remove this message and the two global vars used in dogstatd-ruby 5.4.0
110
+ if RUBY_VERSION < '2.1.0' && $deprecation_message_mutex.try_lock && !$deprecation_message_done
111
+ if logger != nil
112
+ logger.warn { "deprecation: dogstatsd-ruby will drop support of Ruby < 2.1.0 in a next minor release" }
113
+ else
114
+ puts("warning: deprecation: dogstatsd-ruby will drop support of Ruby < 2.1.0 in a next minor release")
115
+ end
116
+ $deprecation_message_done = true
117
+ $deprecation_message_mutex.unlock
118
+ end
119
+
100
120
  @forwarder = Forwarder.new(
101
121
  host: host,
102
122
  port: port,
@@ -105,6 +125,8 @@ module Datadog
105
125
  global_tags: tags,
106
126
  logger: logger,
107
127
 
128
+ single_thread: single_thread,
129
+
108
130
  buffer_max_payload_size: buffer_max_payload_size,
109
131
  buffer_max_pool_size: buffer_max_pool_size,
110
132
  buffer_overflowing_stategy: buffer_overflowing_stategy,
@@ -320,7 +342,10 @@ module Datadog
320
342
  end
321
343
 
322
344
  # Close the underlying socket
323
- def close
345
+ #
346
+ # @param [Boolean, true] flush Should we flush the metrics before closing
347
+ def close(flush: true)
348
+ flush(sync: true) if flush
324
349
  forwarder.close
325
350
  end
326
351
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: dogstatsd-ruby
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.0.1
4
+ version: 5.3.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Rein Henrichs
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2021-04-09 00:00:00.000000000 Z
12
+ date: 2021-10-21 00:00:00.000000000 Z
13
13
  dependencies: []
14
14
  description: A Ruby DogStatsd client
15
15
  email: code@datadoghq.com
@@ -32,7 +32,9 @@ files:
32
32
  - lib/datadog/statsd/serialization/service_check_serializer.rb
33
33
  - lib/datadog/statsd/serialization/stat_serializer.rb
34
34
  - lib/datadog/statsd/serialization/tag_serializer.rb
35
+ - lib/datadog/statsd/single_thread_sender.rb
35
36
  - lib/datadog/statsd/telemetry.rb
37
+ - lib/datadog/statsd/threaded_sender.rb
36
38
  - lib/datadog/statsd/udp_connection.rb
37
39
  - lib/datadog/statsd/uds_connection.rb
38
40
  - lib/datadog/statsd/version.rb
@@ -41,9 +43,9 @@ licenses:
41
43
  - MIT
42
44
  metadata:
43
45
  bug_tracker_uri: https://github.com/DataDog/dogstatsd-ruby/issues
44
- changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.0.1/CHANGELOG.md
45
- documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.0.1
46
- source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.0.1
46
+ changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.3.1/CHANGELOG.md
47
+ documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.3.1
48
+ source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.3.1
47
49
  post_install_message:
48
50
  rdoc_options: []
49
51
  require_paths: