speedshop-cloudwatch 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: cb7393054526aee599b81c7f6031475244449ec5e2f6137f509b0fde8811c496
4
- data.tar.gz: 45eababb5f976f5fe8eeacf13d43a13e1edc3a06b3a57e5f7a127ffa0385324e
3
+ metadata.gz: 19039fb6a3ef58a84cae232f0c6a501451c2ee7d52a49762aca926f9e22f733c
4
+ data.tar.gz: f8084e387ec56912545f0c4b04edf6f26210f57fb015189578311ce5dfc421d2
5
5
  SHA512:
6
- metadata.gz: f09b6d50481cb2ea0bb2d25a01fe93955321c1f528145a6e864b959d81724e87067e0851d8dd13d78d3864bbf60d6342bba737696350d2f10be50f6b5e06f1ec
7
- data.tar.gz: 9c719ebf214f5112f95c15b31327ef71dff873b5eeb71ac2c5fbdf2ad1dffcba908abb74867f1429406095b8c7b3712d999650a8ca6d38af6be405ac68cd426e
6
+ metadata.gz: 2b62fe5780fd5646a230f51c8b5d4cd4dfb2ef47915cf9ddd983dbe0af00c97d5b52106c4e74729b771c3466a4d872ef2965653c5f2e3606a405158a71f65ffd
7
+ data.tar.gz: e49872a3eb6a96dc6a51f14defcf77b6139d3a2bb7479e0c33600fbfb956f291337ab32f549e7beb40befa3094f3a51ee6859fd10008b42b7784c9f24c37ebc7
data/CHANGELOG.md ADDED
@@ -0,0 +1,25 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [0.2.0] - 2026-04-13
9
+
10
+ ### Added
11
+ - Added an optional Yabeda adapter so Yabeda-defined metrics can be delivered through the gem's async CloudWatch reporter pipeline.
12
+
13
+ ### Changed
14
+ - Reduced CloudWatch metric cardinality and cost by removing the `WorkerIndex`, `Tag`, and `Hostname` dimensions from the affected Puma and Sidekiq metrics.
15
+
16
+ ### Upgrade Notes
17
+ - Existing CloudWatch dashboards or alarms that filter on `WorkerIndex`, `Tag`, or `Hostname` dimensions will need to be updated before upgrading.
18
+
19
+ ## [0.1.0] - 2025-11-17
20
+
21
+ ### Added
22
+ - Initial public release.
23
+
24
+ [0.2.0]: https://github.com/speedshop/speedshop-cloudwatch/compare/v0.1.0...v0.2.0
25
+ [0.1.0]: https://github.com/speedshop/speedshop-cloudwatch/releases/tag/v0.1.0
data/README.md CHANGED
@@ -49,6 +49,12 @@ config.metrics[:sidekiq] = [
49
49
  gem 'speedshop-cloudwatch'
50
50
  ```
51
51
 
52
+ If you want to export metrics defined with Yabeda's DSL through this gem, also add:
53
+
54
+ ```ruby
55
+ gem 'yabeda'
56
+ ```
57
+
52
58
  See each integration below for instructions on how to setup and configure that integration.
53
59
 
54
60
  ## Configuration
@@ -204,6 +210,44 @@ QueueLatency - Time job spent waiting in queue before execution (Seconds)
204
210
 
205
211
  This metric includes QueueName dimension and is aggregated per interval using CloudWatch StatisticSets.
206
212
 
213
+ ## Yabeda
214
+
215
+ If you define application metrics with [Yabeda](https://github.com/yabeda-rb/yabeda), this gem can act as a Yabeda adapter and send those metrics through the same async `Reporter` pipeline used by the built-in integrations.
216
+
217
+ This is optional. If you use it, you'll also want Yabeda metric collectors such as [yabeda-sidekiq](https://github.com/yabeda-rb/yabeda-sidekiq), [yabeda-puma-plugin](https://github.com/yabeda-rb/yabeda-puma-plugin), or [yabeda-rack-queue](https://github.com/speedshop/yabeda-rack-queue).
218
+
219
+ ```ruby
220
+ # Gemfile
221
+ gem 'speedshop-cloudwatch'
222
+ gem 'yabeda'
223
+ gem 'yabeda-sidekiq'
224
+ ```
225
+
226
+ ```ruby
227
+ # config/initializers/yabeda.rb
228
+ require 'speedshop/cloudwatch/yabeda'
229
+ require 'yabeda/sidekiq'
230
+
231
+ Yabeda.register_adapter(:cloudwatch, Speedshop::Cloudwatch::Yabeda.new)
232
+ ```
233
+
234
+ If you're using collector-driven Yabeda integrations such as `yabeda-puma-plugin`, start the reporter in that process during boot so `Yabeda.collect!` has somewhere to run. For Puma, that usually means `config/puma.rb`:
235
+
236
+ ```ruby
237
+ plugin :yabeda
238
+ Speedshop::Cloudwatch.start!
239
+ ```
240
+
241
+ Direct Yabeda `increment`, `set`, `measure`, and `observe` calls lazy-start the reporter on first use. Periodic Yabeda collectors only run inside that reporter thread.
242
+
243
+ Behavior notes:
244
+
245
+ - Group names become CloudWatch namespaces in [`namespace_for`](lib/speedshop/cloudwatch/yabeda.rb).
246
+ - Yabeda tags become CloudWatch dimensions, and `config.dimensions` are appended in [`dimensions_for`](lib/speedshop/cloudwatch/yabeda.rb).
247
+ - Known Yabeda units are mapped to CloudWatch units in [`UNIT_MAP`](lib/speedshop/cloudwatch/yabeda.rb); unknown units default to `"None"` in [`unit_for`](lib/speedshop/cloudwatch/yabeda.rb).
248
+ - Direct Yabeda updates are enqueued by the adapter, and periodic Yabeda collectors are run by [`Yabeda::Collector`](lib/speedshop/cloudwatch/yabeda.rb) through the async [`Reporter`](lib/speedshop/cloudwatch/reporter.rb).
249
+ - This adapter is opt-in and is **not** loaded by `require 'speedshop/cloudwatch/all'`.
250
+
207
251
  ## Rails
208
252
 
209
253
  When running in a Rails app we:
@@ -1,5 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "speedshop/cloudwatch/observations"
4
+
3
5
  module Speedshop
4
6
  module Cloudwatch
5
7
  module ActiveJob
@@ -9,10 +11,14 @@ module Speedshop
9
11
 
10
12
  def report_job_metrics
11
13
  begin
12
- if enqueued_at
13
- queue_time = Time.now.to_f - enqueued_at.to_f
14
- # Drop JobClass to reduce time series cardinality and allow aggregation into StatisticSets per queue
15
- Reporter.instance.report(metric: :QueueLatency, value: queue_time, dimensions: {QueueName: queue_name}, integration: :active_job)
14
+ observation = Observations::ActiveJob.queue_latency(self)
15
+ if observation
16
+ MetricMapper.instance.report(
17
+ metric: :QueueLatency,
18
+ value: observation[:value],
19
+ dimensions: observation[:dimensions],
20
+ integration: :active_job
21
+ )
16
22
  end
17
23
  rescue => e
18
24
  Speedshop::Cloudwatch.log_error("Failed to collect ActiveJob metrics: #{e.message}", e)
@@ -0,0 +1,74 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "singleton"
4
+ require_relative "metrics"
5
+
6
+ module Speedshop
7
+ module Cloudwatch
8
+ # Maps this gem's built-in metric families in METRICS to CloudWatch datums.
9
+ # The Yabeda adapter bypasses this class and builds datums directly because
10
+ # Yabeda metrics are dynamic and don't need to be present in METRICS.
11
+ class MetricMapper
12
+ include Singleton
13
+
14
+ def report(metric:, value: nil, statistic_values: nil, dimensions: {}, integration: nil)
15
+ return unless config.environment_enabled?
16
+
17
+ metric_name = metric.to_sym
18
+ resolved_integration = integration || find_integration_for_metric(metric_name)
19
+ return unless resolved_integration
20
+ return unless metric_allowed?(resolved_integration, metric_name)
21
+
22
+ Reporter.instance.enqueue(build_datum(
23
+ metric_name: metric_name,
24
+ integration: resolved_integration,
25
+ value: value,
26
+ statistic_values: statistic_values,
27
+ dimensions: dimensions
28
+ ))
29
+ end
30
+
31
+ private
32
+
33
+ def config
34
+ Config.instance
35
+ end
36
+
37
+ def build_datum(metric_name:, integration:, value:, statistic_values:, dimensions:)
38
+ metric_object = METRICS[integration]&.find { |metric| metric.name == metric_name }
39
+ datum = {
40
+ metric_name: metric_name.to_s,
41
+ namespace: config.namespaces[integration],
42
+ unit: metric_object&.unit || "None",
43
+ dimensions: metric_dimensions(dimensions),
44
+ timestamp: Time.now
45
+ }
46
+
47
+ if statistic_values
48
+ datum[:statistic_values] = statistic_values
49
+ else
50
+ datum[:value] = value
51
+ end
52
+
53
+ datum
54
+ end
55
+
56
+ def metric_dimensions(dimensions)
57
+ metric_dimensions = dimensions.map { |name, value| {name: name.to_s, value: value.to_s} }
58
+ metric_dimensions + custom_dimensions
59
+ end
60
+
61
+ def metric_allowed?(integration, metric_name)
62
+ config.metrics[integration].include?(metric_name.to_sym)
63
+ end
64
+
65
+ def custom_dimensions
66
+ config.dimensions.map { |name, value| {name: name.to_s, value: value.to_s} }
67
+ end
68
+
69
+ def find_integration_for_metric(metric_name)
70
+ METRICS.find { |integration, metrics| metrics.any? { |metric| metric.name == metric_name } }&.first
71
+ end
72
+ end
73
+ end
74
+ end
@@ -0,0 +1,20 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Speedshop
4
+ module Cloudwatch
5
+ module Observations
6
+ module ActiveJob
7
+ module_function
8
+
9
+ def queue_latency(job, now: Time.now.to_f)
10
+ return unless job.enqueued_at
11
+
12
+ {
13
+ value: now - job.enqueued_at.to_f,
14
+ dimensions: {QueueName: job.queue_name}
15
+ }
16
+ end
17
+ end
18
+ end
19
+ end
20
+ end
@@ -0,0 +1,46 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Speedshop
4
+ module Cloudwatch
5
+ module Observations
6
+ module Puma
7
+ module_function
8
+
9
+ def from_stats(stats)
10
+ if stats[:worker_status]
11
+ clustered_observations(stats)
12
+ else
13
+ single_mode_observations(stats)
14
+ end
15
+ end
16
+
17
+ def clustered_observations(stats)
18
+ observations = %i[workers booted_workers old_workers].map do |metric|
19
+ {metric: metric_name_for(metric), value: stats[metric] || 0}
20
+ end
21
+
22
+ stats[:worker_status].each do |worker_status|
23
+ last_status = worker_status[:last_status] || {}
24
+ %i[running backlog pool_capacity max_threads].each do |metric|
25
+ next unless last_status.key?(metric)
26
+
27
+ observations << {metric: metric_name_for(metric), value: last_status[metric]}
28
+ end
29
+ end
30
+
31
+ observations
32
+ end
33
+
34
+ def single_mode_observations(stats)
35
+ %i[running backlog pool_capacity max_threads].map do |metric|
36
+ {metric: metric_name_for(metric), value: stats[metric] || 0}
37
+ end
38
+ end
39
+
40
+ def metric_name_for(symbol)
41
+ symbol.to_s.split("_").map(&:capitalize).join.to_sym
42
+ end
43
+ end
44
+ end
45
+ end
46
+ end
@@ -0,0 +1,22 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Speedshop
4
+ module Cloudwatch
5
+ module Observations
6
+ module Rack
7
+ module_function
8
+
9
+ def request_queue_time(env, now_ms: current_time_ms)
10
+ header = env["HTTP_X_REQUEST_START"] || env["HTTP_X_QUEUE_START"]
11
+ return unless header
12
+
13
+ now_ms - header.gsub("t=", "").to_f
14
+ end
15
+
16
+ def current_time_ms
17
+ Time.now.to_f * 1000
18
+ end
19
+ end
20
+ end
21
+ end
22
+ end
@@ -0,0 +1,65 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "sidekiq/api" if defined?(::Sidekiq)
4
+
5
+ module Speedshop
6
+ module Cloudwatch
7
+ module Observations
8
+ module Sidekiq
9
+ module_function
10
+
11
+ def collect(sidekiq_queues: Config.instance.sidekiq_queues)
12
+ stats = ::Sidekiq::Stats.new
13
+ processes = ::Sidekiq::ProcessSet.new.to_a
14
+
15
+ stat_observations(stats) + utilization_observations(processes) + queue_observations(sidekiq_queues)
16
+ end
17
+
18
+ def stat_observations(stats)
19
+ {
20
+ EnqueuedJobs: stats.enqueued,
21
+ ProcessedJobs: stats.processed,
22
+ FailedJobs: stats.failed,
23
+ ScheduledJobs: stats.scheduled_size,
24
+ RetryJobs: stats.retry_size,
25
+ DeadJobs: stats.dead_size,
26
+ Workers: stats.workers_size,
27
+ Processes: stats.processes_size,
28
+ DefaultQueueLatency: stats.default_queue_latency
29
+ }.map do |metric, value|
30
+ {metric: metric, value: value}
31
+ end
32
+ end
33
+
34
+ def utilization_observations(processes)
35
+ observations = [{metric: :Capacity, value: processes.sum { |process| process["concurrency"] }}]
36
+
37
+ utilization = avg_utilization(processes) * 100.0
38
+ observations << {metric: :Utilization, value: utilization} unless utilization.nan?
39
+ observations
40
+ end
41
+
42
+ def queue_observations(sidekiq_queues)
43
+ queues_to_monitor(sidekiq_queues).flat_map do |queue|
44
+ [
45
+ {metric: :QueueLatency, value: queue.latency, dimensions: {QueueName: queue.name}},
46
+ {metric: :QueueSize, value: queue.size, dimensions: {QueueName: queue.name}}
47
+ ]
48
+ end
49
+ end
50
+
51
+ def queues_to_monitor(sidekiq_queues)
52
+ all_queues = ::Sidekiq::Queue.all
53
+ return all_queues if sidekiq_queues.nil? || sidekiq_queues.empty?
54
+
55
+ all_queues.select { |queue| sidekiq_queues.include?(queue.name) }
56
+ end
57
+
58
+ def avg_utilization(processes)
59
+ utils = processes.map { |process| process["busy"] / process["concurrency"].to_f }.reject(&:nan?)
60
+ utils.sum / utils.size.to_f
61
+ end
62
+ end
63
+ end
64
+ end
65
+ end
@@ -0,0 +1,6 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "speedshop/cloudwatch/observations/active_job"
4
+ require "speedshop/cloudwatch/observations/puma"
5
+ require "speedshop/cloudwatch/observations/rack"
6
+ require "speedshop/cloudwatch/observations/sidekiq"
@@ -1,56 +1,21 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "speedshop/cloudwatch/observations"
4
+
3
5
  module Speedshop
4
6
  module Cloudwatch
5
7
  class Puma
6
8
  def collect
7
- stats = ::Puma.stats_hash
8
-
9
- if stats[:worker_status]
10
- %i[workers booted_workers old_workers].each do |m|
11
- Reporter.instance.report(metric: metric_name_for(m), value: stats[m] || 0)
12
- end
13
- report_aggregate_worker_stats(stats)
14
- else
15
- # Single mode - report worker stats without dimensions
16
- %i[running backlog pool_capacity max_threads].each do |m|
17
- Reporter.instance.report(metric: metric_name_for(m), value: stats[m] || 0)
18
- end
19
- end
20
- rescue => e
21
- Speedshop::Cloudwatch.log_error("Failed to collect Puma metrics: #{e.message}", e)
22
- end
23
-
24
- private
25
-
26
- def report_aggregate_worker_stats(stats)
27
- statuses = stats[:worker_status].map { |w| w[:last_status] || {} }
28
- metrics = %i[running backlog pool_capacity max_threads]
29
-
30
- metrics.each do |m|
31
- values = statuses.map { |s| s[m] }.compact
32
- next if values.empty?
33
-
34
- sample_count = values.length
35
- sum = values.inject(0) { |acc, v| acc + v.to_f }
36
- minimum = values.min.to_f
37
- maximum = values.max.to_f
38
-
39
- Reporter.instance.report(
40
- metric: metric_name_for(m),
41
- statistic_values: {
42
- sample_count: sample_count,
43
- sum: sum,
44
- minimum: minimum,
45
- maximum: maximum
46
- },
9
+ Observations::Puma.from_stats(::Puma.stats_hash).each do |observation|
10
+ MetricMapper.instance.report(
11
+ metric: observation[:metric],
12
+ value: observation[:value],
13
+ dimensions: observation[:dimensions] || {},
47
14
  integration: :puma
48
15
  )
49
16
  end
50
- end
51
-
52
- def metric_name_for(symbol)
53
- symbol.to_s.split("_").map(&:capitalize).join.to_sym
17
+ rescue => e
18
+ Speedshop::Cloudwatch.log_error("Failed to collect Puma metrics: #{e.message}", e)
54
19
  end
55
20
  end
56
21
  end
@@ -1,5 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "speedshop/cloudwatch/observations"
4
+
3
5
  module Speedshop
4
6
  module Cloudwatch
5
7
  class Rack
@@ -9,10 +11,8 @@ module Speedshop
9
11
 
10
12
  def call(env)
11
13
  begin
12
- if (header = env["HTTP_X_REQUEST_START"] || env["HTTP_X_QUEUE_START"])
13
- queue_time = (Time.now.to_f * 1000) - header.gsub("t=", "").to_f
14
- Reporter.instance.report(metric: :RequestQueueTime, value: queue_time)
15
- end
14
+ queue_time = Observations::Rack.request_queue_time(env)
15
+ MetricMapper.instance.report(metric: :RequestQueueTime, value: queue_time) if queue_time
16
16
  rescue => e
17
17
  Speedshop::Cloudwatch.log_error("Failed to collect Rack metrics: #{e.message}", e)
18
18
  end
@@ -1,7 +1,6 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "singleton"
4
- require_relative "metrics"
5
4
 
6
5
  module Speedshop
7
6
  module Cloudwatch
@@ -26,11 +25,8 @@ module Speedshop
26
25
  @mutex.synchronize do
27
26
  return if started?
28
27
 
28
+ reset_after_fork! if forked?
29
29
  initialize_collectors
30
- if forked?
31
- @collectors.clear
32
- @queue.clear
33
- end
34
30
 
35
31
  Speedshop::Cloudwatch.log_info("Starting metric reporter (collectors: #{@collectors.map(&:class).join(", ")})")
36
32
  @running = true
@@ -68,30 +64,12 @@ module Speedshop
68
64
  end
69
65
  end
70
66
 
71
- def report(metric:, value: nil, statistic_values: nil, dimensions: {}, integration: nil)
67
+ def enqueue(datum)
72
68
  return unless config.environment_enabled?
73
69
 
74
- metric_name = metric.to_sym
75
- int = integration || find_integration_for_metric(metric_name)
76
- return unless int
77
- return unless metric_allowed?(int, metric_name)
78
-
79
- metric_object = METRICS[int]&.find { |m| m.name == metric_name }
80
- ns = config.namespaces[int]
81
- unit = metric_object&.unit || "None"
82
-
83
- dimensions_array = dimensions.map { |k, v| {name: k.to_s, value: v.to_s} }
84
- all_dimensions = dimensions_array + custom_dimensions
85
-
86
- datum = {metric_name: metric_name.to_s, namespace: ns, unit: unit,
87
- dimensions: all_dimensions, timestamp: Time.now}
88
- if statistic_values
89
- datum[:statistic_values] = statistic_values
90
- else
91
- datum[:value] = value
92
- end
93
-
94
70
  @mutex.synchronize do
71
+ reset_after_fork! if forked?
72
+
95
73
  if @queue.size >= config.queue_max_size
96
74
  @queue.shift
97
75
  @dropped_since_last_flush += 1
@@ -137,10 +115,20 @@ module Speedshop
137
115
  @pid != Process.pid
138
116
  end
139
117
 
118
+ def reset_after_fork!
119
+ @collectors.clear
120
+ @queue.clear
121
+ @thread = nil
122
+ @running = false
123
+ @dropped_since_last_flush = 0
124
+ @pid = Process.pid
125
+ end
126
+
140
127
  def initialize_collectors
141
128
  config.collectors.each do |integration|
142
129
  @collectors << Speedshop::Cloudwatch::Puma.new if integration == :puma
143
130
  @collectors << Speedshop::Cloudwatch::Sidekiq.new if integration == :sidekiq
131
+ @collectors << Speedshop::Cloudwatch::Yabeda::Collector.new if integration == :yabeda
144
132
  rescue => e
145
133
  Speedshop::Cloudwatch.log_error("Failed to initialize collector for #{integration}: #{e.message}", e)
146
134
  end
@@ -230,7 +218,7 @@ module Speedshop
230
218
  def group_metrics(ns_metrics)
231
219
  groups = {}
232
220
  ns_metrics.each do |m|
233
- key = [m[:metric_name], m[:unit], normalized_dimensions_key(m[:dimensions])]
221
+ key = [m[:metric_name], m[:unit], normalized_dimensions_key(m[:dimensions]), m[:aggregation_strategy]]
234
222
  (groups[key] ||= []) << m
235
223
  end
236
224
  groups.values
@@ -239,6 +227,10 @@ module Speedshop
239
227
  def aggregate_group(items)
240
228
  return items.first if items.size == 1
241
229
 
230
+ strategy = items.first[:aggregation_strategy]
231
+ return aggregate_most_recent_group(items) if strategy == :most_recent
232
+ return aggregate_max_group(items) if strategy == :max
233
+
242
234
  sample_count, sum, minimum, maximum = aggregate_values(items)
243
235
  {
244
236
  metric_name: items.first[:metric_name],
@@ -275,6 +267,21 @@ module Speedshop
275
267
  [sample_count, sum, minimum, maximum]
276
268
  end
277
269
 
270
+ def aggregate_most_recent_group(items)
271
+ items.last
272
+ end
273
+
274
+ def aggregate_max_group(items)
275
+ items.max_by { |item| item_value_for_max(item) }
276
+ end
277
+
278
+ def item_value_for_max(item)
279
+ return item[:statistic_values][:maximum].to_f if item[:statistic_values]
280
+ return item[:value].to_f if item.key?(:value)
281
+
282
+ -Float::INFINITY
283
+ end
284
+
278
285
  def build_statistic_values(sample_count, sum, minimum, maximum)
279
286
  {
280
287
  sample_count: sample_count,
@@ -288,18 +295,6 @@ module Speedshop
288
295
  (dims || []).sort_by { |d| d[:name].to_s }.map { |d| "#{d[:name]}=#{d[:value]}" }.join("|")
289
296
  end
290
297
 
291
- def metric_allowed?(integration, metric_name)
292
- config.metrics[integration].include?(metric_name.to_sym)
293
- end
294
-
295
- def custom_dimensions
296
- config.dimensions.map { |name, value| {name: name.to_s, value: value.to_s} }
297
- end
298
-
299
- def find_integration_for_metric(metric_name)
300
- METRICS.find { |int, metrics| metrics.any? { |m| m.name == metric_name } }&.first
301
- end
302
-
303
298
  def log_overflow_if_needed
304
299
  dropped = nil
305
300
  @mutex.synchronize do
@@ -24,18 +24,20 @@
24
24
  # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
25
25
  # THE SOFTWARE.
26
26
 
27
- require "sidekiq/api" if defined?(::Sidekiq)
27
+ require "speedshop/cloudwatch/observations"
28
28
 
29
29
  module Speedshop
30
30
  module Cloudwatch
31
31
  class Sidekiq
32
32
  def collect
33
- stats = ::Sidekiq::Stats.new
34
- processes = ::Sidekiq::ProcessSet.new.to_a
35
-
36
- report_stats(stats)
37
- report_utilization(processes)
38
- report_queue_metrics
33
+ Observations::Sidekiq.collect.each do |observation|
34
+ metric_mapper.report(
35
+ metric: observation[:metric],
36
+ value: observation[:value],
37
+ dimensions: observation[:dimensions] || {},
38
+ integration: :sidekiq
39
+ )
40
+ end
39
41
  rescue => e
40
42
  Speedshop::Cloudwatch.log_error("Failed to collect Sidekiq metrics: #{e.message}", e)
41
43
  end
@@ -68,48 +70,8 @@ module Speedshop
68
70
 
69
71
  private
70
72
 
71
- def reporter
72
- Speedshop::Cloudwatch.reporter
73
- end
74
-
75
- def report_stats(stats)
76
- {
77
- EnqueuedJobs: stats.enqueued, ProcessedJobs: stats.processed, FailedJobs: stats.failed,
78
- ScheduledJobs: stats.scheduled_size, RetryJobs: stats.retry_size, DeadJobs: stats.dead_size,
79
- Workers: stats.workers_size, Processes: stats.processes_size,
80
- DefaultQueueLatency: stats.default_queue_latency
81
- }.each { |m, v| reporter.report(metric: m, value: v, integration: :sidekiq) }
82
- end
83
-
84
- def report_utilization(processes)
85
- capacity = processes.sum { |p| p["concurrency"] }
86
- reporter.report(metric: :Capacity, value: capacity)
87
-
88
- utilization = avg_utilization(processes) * 100.0
89
- reporter.report(metric: :Utilization, value: utilization) unless utilization.nan?
90
- end
91
-
92
- def report_queue_metrics
93
- queues_to_monitor.each do |q|
94
- reporter.report(metric: :QueueLatency, value: q.latency, dimensions: {QueueName: q.name})
95
- reporter.report(metric: :QueueSize, value: q.size, dimensions: {QueueName: q.name})
96
- end
97
- end
98
-
99
- def queues_to_monitor
100
- all_queues = ::Sidekiq::Queue.all
101
- configured = Speedshop::Cloudwatch.config.sidekiq_queues
102
-
103
- if configured.nil? || configured.empty?
104
- all_queues
105
- else
106
- all_queues.select { |q| configured.include?(q.name) }
107
- end
108
- end
109
-
110
- def avg_utilization(processes)
111
- utils = processes.map { |p| p["busy"] / p["concurrency"].to_f }.reject(&:nan?)
112
- utils.sum / utils.size.to_f
73
+ def metric_mapper
74
+ Speedshop::Cloudwatch.metric_mapper
113
75
  end
114
76
  end
115
77
  end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Speedshop
4
4
  module Cloudwatch
5
- VERSION = "0.1.0"
5
+ VERSION = "0.2.0"
6
6
  end
7
7
  end
@@ -0,0 +1,150 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "speedshop/cloudwatch"
4
+
5
+ begin
6
+ require "yabeda" unless defined?(::Yabeda)
7
+ rescue LoadError => e
8
+ if e.path == "yabeda"
9
+ raise LoadError, "speedshop/cloudwatch/yabeda requires the `yabeda` gem. Add `gem \"yabeda\"` to your Gemfile."
10
+ end
11
+
12
+ raise
13
+ end
14
+
15
+ module Speedshop
16
+ module Cloudwatch
17
+ class Yabeda < ::Yabeda::BaseAdapter
18
+ UNIT_MAP = {
19
+ seconds: "Seconds",
20
+ milliseconds: "Milliseconds",
21
+ microseconds: "Microseconds",
22
+ bytes: "Bytes",
23
+ kilobytes: "Kilobytes",
24
+ megabytes: "Megabytes",
25
+ gigabytes: "Gigabytes",
26
+ terabytes: "Terabytes",
27
+ bits: "Bits",
28
+ kilobits: "Kilobits",
29
+ megabits: "Megabits",
30
+ gigabits: "Gigabits",
31
+ terabits: "Terabits",
32
+ percent: "Percent",
33
+ count: "Count"
34
+ }.freeze
35
+
36
+ def initialize(namespace_formatter: nil, metric_name_formatter: nil, dimension_name_formatter: nil, allowlist: nil)
37
+ @namespace_formatter = namespace_formatter
38
+ @metric_name_formatter = metric_name_formatter
39
+ @dimension_name_formatter = dimension_name_formatter
40
+ @allowlist = normalize_allowlist(allowlist)
41
+
42
+ Speedshop::Cloudwatch.configure do |config|
43
+ config.collectors << :yabeda unless config.collectors.include?(:yabeda)
44
+ end
45
+ end
46
+
47
+ # CloudWatch doesn't require pre-registration of metrics, but Yabeda's adapter
48
+ # interface expects these hooks to exist for backends that do.
49
+ def register_counter!(_metric)
50
+ end
51
+
52
+ def register_gauge!(_metric)
53
+ end
54
+
55
+ def register_histogram!(_metric)
56
+ end
57
+
58
+ def register_summary!(_metric)
59
+ end
60
+
61
+ def perform_counter_increment!(counter, tags, increment)
62
+ enqueue_metric(counter, tags, increment)
63
+ end
64
+
65
+ def perform_gauge_set!(gauge, tags, value)
66
+ enqueue_metric(gauge, tags, value)
67
+ end
68
+
69
+ def perform_histogram_measure!(histogram, tags, value)
70
+ enqueue_metric(histogram, tags, value)
71
+ end
72
+
73
+ def perform_summary_observe!(summary, tags, value)
74
+ enqueue_metric(summary, tags, value)
75
+ end
76
+
77
+ class Collector
78
+ def collect
79
+ ::Yabeda.collect!
80
+ end
81
+ end
82
+
83
+ private
84
+
85
+ def enqueue_metric(metric, tags, value)
86
+ namespace = namespace_for(metric)
87
+ metric_name = metric_name_for(metric)
88
+ return unless allowed?(namespace, metric_name)
89
+
90
+ Reporter.instance.enqueue(
91
+ metric_name: metric_name,
92
+ namespace: namespace,
93
+ unit: unit_for(metric),
94
+ dimensions: dimensions_for(tags),
95
+ value: value,
96
+ timestamp: Time.now,
97
+ aggregation_strategy: aggregation_strategy_for(metric)
98
+ )
99
+ end
100
+
101
+ def namespace_for(metric)
102
+ format_value(metric.group.to_s.split("_").map(&:capitalize).join, @namespace_formatter, metric)
103
+ end
104
+
105
+ def metric_name_for(metric)
106
+ format_value(metric.name.to_s, @metric_name_formatter, metric)
107
+ end
108
+
109
+ def unit_for(metric)
110
+ UNIT_MAP.fetch(metric.unit&.to_sym, "None")
111
+ end
112
+
113
+ def dimensions_for(tags)
114
+ tag_dimensions = tags.to_h.map do |name, value|
115
+ {name: format_value(name.to_s, @dimension_name_formatter), value: value.to_s}
116
+ end
117
+ tag_dimensions + Config.instance.dimensions.map do |name, value|
118
+ {name: format_value(name.to_s, @dimension_name_formatter), value: value.to_s}
119
+ end
120
+ end
121
+
122
+ def aggregation_strategy_for(metric)
123
+ return unless metric.is_a?(::Yabeda::Gauge)
124
+
125
+ strategy = metric.aggregation&.to_sym
126
+ strategy if %i[most_recent max].include?(strategy)
127
+ end
128
+
129
+ def allowed?(namespace, metric_name)
130
+ return true unless @allowlist
131
+
132
+ Array(@allowlist[namespace]).map(&:to_s).include?(metric_name.to_s)
133
+ end
134
+
135
+ def normalize_allowlist(allowlist)
136
+ return unless allowlist
137
+
138
+ allowlist.each_with_object({}) do |(namespace, metrics), normalized|
139
+ normalized[namespace.to_s] = Array(metrics).map(&:to_s)
140
+ end
141
+ end
142
+
143
+ def format_value(value, formatter, metric = nil)
144
+ return value unless formatter
145
+
146
+ (formatter.arity == 1) ? formatter.call(value) : formatter.call(value, metric)
147
+ end
148
+ end
149
+ end
150
+ end
@@ -3,6 +3,7 @@
3
3
  require "aws-sdk-cloudwatch"
4
4
  require "speedshop/cloudwatch/config"
5
5
  require "speedshop/cloudwatch/reporter"
6
+ require "speedshop/cloudwatch/metric_mapper"
6
7
  require "speedshop/cloudwatch/version"
7
8
 
8
9
  module Speedshop
@@ -23,6 +24,10 @@ module Speedshop
23
24
  Reporter.instance
24
25
  end
25
26
 
27
+ def metric_mapper
28
+ MetricMapper.instance
29
+ end
30
+
26
31
  def start!
27
32
  reporter.start!
28
33
  end
@@ -21,10 +21,13 @@ Gem::Specification.new do |spec|
21
21
  spec.files = IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL) do |ls|
22
22
  ls.readlines("\x0", chomp: true).select do |path|
23
23
  path.start_with?("lib/", "bin/", "docs/") ||
24
- %w[README.md LICENSE.txt speedshop-cloudwatch.gemspec Rakefile].include?(path)
24
+ %w[README.md CHANGELOG.md LICENSE.txt speedshop-cloudwatch.gemspec Rakefile].include?(path)
25
25
  end
26
26
  end
27
27
  spec.require_paths = ["lib"]
28
28
 
29
29
  spec.add_dependency "aws-sdk-cloudwatch", ">= 1.81.0"
30
+
31
+ spec.add_development_dependency "csv"
32
+ spec.add_development_dependency "yabeda"
30
33
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: speedshop-cloudwatch
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Nate Berkopec
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2025-11-17 00:00:00.000000000 Z
11
+ date: 2026-04-13 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: aws-sdk-cloudwatch
@@ -24,6 +24,34 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: 1.81.0
27
+ - !ruby/object:Gem::Dependency
28
+ name: csv
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: yabeda
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
27
55
  description: This gem helps integrate your Ruby application with AWS CloudWatch, reporting
28
56
  metrics from Puma, Rack, Sidekiq, and ActiveJob in background threads to avoid adding
29
57
  latency to requests and jobs.
@@ -33,6 +61,7 @@ executables: []
33
61
  extensions: []
34
62
  extra_rdoc_files: []
35
63
  files:
64
+ - CHANGELOG.md
36
65
  - LICENSE.txt
37
66
  - README.md
38
67
  - Rakefile
@@ -43,13 +72,20 @@ files:
43
72
  - lib/speedshop/cloudwatch/active_job.rb
44
73
  - lib/speedshop/cloudwatch/all.rb
45
74
  - lib/speedshop/cloudwatch/config.rb
75
+ - lib/speedshop/cloudwatch/metric_mapper.rb
46
76
  - lib/speedshop/cloudwatch/metrics.rb
77
+ - lib/speedshop/cloudwatch/observations.rb
78
+ - lib/speedshop/cloudwatch/observations/active_job.rb
79
+ - lib/speedshop/cloudwatch/observations/puma.rb
80
+ - lib/speedshop/cloudwatch/observations/rack.rb
81
+ - lib/speedshop/cloudwatch/observations/sidekiq.rb
47
82
  - lib/speedshop/cloudwatch/puma.rb
48
83
  - lib/speedshop/cloudwatch/rack.rb
49
84
  - lib/speedshop/cloudwatch/railtie.rb
50
85
  - lib/speedshop/cloudwatch/reporter.rb
51
86
  - lib/speedshop/cloudwatch/sidekiq.rb
52
87
  - lib/speedshop/cloudwatch/version.rb
88
+ - lib/speedshop/cloudwatch/yabeda.rb
53
89
  - speedshop-cloudwatch.gemspec
54
90
  homepage: https://github.com/nateberkopec/speedshop-cloudwatch
55
91
  licenses: