yabeda-sidekiq 0.8.2 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +23 -0
  3. data/README.md +13 -2
  4. data/lib/yabeda/sidekiq/client_middleware.rb +7 -1
  5. data/lib/yabeda/sidekiq/config.rb +3 -0
  6. data/lib/yabeda/sidekiq/version.rb +1 -1
  7. data/lib/yabeda/sidekiq.rb +10 -8
  8. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: '08af80462b44bb1631a4e7103354bf2cd3613807b629d282d31913ec8c2dfc75'
4
- data.tar.gz: 60634a95cc22cbeaff73e0986651956e5348315cb42077f4c6a86447dfd45acc
3
+ metadata.gz: 6d63d31846975b74ec0fc5a7209b8b405065c52ff49074bcad1f36d92bb97e36
4
+ data.tar.gz: 3bd52a1b15a38d2fb80df6e54d9b702311e9311f37b04f5622e6c2b92411b768
5
5
  SHA512:
6
- metadata.gz: 6f880ce4e139f01b07d375e62e252fa990d198e09838aa9d2a3b3d348984adda38debe8ddb33ccc8dd514a2f502f89c27f5acdfafe320479d59a397ac7e56000
7
- data.tar.gz: 29c28c92077f5b3edbd1e844c6ddcba0bb4de99ca164dcfadca30a1c2369979a2cc3ac51192ee700b8d71fc82d68372fc4b12cf00521ae0ef822b5c08d2b9cb0
6
+ metadata.gz: '05919789943b693dcc75c7b3b50a6594b924f4703affdffac32720fae69b8cb1de0e3a9818301c16edc8d7b08c57e6431474ebce66d81fa5ca5a33805846ea46'
7
+ data.tar.gz: 17699edb8dc44c6ee0a7a89c0488253af8dbbb978f012e8d5c828c180e59644c82092f03dad5c517cf973c79853d167ec56c1da266b0e2de6b1c555b7ce05876
data/CHANGELOG.md CHANGED
@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
7
7
 
8
8
  ## Unreleased
9
9
 
10
+ ## 0.10.0 - 2022-10-25
11
+
12
+ ### Added
13
+
14
+ - New metric `sidekiq_jobs_rerouted_total_count` to measure jobs that on enqueue were pushed to different queue from the one specified in worker's `sidekiq_options`. See [#30](https://github.com/yabeda-rb/yabeda-sidekiq/pull/30). [@LukinEgor]
15
+
16
+ ### Fixed
17
+
18
+ - In `sidekiq_jobs_enqueued_total_count` track real queue that job was pushed into, not the one specified in `sidekiq_options` (sometimes they may be different). See [#30](https://github.com/yabeda-rb/yabeda-sidekiq/pull/30). [@LukinEgor]
19
+
20
+ ## 0.9.0 - 2022-09-26
21
+
22
+ ### Added
23
+
24
+ - Configuration setting to declare worker in-process metrics outside workers.
25
+
26
+ It can be needed for official Prometheus client in multi-process mode where separate process expose metrics from Sidekiq worker processes.
27
+
28
+ - `most_recent` aggregation for all cluster-wide gauges.
29
+
30
+ It is also needed for official Prometheus client in multi-process mode to reduce number of time series.
31
+
10
32
  ## 0.8.2 - 2022-09-14
11
33
 
12
34
  ### Added
@@ -96,3 +118,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
96
118
  [@dsalahutdinov]: https://github.com/dsalahutdinov "Salahutdinov Dmitry"
97
119
  [@asusikov]: https://github.com/asusikov "Alexander Susikov"
98
120
  [@mrexox]: https://github.com/mrexox "Valentine Kiselev"
121
+ [@LukinEgor]: https://github.com/LukinEgor "Egor Lukin"
data/README.md CHANGED
@@ -62,6 +62,16 @@ Metrics representing state of the whole Sidekiq installation (queues, processes,
62
62
 
63
63
  By default all sidekiq worker processes (servers) collects global metrics about whole Sidekiq installation. This can be overridden by setting `collect_cluster_metrics` config key to `true` for non-Sidekiq processes or to `false` for Sidekiq processes (e.g. by setting `YABEDA_SIDEKIQ_COLLECT_CLUSTER_METRICS` env variable to `no`, see other methods in [anyway_config] docs).
64
64
 
65
+ ### Client metrics
66
+
67
+ Metrics collected where jobs are being pushed to queues (everywhere):
68
+
69
+ - Total number of enqueued jobs: `sidekiq_jobs_enqueued_total_count` (segmented by `queue` and `worker` class name)
70
+
71
+ - Total number of rerouted jobs: `sidekiq_jobs_rerouted_total_count` (segmented by origin queue `from_queue`, rerouted queue `to_queue`, and `worker` class name).
72
+
73
+ Rerouted jobs are jobs that on enqueue were pushed to different queue from the one specified in worker's `sidekiq_options`, most probably by some middleware.
74
+
65
75
  ## Custom tags
66
76
 
67
77
  You can add additional tags to these metrics by declaring `yabeda_tags` method in your worker.
@@ -89,9 +99,10 @@ end
89
99
 
90
100
  Configuration is handled by [anyway_config] gem. With it you can load settings from environment variables (upcased and prefixed with `YABEDA_SIDEKIQ_`), YAML files, and other sources. See [anyway_config] docs for details.
91
101
 
92
- Config key | Type | Default | Description |
93
- ------------------------- | -------- | ------------------------------------------------------- | ----------- |
102
+ Config key | Type | Default | Description |
103
+ ------------------------- | -------- | ------------------------------------------------------- |----------------------------------------------------------------------------------------------------------------------------------------------------|
94
104
  `collect_cluster_metrics` | boolean | Enabled in Sidekiq worker processes, disabled otherwise | Defines whether this Ruby process should collect and expose metrics representing state of the whole Sidekiq installation (queues, processes, etc). |
105
+ `declare_process_metrics` | boolean | Enabled in Sidekiq worker processes, disabled otherwise | Declare metrics that are only tracked inside worker process even outside of them. Useful for multiprocess metric collection. |
95
106
 
96
107
  # Roadmap (TODO or Help wanted)
97
108
 
data/lib/yabeda/sidekiq/client_middleware.rb CHANGED
@@ -5,8 +5,14 @@ module Yabeda
5
5
  # Client middleware to count number of enqueued jobs
6
6
  class ClientMiddleware
7
7
  def call(worker, job, queue, _redis_pool)
8
- labels = Yabeda::Sidekiq.labelize(worker, job, queue)
8
+ labels = Yabeda::Sidekiq.labelize(worker, job, job["queue"] || queue)
9
9
  Yabeda.sidekiq_jobs_enqueued_total.increment(labels)
10
+
11
+ if job["queue"] && job["queue"] != queue
12
+ labels = Yabeda::Sidekiq.labelize(worker, job, queue)
13
+ Yabeda.sidekiq_jobs_rerouted_total.increment({ from_queue: queue, to_queue: job["queue"], **labels.except(:queue) })
14
+ end
15
+
10
16
  yield
11
17
  end
12
18
  end
data/lib/yabeda/sidekiq/config.rb CHANGED
@@ -13,6 +13,9 @@ module Yabeda
13
13
  # - force disable if you don't want multiple Sidekiq workers to report the same numbers (that causes excess load to both Redis and monitoring)
14
14
  # - force enable if you want non-Sidekiq process to collect them (like dedicated metric exporter process)
15
15
  attr_config collect_cluster_metrics: ::Sidekiq.server?
16
+
17
+ # Declare metrics that are only tracked inside worker process even outside them
18
+ attr_config declare_process_metrics: ::Sidekiq.server?
16
19
  end
17
20
  end
18
21
  end
data/lib/yabeda/sidekiq/version.rb CHANGED
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Yabeda
4
4
  module Sidekiq
5
- VERSION = "0.8.2"
5
+ VERSION = "0.10.0"
6
6
  end
7
7
  end
data/lib/yabeda/sidekiq.rb CHANGED
@@ -26,8 +26,9 @@ module Yabeda
26
26
  group :sidekiq
27
27
 
28
28
  counter :jobs_enqueued_total, tags: %i[queue worker], comment: "A counter of the total number of jobs sidekiq enqueued."
29
+ counter :jobs_rerouted_total, tags: %i[from_queue to_queue worker], comment: "A counter of the total number of rerouted jobs sidekiq enqueued."
29
30
 
30
- if ::Sidekiq.server?
31
+ if config.declare_process_metrics # defaults to +::Sidekiq.server?+
31
32
  counter :jobs_executed_total, tags: %i[queue worker], comment: "A counter of the total number of jobs sidekiq executed."
32
33
  counter :jobs_success_total, tags: %i[queue worker], comment: "A counter of the total number of jobs successfully processed by sidekiq."
33
34
  counter :jobs_failed_total, tags: %i[queue worker], comment: "A counter of the total number of jobs failed in sidekiq."
@@ -48,13 +49,14 @@ module Yabeda
48
49
  # Metrics not specific for current Sidekiq process, but representing state of the whole Sidekiq installation (queues, processes, etc)
49
50
  # You can opt-out from collecting these by setting YABEDA_SIDEKIQ_COLLECT_CLUSTER_METRICS to falsy value (+no+ or +false+)
50
51
  if config.collect_cluster_metrics # defaults to +::Sidekiq.server?+
51
- gauge :jobs_waiting_count, tags: %i[queue], comment: "The number of jobs waiting to process in sidekiq."
52
- gauge :active_workers_count, tags: [], comment: "The number of currently running machines with sidekiq workers."
53
- gauge :jobs_scheduled_count, tags: [], comment: "The number of jobs scheduled for later execution."
54
- gauge :jobs_retry_count, tags: [], comment: "The number of failed jobs waiting to be retried"
55
- gauge :jobs_dead_count, tags: [], comment: "The number of jobs exceeded their retry count."
56
- gauge :active_processes, tags: [], comment: "The number of active Sidekiq worker processes."
57
- gauge :queue_latency, tags: %i[queue], comment: "The queue latency, the difference in seconds since the oldest job in the queue was enqueued"
52
+ gauge :jobs_waiting_count, tags: %i[queue], aggregation: :most_recent, comment: "The number of jobs waiting to process in sidekiq."
53
+ gauge :active_workers_count, tags: [], aggregation: :most_recent, comment: "The number of currently running machines with sidekiq workers."
54
+ gauge :jobs_scheduled_count, tags: [], aggregation: :most_recent, comment: "The number of jobs scheduled for later execution."
55
+ gauge :jobs_retry_count, tags: [], aggregation: :most_recent, comment: "The number of failed jobs waiting to be retried"
56
+ gauge :jobs_dead_count, tags: [], aggregation: :most_recent, comment: "The number of jobs exceeded their retry count."
57
+ gauge :active_processes, tags: [], aggregation: :most_recent, comment: "The number of active Sidekiq worker processes."
58
+ gauge :queue_latency, tags: %i[queue], aggregation: :most_recent,
59
+ comment: "The queue latency, the difference in seconds since the oldest job in the queue was enqueued"
58
60
  end
59
61
 
60
62
  collect do
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: yabeda-sidekiq
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.8.2
4
+ version: 0.10.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrey Novikov
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-09-14 00:00:00.000000000 Z
11
+ date: 2022-10-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: anyway_config