rails_autoscale_agent 0.9.1 → 0.10.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +109 -0
- data/README.md +27 -1
- data/lib/rails_autoscale_agent/autoscale_api.rb +1 -1
- data/lib/rails_autoscale_agent/config.rb +21 -13
- data/lib/rails_autoscale_agent/registration.rb +1 -1
- data/lib/rails_autoscale_agent/report.rb +1 -1
- data/lib/rails_autoscale_agent/reporter.rb +25 -8
- data/lib/rails_autoscale_agent/request.rb +25 -16
- data/lib/rails_autoscale_agent/store.rb +5 -0
- data/lib/rails_autoscale_agent/version.rb +1 -1
- data/lib/rails_autoscale_agent/worker_adapters/delayed_job.rb +52 -12
- data/lib/rails_autoscale_agent/worker_adapters/que.rb +15 -1
- data/lib/rails_autoscale_agent/worker_adapters/resque.rb +8 -1
- data/lib/rails_autoscale_agent/worker_adapters/sidekiq.rb +39 -9
- data/rails_autoscale_agent.gemspec +2 -0
- metadata +9 -8
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ed5e9567b63e205a9477e80cb3b1cb5d0d198f60288957c8560eb988f0ab5322
|
4
|
+
data.tar.gz: 0ff58213c0347a71218c668fbcea5f3bae8e6e24701b7c9ce3317d84dfce8ef7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3da81efface83a93ac9c3676b7edbcaf431911cfe428c054a39203663048bc7a320ebd6881606d7b18e563c145d8fcfc1cd6d3acf945bff82525195796f1be96
|
7
|
+
data.tar.gz: a4012e4ab5876cddb28bfea34a2b71d6239c34dc5b167cb54616e5635a67d36915245c2b96bc536256ee7109640db6a035d0b5b99bfa20fab76a88900aba6f3f
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,109 @@
|
|
1
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
2
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
3
|
+
|
4
|
+
## [Unreleased](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.10.1...master)
|
5
|
+
|
6
|
+
_There are no currently unreleased changes._
|
7
|
+
|
8
|
+
## [0.10.1](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.9.1...v0.10.1) - 2021-01-03
|
9
|
+
|
10
|
+
### Added
|
11
|
+
|
12
|
+
- Add support for [long-running jobs](https://railsautoscale.com/docs/long-running-jobs/) in Sidekiq and Delayed Job.
|
13
|
+
- Handle x-request-start measured in seconds (instead of milliseconds) to support nginx buildpack ([cd092f3](https://github.com/adamlogic/rails_autoscale_agent/commit/cd092f38718abf5ffaea866bcae7831d4c910ffd))
|
14
|
+
- Override worker adapter config via env var ([75dd06b](https://github.com/adamlogic/rails_autoscale_agent/commit/75dd06b2a7ff4eeab829eec24d503dc067c8fe32))
|
15
|
+
|
16
|
+
### Changed
|
17
|
+
|
18
|
+
- Require Ruby 2.5 or newer. ([b033050](https://github.com/adamlogic/rails_autoscale_agent/commit/b033050b7f9d4d7f1e50dbd780cf0e1822249268))
|
19
|
+
- Only report worker metrics from web.1 to avoid redundant data. ([d5d5fa8](https://github.com/adamlogic/rails_autoscale_agent/commit/d5d5fa87fb4d7d046832a64edde9ed0c3a6ec75f))
|
20
|
+
- Don't collect worker metrics for an unreasonable number of queues. ([a9358af](https://github.com/adamlogic/rails_autoscale_agent/commit/a9358af74a29a941d1f1d60a0222077dafd5ce08))
|
21
|
+
|
22
|
+
### Fixed
|
23
|
+
|
24
|
+
- Avoid holding onto database connections (DJ & Que only). ([3919ca5](https://github.com/adamlogic/rails_autoscale_agent/commit/3919ca54420cafa82abf9f8cd251569f9637482b))
|
25
|
+
- Better error handling for worker adapters. ([190786e](https://github.com/adamlogic/rails_autoscale_agent/commit/190786e4a910d41e394a3129aac1d23b594dbd9b))
|
26
|
+
- Don't collect metrics of the reporter isn't running. Avoids memory bloat. ([247c322](https://github.com/adamlogic/rails_autoscale_agent/commit/247c322cffc625a8c6b2395080a048ffb94e7f3b))
|
27
|
+
|
28
|
+
## [0.10.0](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.9.1...v0.10.0) - 2021-01-03 [YANKED]
|
29
|
+
|
30
|
+
_I released the wrong branch 🤦♂️_
|
31
|
+
|
32
|
+
## [0.9.1](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.9.0...v0.9.1) - 2020-07-29
|
33
|
+
|
34
|
+
### Fixed
|
35
|
+
|
36
|
+
- Fix a bug in error handling. ([3018542](https://github.com/adamlogic/rails_autoscale_agent/commit/3018542cd046fc4e1bd6e7da86e72a6aa2d50a8f))
|
37
|
+
- Remove unintentional Rails dependency.
|
38
|
+
|
39
|
+
## [0.9.0](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.8.3...v0.9.0) - 2020-07-12
|
40
|
+
|
41
|
+
### Added
|
42
|
+
|
43
|
+
- Add support for Resque workers.
|
44
|
+
- Add dev mode for working on the agent gem itself. ([47e3fca](https://github.com/adamlogic/rails_autoscale_agent/commit/47e3fca5b788f48567a345d9cab3a26b9cd87693))
|
45
|
+
- Report agent exceptions to Rails Autoscale.
|
46
|
+
|
47
|
+
### Changed
|
48
|
+
|
49
|
+
- Adjust queue time metric to exclude time waiting for large request bodies. ([#25](https://github.com/adamlogic/rails_autoscale_agent/pull/25))
|
50
|
+
- Que and DJ jobs without a queue name will be included in the "default" queue metrics.
|
51
|
+
|
52
|
+
### Fixed
|
53
|
+
|
54
|
+
- Multiple fixes to the Delayed Job SQL query.
|
55
|
+
|
56
|
+
## [0.8.3](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.8.2...v0.8.3) - 2020-05-26
|
57
|
+
|
58
|
+
### Fixed
|
59
|
+
|
60
|
+
- Ignored failed job in Delayed Job adapter. ([fa72fc2](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.8.2...v0.8.3))
|
61
|
+
|
62
|
+
## [0.8.2](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.8.1...v0.8.2) - 2020-05-22
|
63
|
+
|
64
|
+
### Fixed
|
65
|
+
|
66
|
+
- Ignore worker metrics from unnamed queues (DJ & Que only). These metrics were being lumped with web metrics. ([#21](https://github.com/adamlogic/rails_autoscale_agent/pull/21))
|
67
|
+
|
68
|
+
## [0.8.1](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.8.0...v0.8.1) - 2020-05-04
|
69
|
+
|
70
|
+
### Fixed
|
71
|
+
|
72
|
+
- Ignore failed jobs in Que adapter. ([#18](https://github.com/adamlogic/rails_autoscale_agent/pull/18))
|
73
|
+
|
74
|
+
## [0.8.0](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.7.0...v0.8.0) - 2020-03-21
|
75
|
+
|
76
|
+
### Added
|
77
|
+
|
78
|
+
- Add support for Delayed Job ([#14](https://github.com/adamlogic/rails_autoscale_agent/pull/14))
|
79
|
+
- Add support for Que ([#15](https://github.com/adamlogic/rails_autoscale_agent/pull/15))
|
80
|
+
|
81
|
+
## [0.7.0](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.6.3...v0.7.0) - 2019-12-04
|
82
|
+
|
83
|
+
### Added
|
84
|
+
|
85
|
+
- Make worker adapters configurable. ([012d937](https://github.com/adamlogic/rails_autoscale_agent/commit/012d9379296763f5e42df95f05b066fe82ab0051))
|
86
|
+
|
87
|
+
## [0.6.3](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.6.2...v0.6.3) - 2019-06-25
|
88
|
+
|
89
|
+
### Fixed
|
90
|
+
|
91
|
+
- Fix issues with logging.
|
92
|
+
|
93
|
+
## [0.6.2](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.6.1...v0.6.2) - 2019-06-22
|
94
|
+
|
95
|
+
### Fixed
|
96
|
+
|
97
|
+
- Fix issues with logging.
|
98
|
+
|
99
|
+
## [0.6.1](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.6.0...v0.6.1) - 2019-05-06
|
100
|
+
|
101
|
+
### Fixed
|
102
|
+
|
103
|
+
- Don't assume Sidekiq is present.
|
104
|
+
|
105
|
+
## [0.6.0](https://github.com/adamlogic/rails_autoscale_agent/compare/v0.4.1...v0.6.0) - 2019-05-03
|
106
|
+
|
107
|
+
### Added
|
108
|
+
|
109
|
+
- Add support for autoscaling Sidekiq.
|
data/README.md
CHANGED
@@ -22,7 +22,7 @@ The agent will only communicate with Rails Autoscale if a `RAILS_AUTOSCALE_URL`
|
|
22
22
|
|
23
23
|
## Non-Rails Rack apps
|
24
24
|
|
25
|
-
You'll need to insert the `RailsAutoscaleAgent::Middleware` manually. Insert it before `Rack::Runtime` to ensure accuracy of request queue timings.
|
25
|
+
You'll need to `require 'rails_autoscale_agent/middleware'` and insert the `RailsAutoscaleAgent::Middleware` manually. Insert it before `Rack::Runtime` to ensure accuracy of request queue timings.
|
26
26
|
|
27
27
|
## What data is collected?
|
28
28
|
|
@@ -37,6 +37,32 @@ The middleware agent runs in its own thread so your web requests are not impacte
|
|
37
37
|
|
38
38
|
Rails Autoscale aggregates and stores this information to power the autoscaling algorithm and dashboard visualizations.
|
39
39
|
|
40
|
+
## Configuration
|
41
|
+
|
42
|
+
Most Rails Autoscale configurations are handled via the settings page on your Rails Autoscale dashboard, but there a few ways you can directly change the behavior of the agent via environment variables:
|
43
|
+
|
44
|
+
- `RAILS_AUTOSCALE_DEBUG` - Enables debug logging. See more in the [logging](#logging) section below.
|
45
|
+
- `RAILS_AUTOSCALE_WORKER_ADAPTER` - Overrides the available worker adapters. See more in the [worker adapters](#worker_adapters) section below.
|
46
|
+
- `RAILS_AUTOSCALE_LONG_JOBS` - Enables reporting for active workers. See [Handling Long-Running Background Jobs](https://railsautoscale.com/docs/long-running-jobs/) in the Rails Autoscale docs for more.
|
47
|
+
|
48
|
+
## Worker adapters
|
49
|
+
|
50
|
+
Rails Autoscale supports autoscaling worker dynos. Out of the box, four job backends are supported: Sidekiq, Resque, Delayed Job, and Que. The agent will automatically enable the appropriate worker adapter based on what you have installed in your app.
|
51
|
+
|
52
|
+
In some scenarios you might want to override this behavior. Let's say you have both Sidekiq and Resque installed 🤷♂️, but you only want Rails Autoscale to collect metrics for Sidekiq. Here's how you'd override that:
|
53
|
+
|
54
|
+
```
|
55
|
+
heroku config:add RAILS_AUTOSCALE_WORKER_ADAPTER=sidekiq
|
56
|
+
```
|
57
|
+
|
58
|
+
You can also disable collection of worker metrics altogether:
|
59
|
+
|
60
|
+
```
|
61
|
+
heroku config:add RAILS_AUTOSCALE_WORKER_ADAPTER=""
|
62
|
+
```
|
63
|
+
|
64
|
+
It's also possible to write a custom worker adapter. See [these docs](https://railsautoscale.com/docs/custom-worker-adapter/) for details.
|
65
|
+
|
40
66
|
## Troubleshooting
|
41
67
|
|
42
68
|
Once installed, you should see something like this in your development log:
|
@@ -4,29 +4,26 @@ require 'singleton'
|
|
4
4
|
|
5
5
|
module RailsAutoscaleAgent
|
6
6
|
class Config
|
7
|
+
DEFAULT_WORKER_ADAPTERS = 'sidekiq,delayed_job,que,resque'
|
8
|
+
|
7
9
|
include Singleton
|
8
10
|
|
9
11
|
attr_accessor :report_interval, :logger, :api_base_url, :max_request_size,
|
10
|
-
:dyno, :
|
12
|
+
:dyno, :addon_name, :worker_adapters, :dev_mode, :debug, :quiet,
|
13
|
+
:track_long_running_jobs,
|
14
|
+
|
15
|
+
# legacy configs, no longer used
|
16
|
+
:sidekiq_latency_for_active_jobs, :latency_for_active_jobs
|
11
17
|
|
12
18
|
def initialize
|
13
|
-
|
14
|
-
require 'rails_autoscale_agent/worker_adapters/delayed_job'
|
15
|
-
require 'rails_autoscale_agent/worker_adapters/que'
|
16
|
-
require 'rails_autoscale_agent/worker_adapters/resque'
|
17
|
-
@worker_adapters = [
|
18
|
-
WorkerAdapters::Sidekiq.instance,
|
19
|
-
WorkerAdapters::DelayedJob.instance,
|
20
|
-
WorkerAdapters::Que.instance,
|
21
|
-
WorkerAdapters::Resque.instance,
|
22
|
-
]
|
19
|
+
@worker_adapters = prepare_worker_adapters
|
23
20
|
|
24
21
|
# Allow the add-on name to be configured - needed for testing
|
25
22
|
@addon_name = ENV['RAILS_AUTOSCALE_ADDON'] || 'RAILS_AUTOSCALE'
|
26
23
|
@api_base_url = ENV["#{@addon_name}_URL"]
|
27
24
|
@dev_mode = ENV['RAILS_AUTOSCALE_DEV'] == 'true'
|
28
25
|
@debug = dev_mode? || ENV['RAILS_AUTOSCALE_DEBUG'] == 'true'
|
29
|
-
@
|
26
|
+
@track_long_running_jobs = ENV['RAILS_AUTOSCALE_LONG_JOBS'] == 'true'
|
30
27
|
@max_request_size = 100_000 # ignore request payloads over 100k since they skew the queue times
|
31
28
|
@report_interval = 10 # this default will be overwritten during Reporter#register!
|
32
29
|
@logger ||= defined?(Rails) ? Rails.logger : ::Logger.new(STDOUT)
|
@@ -34,7 +31,7 @@ module RailsAutoscaleAgent
|
|
34
31
|
end
|
35
32
|
|
36
33
|
def to_s
|
37
|
-
"#{@dyno}##{
|
34
|
+
"#{@dyno}##{Process.pid}"
|
38
35
|
end
|
39
36
|
|
40
37
|
def ignore_large_requests?
|
@@ -44,5 +41,16 @@ module RailsAutoscaleAgent
|
|
44
41
|
alias_method :dev_mode?, :dev_mode
|
45
42
|
alias_method :debug?, :debug
|
46
43
|
alias_method :quiet?, :quiet
|
44
|
+
|
45
|
+
private
|
46
|
+
|
47
|
+
def prepare_worker_adapters
|
48
|
+
adapter_names = (ENV['RAILS_AUTOSCALE_WORKER_ADAPTER'] || DEFAULT_WORKER_ADAPTERS).split(',')
|
49
|
+
adapter_names.map do |adapter_name|
|
50
|
+
require "rails_autoscale_agent/worker_adapters/#{adapter_name}"
|
51
|
+
adapter_constant_name = adapter_name.capitalize.gsub(/(?:_)(.)/i) { $1.upcase }
|
52
|
+
WorkerAdapters.const_get(adapter_constant_name).instance
|
53
|
+
end
|
54
|
+
end
|
47
55
|
end
|
48
56
|
end
|
@@ -20,6 +20,7 @@ module RailsAutoscaleAgent
|
|
20
20
|
def start!(config, store)
|
21
21
|
@started = true
|
22
22
|
@worker_adapters = config.worker_adapters.select(&:enabled?)
|
23
|
+
@dyno_num = config.dyno.to_s.split('.').last.to_i
|
23
24
|
|
24
25
|
if !config.api_base_url && !config.dev_mode?
|
25
26
|
logger.info "Reporter not started: #{config.addon_name}_URL is not set"
|
@@ -34,15 +35,14 @@ module RailsAutoscaleAgent
|
|
34
35
|
multiplier = 1 - (rand / 4) # between 0.75 and 1.0
|
35
36
|
sleep config.report_interval * multiplier
|
36
37
|
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
# https://ruby-doc.org/core-2.2.0/Thread.html#class-Thread-label-Exception+handling
|
43
|
-
logger.error "Reporter error: #{ex.inspect}"
|
44
|
-
AutoscaleApi.new(config).report_exception!(ex)
|
38
|
+
# It's redundant to report worker metrics from every web dyno, so only report from web.1
|
39
|
+
if @dyno_num == 1
|
40
|
+
@worker_adapters.map do |adapter|
|
41
|
+
report_exceptions(config) { adapter.collect!(store) }
|
42
|
+
end
|
45
43
|
end
|
44
|
+
|
45
|
+
report_exceptions(config) { report!(config, store) }
|
46
46
|
end
|
47
47
|
end
|
48
48
|
end
|
@@ -51,6 +51,8 @@ module RailsAutoscaleAgent
|
|
51
51
|
@started
|
52
52
|
end
|
53
53
|
|
54
|
+
private
|
55
|
+
|
54
56
|
def report!(config, store)
|
55
57
|
report = store.pop_report
|
56
58
|
|
@@ -86,5 +88,20 @@ module RailsAutoscaleAgent
|
|
86
88
|
logger.error "Reporter failed to register: #{result.failure_message}"
|
87
89
|
end
|
88
90
|
end
|
91
|
+
|
92
|
+
def report_exceptions(config)
|
93
|
+
begin
|
94
|
+
yield
|
95
|
+
rescue => ex
|
96
|
+
# Exceptions in threads other than the main thread will fail silently
|
97
|
+
# https://ruby-doc.org/core-2.2.0/Thread.html#class-Thread-label-Exception+handling
|
98
|
+
logger.error "Reporter error: #{ex.inspect}"
|
99
|
+
AutoscaleApi.new(config).report_exception!(ex)
|
100
|
+
end
|
101
|
+
rescue => ex
|
102
|
+
# An exception was encountered while trying to report the original exception.
|
103
|
+
# Swallow the error so the reporter continues to report.
|
104
|
+
logger.error "Exception reporting error: #{ex.inspect}"
|
105
|
+
end
|
89
106
|
end
|
90
107
|
end
|
@@ -1,5 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require 'rails_autoscale_agent/logger'
|
4
|
+
|
3
5
|
module RailsAutoscaleAgent
|
4
6
|
class Request
|
5
7
|
include Logger
|
@@ -9,32 +11,39 @@ module RailsAutoscaleAgent
|
|
9
11
|
@id = env['HTTP_X_REQUEST_ID']
|
10
12
|
@size = env['rack.input'].respond_to?(:size) ? env['rack.input'].size : 0
|
11
13
|
@request_body_wait = env['puma.request_body_wait'].to_i
|
14
|
+
@request_start_header = env['HTTP_X_REQUEST_START']
|
15
|
+
end
|
16
|
+
|
17
|
+
def ignore?
|
18
|
+
@config.ignore_large_requests? && @size > @config.max_request_size
|
19
|
+
end
|
12
20
|
|
13
|
-
|
14
|
-
|
15
|
-
|
21
|
+
def started_at
|
22
|
+
if @request_start_header
|
23
|
+
# Heroku sets the header as an integer, measured in milliseconds.
|
24
|
+
# If nginx is involved, it might be in seconds with fractional milliseconds,
|
25
|
+
# and it might be preceeded by "t=". We can all cases by removing non-digits
|
26
|
+
# and treating as milliseconds.
|
27
|
+
Time.at(@request_start_header.gsub(/\D/, '').to_i / 1000.0)
|
28
|
+
elsif @config.dev_mode?
|
16
29
|
# In dev mode, fake a queue time of 0-1000ms
|
17
30
|
Time.now - rand + @request_body_wait
|
18
31
|
end
|
19
32
|
end
|
20
33
|
|
21
|
-
def
|
22
|
-
|
23
|
-
end
|
34
|
+
def queue_time(now = Time.now)
|
35
|
+
return if started_at.nil?
|
24
36
|
|
25
|
-
|
26
|
-
if @entered_queue_at
|
27
|
-
queue_time = ((Time.now - @entered_queue_at) * 1000).to_i
|
37
|
+
queue_time = ((now - started_at) * 1000).to_i
|
28
38
|
|
29
|
-
|
30
|
-
|
31
|
-
|
39
|
+
# Subtract the time Puma spent waiting on the request body. It's irrelevant to capacity-related queue time.
|
40
|
+
# Without this, slow clients and large request payloads will skew queue time.
|
41
|
+
queue_time -= @request_body_wait
|
32
42
|
|
33
|
-
|
43
|
+
logger.debug "Request queue_time=#{queue_time}ms body_wait=#{@request_body_wait}ms request_id=#{@id} size=#{@size}"
|
34
44
|
|
35
|
-
|
36
|
-
|
37
|
-
end
|
45
|
+
# Safeguard against negative queue times (should not happen in practice)
|
46
|
+
queue_time > 0 ? queue_time : 0
|
38
47
|
end
|
39
48
|
end
|
40
49
|
end
|
@@ -16,10 +16,15 @@ module RailsAutoscaleAgent
|
|
16
16
|
end
|
17
17
|
|
18
18
|
def push(value, time = Time.now, queue_name = nil, metric = nil)
|
19
|
+
# If it's been two minutes since clearing out the store, stop collecting measurements.
|
20
|
+
# There could be an issue with the reporter, and continuing to collect will consume linear memory.
|
21
|
+
return if @last_pop && @last_pop < Time.now - 120
|
22
|
+
|
19
23
|
@measurements << Measurement.new(time, value, queue_name, metric)
|
20
24
|
end
|
21
25
|
|
22
26
|
def pop_report
|
27
|
+
@last_pop = Time.now
|
23
28
|
report = Report.new
|
24
29
|
|
25
30
|
while measurement = @measurements.shift
|
@@ -10,17 +10,11 @@ module RailsAutoscaleAgent
|
|
10
10
|
|
11
11
|
attr_writer :queues
|
12
12
|
|
13
|
-
def queues
|
14
|
-
# Track the known queues so we can continue reporting on queues that don't
|
15
|
-
# have enqueued jobs at the time of reporting.
|
16
|
-
# Assume a "default" queue so we always report *something*, even when nothing
|
17
|
-
# is enqueued.
|
18
|
-
@queues ||= Set.new(['default'])
|
19
|
-
end
|
20
|
-
|
21
13
|
def enabled?
|
22
14
|
if defined?(::Delayed::Job) && defined?(::Delayed::Backend::ActiveRecord)
|
23
|
-
|
15
|
+
log_msg = String.new("DelayedJob enabled (#{::ActiveRecord::Base.default_timezone})")
|
16
|
+
log_msg << " with long-running job support" if track_long_running_jobs?
|
17
|
+
logger.info log_msg
|
24
18
|
true
|
25
19
|
end
|
26
20
|
end
|
@@ -36,8 +30,29 @@ module RailsAutoscaleAgent
|
|
36
30
|
GROUP BY queue
|
37
31
|
SQL
|
38
32
|
|
39
|
-
run_at_by_queue = Hash[
|
40
|
-
|
33
|
+
run_at_by_queue = Hash[select_rows(sql)]
|
34
|
+
|
35
|
+
# Don't collect worker metrics if there are unreasonable number of queues
|
36
|
+
if run_at_by_queue.size > 50
|
37
|
+
logger.debug "Skipping DelayedJob metrics - #{run_at_by_queue.size} queues"
|
38
|
+
return
|
39
|
+
end
|
40
|
+
|
41
|
+
self.queues = queues | run_at_by_queue.keys
|
42
|
+
|
43
|
+
if track_long_running_jobs?
|
44
|
+
sql = <<~SQL
|
45
|
+
SELECT COALESCE(queue, 'default'), count(*)
|
46
|
+
FROM delayed_jobs
|
47
|
+
WHERE locked_at IS NOT NULL
|
48
|
+
AND locked_by IS NOT NULL
|
49
|
+
AND failed_at IS NULL
|
50
|
+
GROUP BY 1
|
51
|
+
SQL
|
52
|
+
|
53
|
+
busy_count_by_queue = Hash[select_rows(sql)]
|
54
|
+
self.queues = queues | busy_count_by_queue.keys
|
55
|
+
end
|
41
56
|
|
42
57
|
queues.each do |queue|
|
43
58
|
run_at = run_at_by_queue[queue]
|
@@ -47,11 +62,36 @@ module RailsAutoscaleAgent
|
|
47
62
|
latency_ms = 0 if latency_ms < 0
|
48
63
|
|
49
64
|
store.push latency_ms, t, queue
|
50
|
-
log_msg << "dj.#{queue}=#{latency_ms} "
|
65
|
+
log_msg << "dj-qt.#{queue}=#{latency_ms} "
|
66
|
+
|
67
|
+
if track_long_running_jobs?
|
68
|
+
busy_count = busy_count_by_queue[queue] || 0
|
69
|
+
store.push busy_count, Time.now, queue, :busy
|
70
|
+
log_msg << "dj-busy.#{queue}=#{busy_count} "
|
71
|
+
end
|
51
72
|
end
|
52
73
|
|
53
74
|
logger.debug log_msg unless log_msg.empty?
|
54
75
|
end
|
76
|
+
|
77
|
+
private
|
78
|
+
|
79
|
+
def queues
|
80
|
+
# Track the known queues so we can continue reporting on queues that don't
|
81
|
+
# have enqueued jobs at the time of reporting.
|
82
|
+
# Assume a "default" queue so we always report *something*, even when nothing
|
83
|
+
# is enqueued.
|
84
|
+
@queues ||= Set.new(['default'])
|
85
|
+
end
|
86
|
+
|
87
|
+
def track_long_running_jobs?
|
88
|
+
Config.instance.track_long_running_jobs
|
89
|
+
end
|
90
|
+
|
91
|
+
def select_rows(sql)
|
92
|
+
# This ensures the agent doesn't hold onto a DB connection any longer than necessary
|
93
|
+
ActiveRecord::Base.connection_pool.with_connection { |c| c.select_rows(sql) }
|
94
|
+
end
|
55
95
|
end
|
56
96
|
end
|
57
97
|
end
|
@@ -37,7 +37,14 @@ module RailsAutoscaleAgent
|
|
37
37
|
GROUP BY 1
|
38
38
|
SQL
|
39
39
|
|
40
|
-
run_at_by_queue = Hash[
|
40
|
+
run_at_by_queue = Hash[select_rows(sql)]
|
41
|
+
|
42
|
+
# Don't collect worker metrics if there are unreasonable number of queues
|
43
|
+
if run_at_by_queue.size > 50
|
44
|
+
logger.debug "Skipping Que metrics - #{run_at_by_queue.size} queues"
|
45
|
+
return
|
46
|
+
end
|
47
|
+
|
41
48
|
self.queues |= run_at_by_queue.keys
|
42
49
|
|
43
50
|
queues.each do |queue|
|
@@ -52,6 +59,13 @@ module RailsAutoscaleAgent
|
|
52
59
|
|
53
60
|
logger.debug log_msg unless log_msg.empty?
|
54
61
|
end
|
62
|
+
|
63
|
+
private
|
64
|
+
|
65
|
+
def select_rows(sql)
|
66
|
+
# This ensures the agent doesn't hold onto a DB connection any longer than necessary
|
67
|
+
ActiveRecord::Base.connection_pool.with_connection { |c| c.select_rows(sql) }
|
68
|
+
end
|
55
69
|
end
|
56
70
|
end
|
57
71
|
end
|
@@ -24,10 +24,17 @@ module RailsAutoscaleAgent
|
|
24
24
|
|
25
25
|
def collect!(store)
|
26
26
|
log_msg = String.new
|
27
|
+
current_queues = ::Resque.queues
|
28
|
+
|
29
|
+
# Don't collect worker metrics if there are unreasonable number of queues
|
30
|
+
if current_queues.size > 50
|
31
|
+
logger.debug "Skipping Resque metrics - #{current_queues.size} queues"
|
32
|
+
return
|
33
|
+
end
|
27
34
|
|
28
35
|
# Ensure we continue to collect metrics for known queue names, even when nothing is
|
29
36
|
# enqueued at the time. Without this, it will appears that the agent is no longer reporting.
|
30
|
-
self.queues |=
|
37
|
+
self.queues |= current_queues
|
31
38
|
|
32
39
|
queues.each do |queue|
|
33
40
|
next if queue.nil? || queue.empty?
|
@@ -8,15 +8,15 @@ module RailsAutoscaleAgent
|
|
8
8
|
include RailsAutoscaleAgent::Logger
|
9
9
|
include Singleton
|
10
10
|
|
11
|
-
attr_writer :
|
12
|
-
|
13
|
-
def known_queue_names
|
14
|
-
@known_queue_names ||= ['default']
|
15
|
-
end
|
11
|
+
attr_writer :queues
|
16
12
|
|
17
13
|
def enabled?
|
18
14
|
require 'sidekiq/api'
|
19
|
-
|
15
|
+
|
16
|
+
log_msg = String.new("Sidekiq enabled")
|
17
|
+
log_msg << " with long-running job support" if track_long_running_jobs?
|
18
|
+
logger.info log_msg
|
19
|
+
|
20
20
|
true
|
21
21
|
rescue LoadError
|
22
22
|
false
|
@@ -28,23 +28,53 @@ module RailsAutoscaleAgent
|
|
28
28
|
obj[queue.name] = queue
|
29
29
|
end
|
30
30
|
|
31
|
+
# Don't collect worker metrics if there are unreasonable number of queues
|
32
|
+
if queues_by_name.size > 50
|
33
|
+
logger.debug "Skipping Sidekiq metrics - #{queues_by_name.size} queues"
|
34
|
+
return
|
35
|
+
end
|
36
|
+
|
31
37
|
# Ensure we continue to collect metrics for known queue names, even when nothing is
|
32
|
-
# enqueued at the time. Without this, it will
|
33
|
-
|
38
|
+
# enqueued at the time. Without this, it will appear that the agent is no longer reporting.
|
39
|
+
queues.each do |queue_name|
|
34
40
|
queues_by_name[queue_name] ||= ::Sidekiq::Queue.new(queue_name)
|
35
41
|
end
|
36
|
-
self.
|
42
|
+
self.queues = queues_by_name.keys
|
43
|
+
|
44
|
+
if track_long_running_jobs?
|
45
|
+
busy_counts = Hash.new { |h,k| h[k] = 0}
|
46
|
+
::Sidekiq::Workers.new.each do |pid, tid, work|
|
47
|
+
busy_counts[work.dig('payload', 'queue')] += 1
|
48
|
+
end
|
49
|
+
end
|
37
50
|
|
38
51
|
queues_by_name.each do |queue_name, queue|
|
39
52
|
latency_ms = (queue.latency * 1000).ceil
|
40
53
|
depth = queue.size
|
54
|
+
|
41
55
|
store.push latency_ms, Time.now, queue.name, :qt
|
42
56
|
store.push depth, Time.now, queue.name, :qd
|
43
57
|
log_msg << "sidekiq-qt.#{queue.name}=#{latency_ms} sidekiq-qd.#{queue.name}=#{depth} "
|
58
|
+
|
59
|
+
if track_long_running_jobs?
|
60
|
+
busy_count = busy_counts[queue.name]
|
61
|
+
store.push busy_count, Time.now, queue.name, :busy
|
62
|
+
log_msg << "sidekiq-busy.#{queue.name}=#{busy_count} "
|
63
|
+
end
|
44
64
|
end
|
45
65
|
|
46
66
|
logger.debug log_msg
|
47
67
|
end
|
68
|
+
|
69
|
+
private
|
70
|
+
|
71
|
+
def queues
|
72
|
+
@queues ||= ['default']
|
73
|
+
end
|
74
|
+
|
75
|
+
def track_long_running_jobs?
|
76
|
+
Config.instance.track_long_running_jobs
|
77
|
+
end
|
48
78
|
end
|
49
79
|
end
|
50
80
|
end
|
metadata
CHANGED
@@ -1,16 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: rails_autoscale_agent
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.10.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Adam McCrea
|
8
|
-
autorequire:
|
8
|
+
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2021-01-03 00:00:00.000000000 Z
|
12
12
|
dependencies: []
|
13
|
-
description:
|
13
|
+
description:
|
14
14
|
email:
|
15
15
|
- adam@adamlogic.com
|
16
16
|
executables: []
|
@@ -22,6 +22,7 @@ files:
|
|
22
22
|
- ".ruby-version"
|
23
23
|
- ".travis.yml"
|
24
24
|
- ".vscode/tasks.json"
|
25
|
+
- CHANGELOG.md
|
25
26
|
- Gemfile
|
26
27
|
- LICENSE.txt
|
27
28
|
- README.md
|
@@ -52,15 +53,15 @@ homepage: https://github.com/adamlogic/rails_autoscale_agent
|
|
52
53
|
licenses:
|
53
54
|
- MIT
|
54
55
|
metadata: {}
|
55
|
-
post_install_message:
|
56
|
+
post_install_message:
|
56
57
|
rdoc_options: []
|
57
58
|
require_paths:
|
58
59
|
- lib
|
59
60
|
required_ruby_version: !ruby/object:Gem::Requirement
|
60
61
|
requirements:
|
61
|
-
- - "
|
62
|
+
- - "~>"
|
62
63
|
- !ruby/object:Gem::Version
|
63
|
-
version: '
|
64
|
+
version: '2.5'
|
64
65
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
65
66
|
requirements:
|
66
67
|
- - ">="
|
@@ -68,7 +69,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
68
69
|
version: '0'
|
69
70
|
requirements: []
|
70
71
|
rubygems_version: 3.1.4
|
71
|
-
signing_key:
|
72
|
+
signing_key:
|
72
73
|
specification_version: 4
|
73
74
|
summary: This gem works with the Rails Autoscale Heroku add-on to automatically scale
|
74
75
|
your web dynos.
|