goodsorcery 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +30 -1
- data/lib/generators/goodsorcery/install/templates/initializer.rb +7 -0
- data/lib/goodsorcery/client.rb +15 -1
- data/lib/goodsorcery/configuration/performance_settings.rb +37 -0
- data/lib/goodsorcery/configuration/validation.rb +10 -0
- data/lib/goodsorcery/configuration.rb +3 -0
- data/lib/goodsorcery/defaults.rb +4 -0
- data/lib/goodsorcery/integrations/active_job.rb +4 -1
- data/lib/goodsorcery/integrations/rails.rb +3 -2
- data/lib/goodsorcery/integrations/sidekiq.rb +18 -2
- data/lib/goodsorcery/performance/agent.rb +104 -0
- data/lib/goodsorcery/performance/aggregator.rb +63 -0
- data/lib/goodsorcery/performance/call_site.rb +61 -0
- data/lib/goodsorcery/performance/flusher.rb +224 -0
- data/lib/goodsorcery/performance/histogram.rb +23 -0
- data/lib/goodsorcery/performance/job_timing.rb +106 -0
- data/lib/goodsorcery/performance/measurement.rb +117 -0
- data/lib/goodsorcery/performance/net_http.rb +37 -0
- data/lib/goodsorcery/performance/payloads.rb +110 -0
- data/lib/goodsorcery/performance/process_label.rb +22 -0
- data/lib/goodsorcery/performance/recorder.rb +202 -0
- data/lib/goodsorcery/performance/span_list.rb +156 -0
- data/lib/goodsorcery/performance/sql_normalizer.rb +149 -0
- data/lib/goodsorcery/performance/subscribers.rb +140 -0
- data/lib/goodsorcery/performance/trace_buffer.rb +78 -0
- data/lib/goodsorcery/performance/trace_builder.rb +121 -0
- data/lib/goodsorcery/performance/transaction_stats.rb +93 -0
- data/lib/goodsorcery/performance/uuid_v7.rb +27 -0
- data/lib/goodsorcery/performance.rb +158 -0
- data/lib/goodsorcery/rack/middleware.rb +10 -3
- data/lib/goodsorcery/transport.rb +12 -1
- data/lib/goodsorcery/version.rb +1 -1
- data/lib/goodsorcery.rb +17 -0
- metadata +21 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7c757cad1765947ffa3b03990f995e418f676ff2a2385d23b9576bb4aec1462c
|
|
4
|
+
data.tar.gz: be2c7da80aa622f3d8aedf18ce6db47d0427f3740bce87b743dde3ff41117d7d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 675c74e11fff3d79ff4c4ab47f78f131174b75b98f09a441cce26b274325541f3ce97c630644879de598367f863b4ea93c01719850b1ce46aac782479844a7c3
|
|
7
|
+
data.tar.gz: 65c4954e28c02247c4b9d00123957827625516a3b218d56773d44b36315de74d9841622d96573d7cd026afcd218949708ca93cb44510d408932b4d8d28aeb81c
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,14 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.0 (2026-11-07)
|
|
4
|
+
|
|
5
|
+
- **Performance monitoring** (opt-in, `config.performance_monitoring = true` or `GOODSORCERY_PERFORMANCE_MONITORING=1`). Records every Rails request and every Active Job and Sidekiq job: count, failures, total time and a shared log-scale histogram (`gs-v1`) with database, external HTTP, view and Ruby time per bucket, plus queue wait for jobs. Metrics are aggregated per transaction per minute in process and sent once a minute to `POST /api/v1/apm/metrics` from a background thread (`goodsorcery-apm`).
|
|
6
|
+
- **Traces** of the slowest request per transaction per minute (`config.performance_traces`, default on), sent to `POST /api/v1/apm/traces`: SQL, `Net::HTTP`, render and enqueue spans; repeated queries grouped with their count, total and app call site (with code context under `send_code_context`); SQL normalised so no literal values leave the process; HTTP spans and the request path without query strings; GC, allocations, host and process; the id of the error reported during the request.
|
|
7
|
+
- `config.performance_ignore_transactions` (default `["Rails::HealthController#show"]`). `GoodSorcery::DeliveryJob` is never recorded.
|
|
8
|
+
- `GoodSorcery.flush_performance` and, with the `:test` delivery, `GoodSorcery.test_performance_payloads`.
|
|
9
|
+
- The gem's own HTTP requests are never measured. Payloads keep their `batch_id` across retries; `429`, `5xx`, `404` (server without performance endpoints: monitoring turns off) and rejected payloads are handled as documented; data is bounded to five minutes and flushed at exit; fork-safe.
|
|
10
|
+
- `bundle exec rake bench:performance` measures the per-request cost.
|
|
11
|
+
|
|
3
12
|
## 0.1.1 (2026-10-11)
|
|
4
13
|
|
|
5
14
|
- `Sidekiq::JobRetry::Handled` and `Sidekiq::JobRetry::Skip` are ignored by default. Sidekiq raises them internally after its retry handling, so reporting them only duplicated the original error.
|
data/README.md
CHANGED
|
@@ -7,6 +7,7 @@ The Ruby client for the [Good Sorcery](https://app.goodsorcery.com) error tracke
|
|
|
7
7
|
- The queue is bounded and **drops rather than blocks** when it is full. Rate limits (`429` + `Retry-After`) and server errors are retried with backoff.
|
|
8
8
|
- It has no runtime dependencies (standard library only). It needs Ruby 3.2 or newer and works with or without Rails.
|
|
9
9
|
- Rails, Rack/Sinatra, Active Job, Action Mailer, Action Cable, Sidekiq and Rake are covered out of the box (see [Integrations](#integrations)).
|
|
10
|
+
- Optional, opt-in [performance monitoring](#performance-monitoring) for Rails apps: response times of every request and job, and timelines of the slowest ones.
|
|
10
11
|
|
|
11
12
|
## Installation
|
|
12
13
|
|
|
@@ -106,6 +107,9 @@ Change settings inside `GoodSorcery.configure`: it recompiles the filters and re
|
|
|
106
107
|
| `shutdown_timeout` | `5` s | How long the exit hook waits for the queue to drain. |
|
|
107
108
|
| `open_timeout` / `read_timeout` | `2` s / `5` s | HTTP timeouts (writes use `read_timeout`). |
|
|
108
109
|
| `active_job_queue` | `"default"` | Queue for `delivery = :active_job`. |
|
|
110
|
+
| `performance_monitoring` | `false` (or `ENV["GOODSORCERY_PERFORMANCE_MONITORING"]` `1`/`0`) | Records every request and job and sends metrics once a minute; see [Performance monitoring](#performance-monitoring). Read when the app boots. |
|
|
111
|
+
| `performance_traces` | `true` | Also sends the timeline of the slowest request per transaction per minute. `false` sends metrics only. |
|
|
112
|
+
| `performance_ignore_transactions` | `["Rails::HealthController#show"]` | Transaction names (exact strings) or regexps never recorded. |
|
|
109
113
|
|
|
110
114
|
### Delivery
|
|
111
115
|
|
|
@@ -230,6 +234,30 @@ Hooked in automatically when Sidekiq is loaded before `goodsorcery` or in a Rail
|
|
|
230
234
|
|
|
231
235
|
Rails apps get this automatically; elsewhere `require "goodsorcery/tasks"` in your Rakefile. An exception that ends a rake run is reported with `transaction = "rake <tasks>"` (task arguments stay out of the name and go, filtered, to `context.rake`), and the queue is flushed, for up to `shutdown_timeout`, before rake exits.
|
|
232
236
|
|
|
237
|
+
## Performance monitoring
|
|
238
|
+
|
|
239
|
+
Off by default. Turn it on in the initializer (or with `GOODSORCERY_PERFORMANCE_MONITORING=1`) and restart:
|
|
240
|
+
|
|
241
|
+
```ruby
|
|
242
|
+
GoodSorcery.configure do |config|
|
|
243
|
+
config.performance_monitoring = true
|
|
244
|
+
# config.performance_traces = true # false = metrics only
|
|
245
|
+
# config.performance_ignore_transactions += ["HealthController#show", /\AAdmin::/]
|
|
246
|
+
end
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
It follows the same environment rules as error reporting and needs an API key: in a disabled environment nothing is subscribed, no thread starts and a request costs one boolean check.
|
|
250
|
+
|
|
251
|
+
**What is measured.** Each controller action (`CheckoutsController#create`, with the HTTP method and the route pattern such as `/checkout`, never the real path) and each job (Active Job with any adapter, and Sidekiq jobs not wrapped by Active Job; with the adapter, queue and queue wait). A request or job counts as failed when an exception escapes it (for a request, one Rails answers with a 5xx: a `RecordNotFound` is recorded as a 404, not a failure), when a request answers with a 5xx status, or when Active Job's `retry_on`/`discard_on` handles a job's exception. Time is split into database (`sql.active_record`, without `SCHEMA` and cached queries), external HTTP (`Net::HTTP` calls), views (template renders, without the queries and HTTP calls made from them) and Ruby (the rest). Requests that never reach a controller (routing 404s, assets) and a job run inline inside a request are not separate transactions.
|
|
252
|
+
|
|
253
|
+
**How it is sent.** Numbers are aggregated in your process per transaction per minute (count, failures, total time and a log-scale histogram with the time per layer) and sent once a minute from one background thread (`goodsorcery-apm`) as a small gzip'd request. The slowest request per transaction per minute also sends its **trace** (built after the response has been sent, on servers such as Puma that allow it): a timeline of its SQL queries, HTTP calls, template renders and job enqueues. Identical queries are grouped with their count, total time and the line of your code that ran them, so N+1 queries stand out. Unsent data is kept for at most five minutes; the payloads are retried with the same batch id.
|
|
254
|
+
|
|
255
|
+
**What a trace never contains.** No params, headers, cookies, session, request or response bodies, query strings, job arguments or user data. SQL is normalised before it leaves the process: string and number literals become `?`, `IN (…)` lists and `VALUES` rows collapse, comments are removed; bind placeholders such as `$1` stay. Quotes are read the way your database reads them (from the connection's adapter), and when that cannot be told, everything after a string holding a backslash becomes `?`. HTTP calls are method, host and path only (never credentials, even when the app passes a whole URL). Code lines around a repeated query's call site are sent only with `send_code_context` (the default). `before_send` and `filter_params` don't apply, because there is nothing key/value-shaped to filter.
|
|
256
|
+
|
|
257
|
+
**Cost.** Per request, a few microseconds of bookkeeping plus about 2 µs per query (`bundle exec rake bench:performance` measures it on your machine). Memory is bounded: at most 2,000 spans and 4 MiB of SQL text per running request, 500 transaction names per minute and kind (the rest are counted as `(other)`), five minutes of metrics and 30 traces per minute.
|
|
258
|
+
|
|
259
|
+
Outside Rails, call `GoodSorcery::Performance.install` after `GoodSorcery.configure`; jobs and `Net::HTTP` are then measured, but web requests need Rails to get their names. In your test suite, the `:test` delivery keeps the payloads in memory: `GoodSorcery.flush_performance` builds them into `GoodSorcery.test_performance_payloads` (`[{ "path" => "/api/v1/apm/metrics", "body" => {...} }]`).
|
|
260
|
+
|
|
233
261
|
## Install check
|
|
234
262
|
|
|
235
263
|
`bin/rails goodsorcery:verify` (or `rake goodsorcery:verify`) checks your setup on demand; see the end of this section.
|
|
@@ -270,7 +298,7 @@ event = GoodSorcery.test_events.last
|
|
|
270
298
|
assert_equal "Payments::DeclinedError", event["exception"]["class"]
|
|
271
299
|
```
|
|
272
300
|
|
|
273
|
-
The `:test` delivery ignores the environment check, stores the filtered event hashes exactly as they would be sent, and never touches the network.
|
|
301
|
+
The `:test` delivery ignores the environment check, stores the filtered event hashes exactly as they would be sent, and never touches the network. With performance monitoring started (`GoodSorcery::Performance.install` after setting `config.performance_monitoring = true`), `GoodSorcery.flush_performance` stores the performance payloads in `GoodSorcery.test_performance_payloads`; `clear_test_events` clears both.
|
|
274
302
|
|
|
275
303
|
## Self-hosted servers
|
|
276
304
|
|
|
@@ -291,6 +319,7 @@ These are for integrations and advanced setups:
|
|
|
291
319
|
bin/setup # bundle install, then the whole test suite
|
|
292
320
|
bundle exec rake test # core, then Rack/Sidekiq/Rake, then Rails tests, each in its own process
|
|
293
321
|
bundle exec rubocop
|
|
322
|
+
bundle exec rake bench:performance # cost of performance monitoring per request (not run in CI)
|
|
294
323
|
bin/console
|
|
295
324
|
```
|
|
296
325
|
|
|
@@ -48,6 +48,13 @@ GoodSorcery.configure do |config|
|
|
|
48
48
|
# config.breadcrumbs = true
|
|
49
49
|
# config.max_breadcrumbs = 40
|
|
50
50
|
|
|
51
|
+
# --- Performance monitoring --------------------------------------------------
|
|
52
|
+
# Off by default. Records every request and job and sends the slowest
|
|
53
|
+
# requests' timelines once a minute (also ENV["GOODSORCERY_PERFORMANCE_MONITORING"]=1).
|
|
54
|
+
# config.performance_monitoring = true
|
|
55
|
+
# config.performance_traces = true # false = metrics only, no traces
|
|
56
|
+
# config.performance_ignore_transactions += ["HealthController#show", /\AAdmin::/]
|
|
57
|
+
|
|
51
58
|
# --- Development -----------------------------------------------------------
|
|
52
59
|
# config.install_check = :verify # after bundle install: :verify, :test_event or false
|
|
53
60
|
# config.logger = Rails.logger
|
data/lib/goodsorcery/client.rb
CHANGED
|
@@ -69,9 +69,21 @@ module GoodSorcery
|
|
|
69
69
|
end
|
|
70
70
|
|
|
71
71
|
def shutdown(timeout = config.shutdown_timeout)
|
|
72
|
+
Performance.shutdown(timeout)
|
|
72
73
|
@delivery ? @delivery.shutdown(timeout) : true
|
|
73
74
|
end
|
|
74
75
|
|
|
76
|
+
# Whether performance monitoring may run (DESIGN.md 8.1): with the :test
|
|
77
|
+
# delivery, or a network delivery in an enabled environment with a key.
|
|
78
|
+
def performance_allowed?
|
|
79
|
+
return true if test_delivery?
|
|
80
|
+
return @gate.enabled? && !config.api_key.to_s.empty? if NETWORK_DELIVERIES.include?(config.delivery)
|
|
81
|
+
|
|
82
|
+
log.warn_once(:apm_custom_delivery,
|
|
83
|
+
"performance monitoring needs the :async, :sync or :active_job delivery; it is off")
|
|
84
|
+
false
|
|
85
|
+
end
|
|
86
|
+
|
|
75
87
|
def verify
|
|
76
88
|
Verification.run(config, transport)
|
|
77
89
|
end
|
|
@@ -123,7 +135,9 @@ module GoodSorcery
|
|
|
123
135
|
event = prepare_event(exception, options)
|
|
124
136
|
return nil unless event
|
|
125
137
|
|
|
126
|
-
delivery.deliver(event) == false ? nil : event["event_id"]
|
|
138
|
+
event_id = delivery.deliver(event) == false ? nil : event["event_id"]
|
|
139
|
+
Performance.note_error(event_id)
|
|
140
|
+
event_id
|
|
127
141
|
rescue StandardError => e
|
|
128
142
|
log.error("notify failed with #{e.class}; the error was not reported")
|
|
129
143
|
nil
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module GoodSorcery
|
|
4
|
+
class Configuration
|
|
5
|
+
# The performance monitoring settings (DESIGN.md 8.1). Monitoring is off
|
|
6
|
+
# unless set here or with GOODSORCERY_PERFORMANCE_MONITORING=1.
|
|
7
|
+
module PerformanceSettings
|
|
8
|
+
def self.included(base)
|
|
9
|
+
base.attr_accessor :performance_ignore_transactions
|
|
10
|
+
base.attr_reader :performance_traces
|
|
11
|
+
base.boolean_setting :performance_traces
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def performance_monitoring
|
|
15
|
+
return @performance_monitoring unless @performance_monitoring.nil?
|
|
16
|
+
|
|
17
|
+
EnvFlag.parse(ENV.fetch("GOODSORCERY_PERFORMANCE_MONITORING", nil)) || false
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def performance_monitoring=(value)
|
|
21
|
+
unless [true, false, nil].include?(value)
|
|
22
|
+
raise ArgumentError, "performance_monitoring must be true, false or nil (got #{value.inspect})"
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
@performance_monitoring = value
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
def assign_performance_defaults
|
|
31
|
+
@performance_monitoring = nil
|
|
32
|
+
@performance_traces = true
|
|
33
|
+
@performance_ignore_transactions = Defaults::PERFORMANCE_IGNORE_TRANSACTIONS.dup
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
end
|
|
@@ -44,6 +44,16 @@ module GoodSorcery
|
|
|
44
44
|
false
|
|
45
45
|
end
|
|
46
46
|
|
|
47
|
+
def boolean_setting(name)
|
|
48
|
+
define_method(:"#{name}=") do |value|
|
|
49
|
+
unless [true, false].include?(value)
|
|
50
|
+
raise ArgumentError, "#{name} must be true or false (got #{value.inspect})"
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
instance_variable_set(:"@#{name}", value)
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
|
|
47
57
|
def choice_setting(name, choices)
|
|
48
58
|
define_method(:"#{name}=") do |value|
|
|
49
59
|
normalized = value.is_a?(String) ? value.to_sym : value
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require_relative "configuration/validation"
|
|
4
|
+
require_relative "configuration/performance_settings"
|
|
4
5
|
|
|
5
6
|
module GoodSorcery
|
|
6
7
|
# Every setting the gem reads (DESIGN.md 2 and 3). Unset credentials,
|
|
@@ -8,6 +9,7 @@ module GoodSorcery
|
|
|
8
9
|
# environment variables (and Rails) when they are read.
|
|
9
10
|
class Configuration
|
|
10
11
|
extend Validation
|
|
12
|
+
include PerformanceSettings
|
|
11
13
|
|
|
12
14
|
DELIVERIES = %i[async sync active_job test].freeze
|
|
13
15
|
DELIVERY_SETTINGS = %i[
|
|
@@ -156,6 +158,7 @@ module GoodSorcery
|
|
|
156
158
|
@install_check = :verify
|
|
157
159
|
@report_handled_errors = true
|
|
158
160
|
assign_capture_defaults
|
|
161
|
+
assign_performance_defaults
|
|
159
162
|
end
|
|
160
163
|
|
|
161
164
|
def assign_capture_defaults
|
data/lib/goodsorcery/defaults.rb
CHANGED
|
@@ -36,6 +36,10 @@ module GoodSorcery
|
|
|
36
36
|
SystemExit
|
|
37
37
|
].freeze
|
|
38
38
|
|
|
39
|
+
# Transactions (exact names or regexps) that performance monitoring
|
|
40
|
+
# never records. GoodSorcery::DeliveryJob is always skipped as well.
|
|
41
|
+
PERFORMANCE_IGNORE_TRANSACTIONS = ["Rails::HealthController#show"].freeze
|
|
42
|
+
|
|
39
43
|
DISABLED_ENVIRONMENTS = %w[development test].freeze
|
|
40
44
|
ENDPOINT = "https://app.goodsorcery.com"
|
|
41
45
|
end
|
|
@@ -26,8 +26,11 @@ module GoodSorcery
|
|
|
26
26
|
|
|
27
27
|
def perform_now
|
|
28
28
|
return super if Integrations.delivery_job?(self)
|
|
29
|
+
return Integrations.run_unit_of_work(JobDetails.new(self)) { super } unless Performance.active?
|
|
29
30
|
|
|
30
|
-
|
|
31
|
+
Performance.measure_job(Performance::ActiveJobTiming.new(self)) do
|
|
32
|
+
Integrations.run_unit_of_work(JobDetails.new(self)) { super }
|
|
33
|
+
end
|
|
31
34
|
end
|
|
32
35
|
end
|
|
33
36
|
|
|
@@ -31,13 +31,14 @@ module GoodSorcery
|
|
|
31
31
|
end
|
|
32
32
|
|
|
33
33
|
# After the app's initializers ran: subscribe to Rails.error and to
|
|
34
|
-
# notifications, hook Sidekiq,
|
|
35
|
-
# filter_parameters apply.
|
|
34
|
+
# notifications, hook Sidekiq, recompile the filters so the final
|
|
35
|
+
# filter_parameters apply, and start performance monitoring if it is on.
|
|
36
36
|
def after_initialize
|
|
37
37
|
subscribe_to_error_reporter
|
|
38
38
|
NotificationBreadcrumbs.subscribe if GoodSorcery.configuration.breadcrumbs
|
|
39
39
|
install_sidekiq if defined?(::Sidekiq)
|
|
40
40
|
GoodSorcery.client.reconfigure(rebuild_delivery: false)
|
|
41
|
+
GoodSorcery::Performance.install if GoodSorcery.configuration.performance_monitoring
|
|
41
42
|
end
|
|
42
43
|
|
|
43
44
|
def subscribe_to_error_reporter
|
|
@@ -30,9 +30,25 @@ module GoodSorcery
|
|
|
30
30
|
attr_accessor :config
|
|
31
31
|
|
|
32
32
|
def call(worker, job, queue, &)
|
|
33
|
-
return
|
|
33
|
+
return wrapped(job, &) if Sidekiq.active_job_wrapper?(job)
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
details = JobDetails.new(job, queue: queue, worker: worker)
|
|
36
|
+
return Integrations.run_unit_of_work(details, &) unless Performance.active?
|
|
37
|
+
|
|
38
|
+
Performance.measure_job(Performance::SidekiqTiming.new(job, queue, worker)) do
|
|
39
|
+
Integrations.run_unit_of_work(details, &)
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
private
|
|
44
|
+
|
|
45
|
+
# A Sidekiq retry runs an Active Job with its original job data, so
|
|
46
|
+
# its enqueued_at is the first enqueue; Sidekiq's own enqueued_at is
|
|
47
|
+
# when the retry was queued, which the queue wait is measured from.
|
|
48
|
+
def wrapped(job, &)
|
|
49
|
+
return yield unless Performance.active? && job.key?("retry_count")
|
|
50
|
+
|
|
51
|
+
Performance::ActiveJobTiming.requeued_at(job["enqueued_at"], &)
|
|
36
52
|
end
|
|
37
53
|
end
|
|
38
54
|
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module GoodSorcery
|
|
4
|
+
module Performance
|
|
5
|
+
# The per-process state: metrics, traces and the flusher. A forked child
|
|
6
|
+
# gets a fresh, empty agent (the parent keeps and sends what it
|
|
7
|
+
# recorded), and its flusher thread starts on the child's first record.
|
|
8
|
+
class Agent
|
|
9
|
+
attr_reader :pid, :aggregator, :trace_buffer, :flusher, :context
|
|
10
|
+
|
|
11
|
+
def initialize(config, log)
|
|
12
|
+
@pid = Process.pid
|
|
13
|
+
@aggregator = Aggregator.new
|
|
14
|
+
@trace_buffer = TraceBuffer.new
|
|
15
|
+
@context = TraceContext.new(config)
|
|
16
|
+
@flusher = Flusher.new(agent: self, log: log)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def current?
|
|
20
|
+
@pid == Process.pid
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def record(recorder, background:)
|
|
24
|
+
minute = (recorder.finished_wall / 60).floor
|
|
25
|
+
@aggregator.record(minute, Sample.from(recorder))
|
|
26
|
+
offer_trace(minute, recorder) if recorder.traces?
|
|
27
|
+
@flusher.ensure_started if background
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
private
|
|
31
|
+
|
|
32
|
+
# A request's trace is built once the response has gone out, when the
|
|
33
|
+
# server offers a hook for that, so the slowest requests are not kept
|
|
34
|
+
# waiting for it.
|
|
35
|
+
def offer_trace(minute, recorder)
|
|
36
|
+
key = [recorder.kind, recorder.name]
|
|
37
|
+
return unless @trace_buffer.candidate?(minute, key, recorder.duration_ms)
|
|
38
|
+
|
|
39
|
+
hooks = AfterReply.hooks(recorder.env)
|
|
40
|
+
return build_trace(minute, key, recorder) unless hooks
|
|
41
|
+
|
|
42
|
+
hooks << AfterReply.callback { build_trace(minute, key, recorder) }
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def build_trace(minute, key, recorder)
|
|
46
|
+
return unless @trace_buffer.candidate?(minute, key, recorder.duration_ms)
|
|
47
|
+
|
|
48
|
+
trace = TraceBuilder.new(recorder, minute, @context).build
|
|
49
|
+
@trace_buffer.offer(minute, key, recorder.duration_ms, trace)
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# The Rack server's hooks run after the response is sent: Rack 3's
|
|
54
|
+
# rack.response_finished (callables get env, status, headers, error)
|
|
55
|
+
# or Puma's rack.after_reply (no arguments).
|
|
56
|
+
module AfterReply
|
|
57
|
+
KEYS = %w[rack.response_finished rack.after_reply].freeze
|
|
58
|
+
|
|
59
|
+
def self.hooks(env)
|
|
60
|
+
return nil unless env.is_a?(Hash)
|
|
61
|
+
|
|
62
|
+
KEYS.each do |key|
|
|
63
|
+
hooks = env[key]
|
|
64
|
+
return hooks if hooks.is_a?(Array) && !hooks.frozen?
|
|
65
|
+
end
|
|
66
|
+
nil
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# A hook that never raises into the server.
|
|
70
|
+
def self.callback(&work)
|
|
71
|
+
proc do |*|
|
|
72
|
+
work.call
|
|
73
|
+
rescue StandardError => e
|
|
74
|
+
GoodSorcery.log.warn_every(:apm_trace_error, 60, "could not build a performance trace (#{e.class})")
|
|
75
|
+
end
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# performance_ignore_transactions: exact names or regexps, plus the gem's
|
|
80
|
+
# own delivery job. Answers are cached per name.
|
|
81
|
+
class IgnoreMatcher
|
|
82
|
+
ALWAYS = ["GoodSorcery::DeliveryJob"].freeze
|
|
83
|
+
MAX_CACHED = 1_000
|
|
84
|
+
|
|
85
|
+
def initialize(list)
|
|
86
|
+
entries = Array(list)
|
|
87
|
+
@names = (entries.grep_v(Regexp).map(&:to_s) + ALWAYS).to_h { |name| [name, true] }
|
|
88
|
+
@patterns = entries.grep(Regexp)
|
|
89
|
+
@cache = {}
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
def ignored?(name)
|
|
93
|
+
return true if @names.key?(name)
|
|
94
|
+
return false if @patterns.empty?
|
|
95
|
+
|
|
96
|
+
cached = @cache[name]
|
|
97
|
+
return cached unless cached.nil?
|
|
98
|
+
|
|
99
|
+
@cache.clear if @cache.size >= MAX_CACHED
|
|
100
|
+
@cache[name] = @patterns.any? { |pattern| pattern.match?(name) }
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
end
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module GoodSorcery
|
|
4
|
+
module Performance
|
|
5
|
+
# Per-process, per-minute metrics (DESIGN.md 8.4). Keyed by the minute in
|
|
6
|
+
# which a unit of work finished (integer Unix minutes), then kind, then
|
|
7
|
+
# name. At most MAX_TRANSACTIONS names per kind per minute (the rest are
|
|
8
|
+
# counted under "(other)") and MAX_PENDING_MINUTES minutes are held.
|
|
9
|
+
class Aggregator
|
|
10
|
+
MAX_TRANSACTIONS = 500
|
|
11
|
+
MAX_PENDING_MINUTES = 5
|
|
12
|
+
OTHER = "(other)"
|
|
13
|
+
|
|
14
|
+
attr_reader :dropped_minutes
|
|
15
|
+
|
|
16
|
+
def initialize
|
|
17
|
+
@lock = Mutex.new
|
|
18
|
+
@minutes = {}
|
|
19
|
+
@dropped_minutes = 0
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def record(minute, sample)
|
|
23
|
+
@lock.synchronize do
|
|
24
|
+
names = (minute_table(minute)[sample.kind] ||= {})
|
|
25
|
+
name = names.key?(sample.name) || names.size < MAX_TRANSACTIONS ? sample.name : OTHER
|
|
26
|
+
(names[name] ||= TransactionStats.new).add(sample, keep_labels: name != OTHER)
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Removes and returns the minutes up to and including +last_minute+
|
|
31
|
+
# (all of them when nil), oldest first: [[minute, {kind => {name => stats}}]].
|
|
32
|
+
def take(last_minute = nil)
|
|
33
|
+
@lock.synchronize do
|
|
34
|
+
taken = @minutes.keys.sort.select { |minute| last_minute.nil? || minute <= last_minute }
|
|
35
|
+
taken.map { |minute| [minute, @minutes.delete(minute)] }
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def minutes
|
|
40
|
+
@lock.synchronize { @minutes.keys.sort }
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def empty?
|
|
44
|
+
@lock.synchronize { @minutes.empty? }
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
private
|
|
48
|
+
|
|
49
|
+
def minute_table(minute)
|
|
50
|
+
table = @minutes[minute]
|
|
51
|
+
return table if table
|
|
52
|
+
|
|
53
|
+
drop_oldest while @minutes.size >= MAX_PENDING_MINUTES
|
|
54
|
+
@minutes[minute] = {}
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def drop_oldest
|
|
58
|
+
@minutes.delete(@minutes.keys.min)
|
|
59
|
+
@dropped_minutes += 1
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
end
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module GoodSorcery
|
|
4
|
+
module Performance
|
|
5
|
+
# Where the app ran a repeated query: the first stack frame that is the
|
|
6
|
+
# app's own code (not a gem, not this library). Captured inside the
|
|
7
|
+
# sql.active_record callback; the code lines are read only when a trace
|
|
8
|
+
# is built.
|
|
9
|
+
class CallSite
|
|
10
|
+
GEM_LIB = "#{File.expand_path("../..", __dir__)}/".freeze
|
|
11
|
+
CONTEXT_BEFORE = 2
|
|
12
|
+
CONTEXT_AFTER = 2
|
|
13
|
+
MAX_LINE_LENGTH = 200
|
|
14
|
+
MAX_PATH_LENGTH = 1_024
|
|
15
|
+
MAX_METHOD_LENGTH = 255
|
|
16
|
+
|
|
17
|
+
attr_reader :path, :line, :label
|
|
18
|
+
|
|
19
|
+
# => a CallSite, or nil when no app frame is on the stack.
|
|
20
|
+
def self.capture(paths)
|
|
21
|
+
Thread.each_caller_location do |location|
|
|
22
|
+
path = location.absolute_path || location.path
|
|
23
|
+
next if path.nil? || path.start_with?(GEM_LIB) || !paths.in_app?(path)
|
|
24
|
+
|
|
25
|
+
return new(path, location.lineno, location.base_label)
|
|
26
|
+
end
|
|
27
|
+
nil
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def initialize(path, line, label)
|
|
31
|
+
@path = path
|
|
32
|
+
@line = line
|
|
33
|
+
@label = label
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def to_h(paths, line_cache: nil)
|
|
37
|
+
hash = { "file" => Text.limit(paths.relative(path), MAX_PATH_LENGTH), "line" => line,
|
|
38
|
+
"method" => label && Text.limit(label, MAX_METHOD_LENGTH) }.compact
|
|
39
|
+
context = line_cache && code_context(line_cache)
|
|
40
|
+
hash["context"] = context if context
|
|
41
|
+
hash
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
private
|
|
45
|
+
|
|
46
|
+
# Two lines before and after the call (5 lines of up to 200 chars).
|
|
47
|
+
def code_context(line_cache)
|
|
48
|
+
lines = line_cache.lines(path)
|
|
49
|
+
return nil unless lines && line.between?(1, lines.size)
|
|
50
|
+
|
|
51
|
+
first = [line - CONTEXT_BEFORE, 1].max
|
|
52
|
+
last = [line + CONTEXT_AFTER, lines.size].min
|
|
53
|
+
{ "start_line" => first, "lines" => limited(lines[(first - 1)..(last - 1)]) }
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def limited(lines)
|
|
57
|
+
lines.map { |text| Text.limit(text, MAX_LINE_LENGTH) }
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
end
|