forge_ops_tracker 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +15 -0
- data/README.md +64 -0
- data/lib/forge_ops_tracker/client.rb +6 -0
- data/lib/forge_ops_tracker/configuration.rb +12 -0
- data/lib/forge_ops_tracker/performance_instrumentation.rb +32 -0
- data/lib/forge_ops_tracker/query_explainer.rb +249 -0
- data/lib/forge_ops_tracker/railtie.rb +16 -2
- data/lib/forge_ops_tracker/version.rb +1 -1
- data/lib/forge_ops_tracker.rb +16 -1
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5b7f1809933ffeccac536f117de7c234002826fe29414fc3fa92f12c22bb1a18
|
|
4
|
+
data.tar.gz: 84a9edadbaeb72dbe62164a84f26c7f23b6a600f176126ca82370709dc72b5c3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ca419d26d17f08e4544bb213c2949dedf7001402f163a15b0def3c0942ffcc534637911d1bc7e8890f6a3dc556f174a3ecea703351fe291d492c1c151838e3f9
|
|
7
|
+
data.tar.gz: 6890f1adbbf73ae399699ee3482a005bbfee521909c6af6acedb8a805a2aa69c3d46f75bf33a3a4722be9f0e6e84a0263850729e42380d1d1b26e23e50f285cb
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,20 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.14.0 (2026-09-25)
|
|
4
|
+
|
|
5
|
+
- Every Active Record database span now carries the query's SQL (`db.statement`), masked so every
|
|
6
|
+
string and number is `?` and bind values are never sent, and the database it ran on
|
|
7
|
+
(`db.system`: `postgresql`, `mysql`, `sqlite`, `mssql`, `oracle`, or the adapter's own name).
|
|
8
|
+
- `ForgeOpsTracker.span` takes optional `statement:` and `db_system:` on a `kind: "database"`
|
|
9
|
+
span, masked the same way.
|
|
10
|
+
- New opt-in `config.explain_slow_queries` (default false) and `config.explain_threshold_ms`
|
|
11
|
+
(default 500). On PostgreSQL, a single plain `SELECT` at least that slow gets a plain
|
|
12
|
+
`EXPLAIN (FORMAT JSON)` (never `EXPLAIN ANALYZE`), run on a background thread with its own
|
|
13
|
+
pooled connection, in a read-only transaction with a 2 second statement timeout that is always
|
|
14
|
+
rolled back. The plan is sent with every string in it masked. Limited to once per 10 minutes per
|
|
15
|
+
distinct query and 10 per minute per process; plans over 64 KB are dropped; any failure is
|
|
16
|
+
logged at debug and never raised.
|
|
17
|
+
|
|
3
18
|
## 0.13.0 (2026-09-25)
|
|
4
19
|
|
|
5
20
|
- New `ForgeOpsTracker.record_change(kind:, title:, details: {}, environment: nil, service: nil,
|
data/README.md
CHANGED
|
@@ -411,6 +411,70 @@ Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn
|
|
|
411
411
|
simply rejected server-side and dropped, exactly like any other delivery failure. The `trace_id`
|
|
412
412
|
on error events is sent either way.
|
|
413
413
|
|
|
414
|
+
### The SQL behind each database span
|
|
415
|
+
|
|
416
|
+
Every Active Record query span carries the query's SQL and the database it ran on, so a slow trace
|
|
417
|
+
shows exactly which query was slow. The SQL is masked before it leaves your process: every string
|
|
418
|
+
and number is replaced by `?`, so `WHERE email = 'a@b.co' AND id = 42` is sent as
|
|
419
|
+
`WHERE email = ? AND id = ?`, and bind values are never sent at all. ForgeOps masks it again on
|
|
420
|
+
arrival. The database comes from the connection's adapter (`postgresql`, `mysql`, `sqlite`,
|
|
421
|
+
`mssql`, `oracle`, or the adapter's own name otherwise).
|
|
422
|
+
|
|
423
|
+
Spans you create yourself can carry SQL too. Pass `statement:` (and optionally `db_system:`) on a
|
|
424
|
+
`kind: "database"` span; it's masked the same way:
|
|
425
|
+
|
|
426
|
+
```ruby
|
|
427
|
+
sql = ActiveRecord::Base.sanitize_sql_array(
|
|
428
|
+
[ "SELECT * FROM orders WHERE customer_id = ? AND status = ?", customer.id, "open" ]
|
|
429
|
+
)
|
|
430
|
+
|
|
431
|
+
ForgeOpsTracker.span("Load open orders", kind: "database", statement: sql, db_system: "postgresql") do
|
|
432
|
+
ReportingDatabase.connection.select_all(sql)
|
|
433
|
+
end
|
|
434
|
+
# Sent as "SELECT * FROM orders WHERE customer_id = ? AND status = ?"
|
|
435
|
+
```
|
|
436
|
+
|
|
437
|
+
### Query plans for slow queries (PostgreSQL)
|
|
438
|
+
|
|
439
|
+
Opt in, and a slow `SELECT` also sends Postgres's plan for it, so ForgeOps can show why it was
|
|
440
|
+
slow (a sequential scan over a big table, a missing index, a bad join order):
|
|
441
|
+
|
|
442
|
+
```ruby
|
|
443
|
+
ForgeOpsTracker.configure do |config|
|
|
444
|
+
config.explain_slow_queries = true # default false
|
|
445
|
+
config.explain_threshold_ms = 500 # default 500; only queries at least this slow
|
|
446
|
+
end
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
What it does:
|
|
450
|
+
|
|
451
|
+
- For a query that took at least `explain_threshold_ms`, runs `EXPLAIN (FORMAT JSON)` with the
|
|
452
|
+
same SQL and bind values, and sends the plan with the query's masked SQL. Every string inside the
|
|
453
|
+
plan (a filter like `(email = 'a@b.co'::text)`) is masked the same way before it's sent. A plan
|
|
454
|
+
over 64 KB is dropped rather than sent.
|
|
455
|
+
- Runs on the gem's own background thread, on a connection it checks out of your app's pool itself,
|
|
456
|
+
inside its own `READ ONLY` transaction with a 2 second `statement_timeout`, always rolled back.
|
|
457
|
+
Never on the connection your request is using, and never inside your request's transaction.
|
|
458
|
+
- At most once per 10 minutes for each distinct (masked) query, and at most 10 per minute per
|
|
459
|
+
process overall.
|
|
460
|
+
|
|
461
|
+
What it doesn't do:
|
|
462
|
+
|
|
463
|
+
- It never runs `EXPLAIN ANALYZE`, so your query is planned again but never executed again.
|
|
464
|
+
- It only explains a single plain `SELECT`: nothing that starts with anything else (including
|
|
465
|
+
`WITH`), nothing with a second statement, no `FOR UPDATE`/`FOR SHARE` (or `FOR NO KEY UPDATE`/
|
|
466
|
+
`FOR KEY SHARE`), and nothing with `INSERT`, `UPDATE`, `DELETE` or `MERGE` anywhere in it.
|
|
467
|
+
- It only runs on PostgreSQL. Other databases are left alone.
|
|
468
|
+
- It never raises into your app. A timeout, a permission error, an exhausted connection pool or
|
|
469
|
+
anything else is logged at debug level and skipped.
|
|
470
|
+
- The original SQL and its bind values are only held in memory until the `EXPLAIN` runs. Only the
|
|
471
|
+
masked SQL and the masked plan are sent.
|
|
472
|
+
|
|
473
|
+
It does use one pooled connection, briefly, for each plan. With the limits above that's at most 10
|
|
474
|
+
short checkouts a minute, but on a pool that's already at capacity, leave it off or raise
|
|
475
|
+
`explain_threshold_ms`. Plans are part of performance monitoring; on a plan without it, they're
|
|
476
|
+
rejected server-side and dropped.
|
|
477
|
+
|
|
414
478
|
## What changed
|
|
415
479
|
|
|
416
480
|
ForgeOps can show what changed in your system next to the errors and slowdowns that followed it.
|
|
@@ -71,6 +71,12 @@ module ForgeOpsTracker
|
|
|
71
71
|
post(configuration.change_snapshots_uri, payload)
|
|
72
72
|
end
|
|
73
73
|
|
|
74
|
+
# Same again; see Configuration#query_plans_uri and QueryExplainer. One plan per call. A plan
|
|
75
|
+
# without performance monitoring answers 403, which is just a false here.
|
|
76
|
+
def deliver_query_plan(payload)
|
|
77
|
+
post(configuration.query_plans_uri, payload)
|
|
78
|
+
end
|
|
79
|
+
|
|
74
80
|
private
|
|
75
81
|
attr_reader :configuration
|
|
76
82
|
|
|
@@ -17,6 +17,7 @@ module ForgeOpsTracker
|
|
|
17
17
|
attr_accessor :metric_flush_interval, :infrastructure_metric_flush_interval
|
|
18
18
|
attr_accessor :gauge_poll_interval
|
|
19
19
|
attr_accessor :track_tracing, :trace_capture_threshold_ms
|
|
20
|
+
attr_accessor :explain_slow_queries, :explain_threshold_ms
|
|
20
21
|
attr_accessor :propagate_traces, :trace_propagation_targets
|
|
21
22
|
attr_accessor :detect_changes, :track_env_var_names
|
|
22
23
|
|
|
@@ -126,6 +127,13 @@ module ForgeOpsTracker
|
|
|
126
127
|
# PerformanceSample-derived threshold elsewhere in this gem: those describe a trend across
|
|
127
128
|
# many requests, this describes a single one being outright slow.
|
|
128
129
|
@trace_capture_threshold_ms = 1_000
|
|
130
|
+
# Opt-in: for a SELECT on PostgreSQL that took at least explain_threshold_ms, ask Postgres for
|
|
131
|
+
# its plan (plain EXPLAIN, never EXPLAIN ANALYZE, so the query is never run again) and send
|
|
132
|
+
# it, masked, so ForgeOps can show why the query was slow. Off by default because it runs one
|
|
133
|
+
# extra statement against the app's own database. See QueryExplainer for exactly what it
|
|
134
|
+
# does and doesn't do, and its rate limits.
|
|
135
|
+
@explain_slow_queries = false
|
|
136
|
+
@explain_threshold_ms = 500
|
|
129
137
|
# Adds a W3C `traceparent` header (https://www.w3.org/TR/trace-context/) to every outbound
|
|
130
138
|
# Net::HTTP request made during a request, so the service on the other end can continue this
|
|
131
139
|
# request's trace instead of starting its own (see Integrations::NetHTTP). On by default, the
|
|
@@ -241,6 +249,10 @@ module ForgeOpsTracker
|
|
|
241
249
|
uri
|
|
242
250
|
end
|
|
243
251
|
|
|
252
|
+
def query_plans_uri
|
|
253
|
+
sibling_uri("/query_plans")
|
|
254
|
+
end
|
|
255
|
+
|
|
244
256
|
def changes_uri
|
|
245
257
|
sibling_uri("/changes")
|
|
246
258
|
end
|
|
@@ -70,6 +70,38 @@ module ForgeOpsTracker
|
|
|
70
70
|
name.nil? || name.empty? ? "SQL" : name
|
|
71
71
|
end
|
|
72
72
|
|
|
73
|
+
# Active Record's adapter names, as ForgeOps' "db.system" values. Anything not listed here is
|
|
74
|
+
# sent as its own adapter name, lowercased.
|
|
75
|
+
DB_SYSTEMS = {
|
|
76
|
+
"postgresql" => "postgresql", "postgis" => "postgresql",
|
|
77
|
+
"mysql" => "mysql", "mysql2" => "mysql", "trilogy" => "mysql",
|
|
78
|
+
"sqlite" => "sqlite", "sqlite3" => "sqlite",
|
|
79
|
+
"sqlserver" => "mssql",
|
|
80
|
+
"oracle" => "oracle", "oracleenhanced" => "oracle"
|
|
81
|
+
}.freeze
|
|
82
|
+
|
|
83
|
+
# The "data" a database span carries: the query's SQL, masked by SqlStatement (every string
|
|
84
|
+
# and number literal becomes "?", so no value the app queried with is ever sent), and which
|
|
85
|
+
# database it ran on, from the connection's adapter. Either is left out when it isn't known.
|
|
86
|
+
def database_span_data(payload)
|
|
87
|
+
{
|
|
88
|
+
"db.statement" => SqlStatement.mask(payload[:sql].to_s),
|
|
89
|
+
"db.system" => db_system(payload[:connection])
|
|
90
|
+
}.compact
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
# payload[:connection] is the adapter instance the query ran on (Rails 6.0 and later).
|
|
94
|
+
def db_system(connection)
|
|
95
|
+
return nil unless connection.respond_to?(:adapter_name)
|
|
96
|
+
|
|
97
|
+
name = connection.adapter_name.to_s.downcase
|
|
98
|
+
return nil if name.empty?
|
|
99
|
+
|
|
100
|
+
DB_SYSTEMS.fetch(name, name)
|
|
101
|
+
rescue StandardError
|
|
102
|
+
nil
|
|
103
|
+
end
|
|
104
|
+
|
|
73
105
|
def job_transaction_name(payload)
|
|
74
106
|
payload[:job].class.name
|
|
75
107
|
end
|
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
require "json"
|
|
2
|
+
require "thread"
|
|
3
|
+
require "time"
|
|
4
|
+
|
|
5
|
+
module ForgeOpsTracker
|
|
6
|
+
# Opt-in (configuration.explain_slow_queries, default false): asks PostgreSQL for the plan of a
|
|
7
|
+
# slow SELECT and sends it, masked, to ForgeOps so a slow query shows why it was slow.
|
|
8
|
+
#
|
|
9
|
+
# What it never does, by design:
|
|
10
|
+
# - EXPLAIN ANALYZE. Plain EXPLAIN only plans the statement, it never executes it.
|
|
11
|
+
# - Touch the app's own connection or transaction. #consider only reads the notification payload
|
|
12
|
+
# and remembers the connection's pool; the EXPLAIN runs later, on this class's own background
|
|
13
|
+
# thread, on a connection it checks out of that pool itself, inside its own READ ONLY
|
|
14
|
+
# transaction with a 2 second statement_timeout, always rolled back.
|
|
15
|
+
# - Explain anything but a single plain SELECT (see .explainable?).
|
|
16
|
+
# - Raise. Every failure (a timeout, a permission error, an exhausted pool, a statement Postgres
|
|
17
|
+
# can't parse) is logged at debug and dropped.
|
|
18
|
+
# - Send the original statement or its bind values anywhere. Both are held in memory only until
|
|
19
|
+
# the EXPLAIN runs; what goes out is the masked statement (exactly as sent on the span) and the
|
|
20
|
+
# plan with every string in it masked the same way, since plan conditions quote literals
|
|
21
|
+
# ("Filter": "(email = 'a@b.co'::text)").
|
|
22
|
+
#
|
|
23
|
+
# Rate limited per process: each distinct masked statement at most once per 10 minutes, and at
|
|
24
|
+
# most 10 EXPLAINs per minute overall. Both are reserved when a query is accepted, not when its
|
|
25
|
+
# EXPLAIN succeeds, so a statement whose EXPLAIN keeps failing still can't be retried more often.
|
|
26
|
+
class QueryExplainer
|
|
27
|
+
STATEMENT_INTERVAL = 600.0
|
|
28
|
+
GLOBAL_WINDOW = 60.0
|
|
29
|
+
GLOBAL_LIMIT = 10
|
|
30
|
+
MAX_PLAN_BYTES = 64 * 1024
|
|
31
|
+
STATEMENT_TIMEOUT = "2s".freeze
|
|
32
|
+
CHECKOUT_TIMEOUT = 1
|
|
33
|
+
QUEUE_SIZE = GLOBAL_LIMIT
|
|
34
|
+
# Past this many remembered statements, the ones whose 10 minutes are up are forgotten, so an
|
|
35
|
+
# app with endless distinct statements can't grow this without bound.
|
|
36
|
+
MAX_REMEMBERED_STATEMENTS = 1_000
|
|
37
|
+
|
|
38
|
+
# One pass, left to right, so a quote inside a comment or comment syntax inside a string can't
|
|
39
|
+
# confuse the other: string literals (single-quoted and dollar-quoted) become "?", comments
|
|
40
|
+
# become a space.
|
|
41
|
+
LITERALS_AND_COMMENTS = %r{
|
|
42
|
+
'(?:[^']|'')*(?:'|\z)
|
|
43
|
+
| (?<tag>\$[A-Za-z_]*\$).*?(?:\k<tag>|\z)
|
|
44
|
+
| /\*.*?(?:\*/|\z)
|
|
45
|
+
| --[^\n]*
|
|
46
|
+
}mx
|
|
47
|
+
LOCKING_CLAUSE = /\bFOR\s+(?:NO\s+KEY\s+UPDATE|KEY\s+SHARE|UPDATE|SHARE)\b/i
|
|
48
|
+
DATA_MODIFYING = /\b(?:INSERT|UPDATE|DELETE|MERGE)\b/i
|
|
49
|
+
|
|
50
|
+
Job = Struct.new(:sql, :binds, :pool, :statement, :duration_ms, :captured_at, keyword_init: true)
|
|
51
|
+
|
|
52
|
+
class << self
|
|
53
|
+
# True on this class's own worker thread while it's running an EXPLAIN. Railtie's
|
|
54
|
+
# sql.active_record subscriber skips those queries entirely, so the EXPLAIN (and its BEGIN,
|
|
55
|
+
# SET LOCAL and ROLLBACK) never shows up as the app's own query timing, breadcrumbs or spans.
|
|
56
|
+
def explaining?
|
|
57
|
+
Thread.current[:forge_ops_tracker_explaining] == true
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
# Whether statement is a single plain SELECT: after dropping comments and string literals,
|
|
61
|
+
# it starts with SELECT, has no ";" other than one trailing one, no locking clause (FOR
|
|
62
|
+
# UPDATE, FOR SHARE, FOR NO KEY UPDATE, FOR KEY SHARE), and no INSERT, UPDATE, DELETE or
|
|
63
|
+
# MERGE anywhere (so no data-modifying CTE either).
|
|
64
|
+
def explainable?(statement)
|
|
65
|
+
return false unless statement.is_a?(String)
|
|
66
|
+
|
|
67
|
+
sql = statement.gsub(LITERALS_AND_COMMENTS) { |match| match.start_with?("'", "$") ? "?" : " " }.strip
|
|
68
|
+
return false unless sql.match?(/\ASELECT\b/i)
|
|
69
|
+
return false if sql.delete_suffix(";").include?(";")
|
|
70
|
+
|
|
71
|
+
!sql.match?(LOCKING_CLAUSE) && !sql.match?(DATA_MODIFYING)
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Masks every String inside a parsed EXPLAIN (FORMAT JSON) result with SqlStatement, leaving
|
|
75
|
+
# keys, numbers, booleans, arrays and the structure itself exactly as they were.
|
|
76
|
+
def mask_plan(value)
|
|
77
|
+
case value
|
|
78
|
+
when Hash then value.transform_values { |inner| mask_plan(inner) }
|
|
79
|
+
when Array then value.map { |inner| mask_plan(inner) }
|
|
80
|
+
when String then SqlStatement.mask(value) || value
|
|
81
|
+
else value
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def initialize(configuration, client: Client.new(configuration), clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
|
|
87
|
+
@configuration = configuration
|
|
88
|
+
@client = client
|
|
89
|
+
@clock = clock
|
|
90
|
+
@queue = SizedQueue.new(QUEUE_SIZE)
|
|
91
|
+
@mutex = Mutex.new
|
|
92
|
+
@thread = nil
|
|
93
|
+
@last_explained = {}
|
|
94
|
+
@recent = []
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# Called from the sql.active_record subscriber with that event's payload and duration. Cheap
|
|
98
|
+
# checks first, so a fast query costs one comparison. Returns true only when an EXPLAIN was
|
|
99
|
+
# queued; never raises.
|
|
100
|
+
def consider(payload, duration_ms:)
|
|
101
|
+
return false unless configuration.explain_slow_queries
|
|
102
|
+
return false unless duration_ms && duration_ms >= configuration.explain_threshold_ms.to_f
|
|
103
|
+
|
|
104
|
+
connection = payload[:connection]
|
|
105
|
+
return false unless PerformanceInstrumentation.db_system(connection) == "postgresql"
|
|
106
|
+
|
|
107
|
+
sql = payload[:sql]
|
|
108
|
+
return false unless self.class.explainable?(sql)
|
|
109
|
+
|
|
110
|
+
pool = connection.respond_to?(:pool) ? connection.pool : nil
|
|
111
|
+
return false unless pool
|
|
112
|
+
|
|
113
|
+
statement = SqlStatement.mask(sql)
|
|
114
|
+
return false unless reserve(statement)
|
|
115
|
+
|
|
116
|
+
enqueue(Job.new(
|
|
117
|
+
sql: sql.dup, binds: bind_values(payload), pool: pool, statement: statement,
|
|
118
|
+
duration_ms: duration_ms.to_f.round(1), captured_at: Time.now.utc.iso8601(3)
|
|
119
|
+
))
|
|
120
|
+
rescue StandardError => e
|
|
121
|
+
log { "EXPLAIN skipped: #{e.class}: #{e.message}" }
|
|
122
|
+
false
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
# Runs one queued EXPLAIN and delivers its plan. Normally called on the worker thread; public so
|
|
126
|
+
# a test can run a job synchronously. Never raises.
|
|
127
|
+
def perform(job)
|
|
128
|
+
plan = explain(job)
|
|
129
|
+
return false unless plan
|
|
130
|
+
|
|
131
|
+
masked_plan = self.class.mask_plan(plan)
|
|
132
|
+
if JSON.generate(masked_plan).bytesize > MAX_PLAN_BYTES
|
|
133
|
+
log { "EXPLAIN plan over #{MAX_PLAN_BYTES} bytes, dropped" }
|
|
134
|
+
return false
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
client.deliver_query_plan(
|
|
138
|
+
statement: job.statement, db_system: "postgresql", plan: masked_plan,
|
|
139
|
+
duration_ms: job.duration_ms, environment: configuration.environment, captured_at: job.captured_at
|
|
140
|
+
)
|
|
141
|
+
rescue StandardError => e
|
|
142
|
+
log { "EXPLAIN failed: #{e.class}: #{e.message}" }
|
|
143
|
+
false
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
private
|
|
147
|
+
attr_reader :configuration, :client
|
|
148
|
+
|
|
149
|
+
# Both rate limits at once, under one lock, so two threads finishing the same slow query
|
|
150
|
+
# together still only explain it once.
|
|
151
|
+
def reserve(statement)
|
|
152
|
+
return false if statement.nil?
|
|
153
|
+
|
|
154
|
+
@mutex.synchronize do
|
|
155
|
+
now = @clock.call
|
|
156
|
+
@recent.reject! { |at| now - at >= GLOBAL_WINDOW }
|
|
157
|
+
return false if @recent.size >= GLOBAL_LIMIT
|
|
158
|
+
|
|
159
|
+
last = @last_explained[statement]
|
|
160
|
+
return false if last && now - last < STATEMENT_INTERVAL
|
|
161
|
+
|
|
162
|
+
if @last_explained.size >= MAX_REMEMBERED_STATEMENTS
|
|
163
|
+
@last_explained.reject! { |_, at| now - at >= STATEMENT_INTERVAL }
|
|
164
|
+
end
|
|
165
|
+
@last_explained[statement] = now
|
|
166
|
+
@recent << now
|
|
167
|
+
true
|
|
168
|
+
end
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
# The values the app's own query was run with, already type cast for the database. Rails
|
|
172
|
+
# passes these either as an Array or as a lambda that builds one.
|
|
173
|
+
def bind_values(payload)
|
|
174
|
+
binds = payload[:type_casted_binds]
|
|
175
|
+
binds = binds.call if binds.respond_to?(:call)
|
|
176
|
+
Array(binds).dup
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
def enqueue(job)
|
|
180
|
+
ensure_worker_started
|
|
181
|
+
@queue.push(job, true)
|
|
182
|
+
true
|
|
183
|
+
rescue ThreadError
|
|
184
|
+
log { "EXPLAIN queue full, dropping" }
|
|
185
|
+
false
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
def ensure_worker_started
|
|
189
|
+
return if @thread&.alive?
|
|
190
|
+
|
|
191
|
+
@mutex.synchronize do
|
|
192
|
+
return if @thread&.alive?
|
|
193
|
+
|
|
194
|
+
@thread = Thread.new { run }
|
|
195
|
+
@thread.abort_on_exception = false
|
|
196
|
+
end
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
def run
|
|
200
|
+
Thread.current[:forge_ops_tracker_explaining] = true
|
|
201
|
+
loop do
|
|
202
|
+
perform(@queue.pop)
|
|
203
|
+
rescue StandardError => e
|
|
204
|
+
log { "EXPLAIN thread error: #{e.class}: #{e.message}" }
|
|
205
|
+
end
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
# The parsed plan (the JSON array Postgres returns), or nil. The connection is checked out
|
|
209
|
+
# explicitly rather than borrowed with with_connection, so it's never one this thread (or
|
|
210
|
+
# any other) already holds. Checked back in only once its transaction is rolled back; if
|
|
211
|
+
# even the ROLLBACK fails, the connection is removed from the pool and closed instead, so a
|
|
212
|
+
# connection in an unknown state is never handed to the app.
|
|
213
|
+
def explain(job)
|
|
214
|
+
connection = job.pool.checkout(CHECKOUT_TIMEOUT)
|
|
215
|
+
begin
|
|
216
|
+
connection.execute("BEGIN READ ONLY")
|
|
217
|
+
connection.execute("SET LOCAL statement_timeout = '#{STATEMENT_TIMEOUT}'")
|
|
218
|
+
result = connection.exec_query("EXPLAIN (FORMAT JSON) #{job.sql}", "EXPLAIN", job.binds)
|
|
219
|
+
value = result.rows.first&.first
|
|
220
|
+
value.is_a?(String) ? JSON.parse(value) : value
|
|
221
|
+
ensure
|
|
222
|
+
release(job.pool, connection, rollback(connection))
|
|
223
|
+
end
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
def rollback(connection)
|
|
227
|
+
connection.execute("ROLLBACK")
|
|
228
|
+
true
|
|
229
|
+
rescue StandardError => e
|
|
230
|
+
log { "EXPLAIN rollback failed: #{e.class}: #{e.message}" }
|
|
231
|
+
false
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
def release(pool, connection, healthy)
|
|
235
|
+
if healthy
|
|
236
|
+
pool.checkin(connection)
|
|
237
|
+
else
|
|
238
|
+
pool.remove(connection)
|
|
239
|
+
connection.disconnect!
|
|
240
|
+
end
|
|
241
|
+
rescue StandardError => e
|
|
242
|
+
log { "EXPLAIN connection release failed: #{e.class}: #{e.message}" }
|
|
243
|
+
end
|
|
244
|
+
|
|
245
|
+
def log
|
|
246
|
+
configuration.logger&.debug { "[ForgeOpsTracker] #{yield}" }
|
|
247
|
+
end
|
|
248
|
+
end
|
|
249
|
+
end
|
|
@@ -28,6 +28,11 @@ module ForgeOpsTracker
|
|
|
28
28
|
def span_queue
|
|
29
29
|
@span_queue ||= ForgeOpsTracker::SpanQueue.new(ForgeOpsTracker.configuration)
|
|
30
30
|
end
|
|
31
|
+
|
|
32
|
+
# One per process, so its rate limits (see QueryExplainer) are per process too.
|
|
33
|
+
def query_explainer
|
|
34
|
+
@query_explainer ||= ForgeOpsTracker::QueryExplainer.new(ForgeOpsTracker.configuration)
|
|
35
|
+
end
|
|
31
36
|
end
|
|
32
37
|
|
|
33
38
|
initializer "forge_ops_tracker.subscribe_error_reporter" do |app|
|
|
@@ -167,6 +172,9 @@ module ForgeOpsTracker
|
|
|
167
172
|
end
|
|
168
173
|
|
|
169
174
|
ActiveSupport::Notifications.subscribe("sql.active_record") do |*args|
|
|
175
|
+
# QueryExplainer's own EXPLAIN (and its BEGIN/SET LOCAL/ROLLBACK), on its own thread: not
|
|
176
|
+
# the app's query, so never timed, traced or explained itself.
|
|
177
|
+
next if QueryExplainer.explaining?
|
|
170
178
|
next unless configuration.enabled?
|
|
171
179
|
|
|
172
180
|
event = ActiveSupport::Notifications::Event.new(*args)
|
|
@@ -184,13 +192,19 @@ module ForgeOpsTracker
|
|
|
184
192
|
# A leaf span, parented onto whatever's currently open (the controller root, or a
|
|
185
193
|
# manually-wrapped ForgeOpsTracker.span block around it): a query has no children of its
|
|
186
194
|
# own to nest anything under, so unlike the controller root above, this never touches the
|
|
187
|
-
# stack itself, just records against whatever's already on top of it.
|
|
195
|
+
# stack itself, just records against whatever's already on top of it. Carries the masked
|
|
196
|
+
# SQL and the database system (see PerformanceInstrumentation.database_span_data).
|
|
188
197
|
if configuration.track_tracing && (buffer = Thread.current[:forge_ops_tracker_spans])
|
|
189
198
|
buffer.record(
|
|
190
199
|
span_id: SecureRandom.hex(8), name: transaction_name, kind: "database",
|
|
191
|
-
started_at: Time.at(event.time), duration_ms: event.duration
|
|
200
|
+
started_at: Time.at(event.time), duration_ms: event.duration,
|
|
201
|
+
data: PerformanceInstrumentation.database_span_data(event.payload)
|
|
192
202
|
)
|
|
193
203
|
end
|
|
204
|
+
|
|
205
|
+
# Opt-in, PostgreSQL only; a no-op unless config.explain_slow_queries is on. Only reads the
|
|
206
|
+
# payload here; the EXPLAIN itself runs later on QueryExplainer's own thread and connection.
|
|
207
|
+
Railtie.query_explainer.consider(event.payload, duration_ms: event.duration) if configuration.explain_slow_queries
|
|
194
208
|
end
|
|
195
209
|
|
|
196
210
|
ActiveSupport::Notifications.subscribe("perform.active_job") do |*args|
|
data/lib/forge_ops_tracker.rb
CHANGED
|
@@ -20,6 +20,7 @@ require "forge_ops_tracker/periodic_poller"
|
|
|
20
20
|
require "forge_ops_tracker/breadcrumb_buffer"
|
|
21
21
|
require "forge_ops_tracker/span_buffer"
|
|
22
22
|
require "forge_ops_tracker/span_queue"
|
|
23
|
+
require "forge_ops_tracker/query_explainer"
|
|
23
24
|
require "forge_ops_tracker/trace_parent"
|
|
24
25
|
require "forge_ops_tracker/request_state"
|
|
25
26
|
require "forge_ops_tracker/middleware/session_tracking"
|
|
@@ -128,12 +129,18 @@ module ForgeOpsTracker
|
|
|
128
129
|
# a standalone buffer for itself: a lone span with no request-level trace around it, and no
|
|
129
130
|
# middleware left running to ever flush it, would just accumulate in this thread's memory
|
|
130
131
|
# forever with nothing to ever send it, which is worse than never recording it at all.
|
|
131
|
-
|
|
132
|
+
#
|
|
133
|
+
# A span with kind: "database" can also carry the SQL it ran and which database it ran on, e.g.
|
|
134
|
+
# `ForgeOpsTracker.span("Load orders", kind: "database", statement: sql, db_system: "postgresql")`.
|
|
135
|
+
# The statement is masked by SqlStatement before it's kept (every string and number becomes
|
|
136
|
+
# "?"), so no value it was run with is ever sent. Both are ignored on any other kind of span.
|
|
137
|
+
def span(name, kind: "service", data: {}, statement: nil, db_system: nil)
|
|
132
138
|
return yield unless configuration.enabled? && configuration.track_tracing
|
|
133
139
|
|
|
134
140
|
buffer = Thread.current[:forge_ops_tracker_spans]
|
|
135
141
|
return yield unless buffer
|
|
136
142
|
|
|
143
|
+
data = database_span_data(data, statement, db_system) if kind.to_s == "database"
|
|
137
144
|
span_id = SecureRandom.hex(8)
|
|
138
145
|
started_at = Time.now.utc
|
|
139
146
|
buffer.push(span_id)
|
|
@@ -147,6 +154,14 @@ module ForgeOpsTracker
|
|
|
147
154
|
end
|
|
148
155
|
|
|
149
156
|
private
|
|
157
|
+
def database_span_data(data, statement, db_system)
|
|
158
|
+
extra = {
|
|
159
|
+
"db.statement" => (SqlStatement.mask(statement.to_s) unless statement.nil?),
|
|
160
|
+
"db.system" => (db_system.to_s.strip.downcase unless db_system.nil? || db_system.to_s.strip.empty?)
|
|
161
|
+
}.compact
|
|
162
|
+
extra.empty? ? data : data.merge(extra)
|
|
163
|
+
end
|
|
164
|
+
|
|
150
165
|
def change_queue
|
|
151
166
|
@change_queue ||= DeliveryQueue.new(configuration, deliver_with: :deliver_change)
|
|
152
167
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: forge_ops_tracker
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.14.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- ForgeOps
|
|
@@ -116,6 +116,7 @@ files:
|
|
|
116
116
|
- lib/forge_ops_tracker/performance_instrumentation.rb
|
|
117
117
|
- lib/forge_ops_tracker/periodic_poller.rb
|
|
118
118
|
- lib/forge_ops_tracker/pii_scrubber.rb
|
|
119
|
+
- lib/forge_ops_tracker/query_explainer.rb
|
|
119
120
|
- lib/forge_ops_tracker/railtie.rb
|
|
120
121
|
- lib/forge_ops_tracker/request_state.rb
|
|
121
122
|
- lib/forge_ops_tracker/session_flusher.rb
|