visionary-sdk 0.5.1 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +103 -7
- data/lib/visionary/agent.rb +6 -1
- data/lib/visionary/config.rb +9 -5
- data/lib/visionary/context.rb +16 -1
- data/lib/visionary/middleware.rb +11 -1
- data/lib/visionary/profiler.rb +522 -0
- data/lib/visionary/retention.rb +4 -1
- data/lib/visionary/schema.rb +5 -1
- data/lib/visionary/store/active_record.rb +5 -0
- data/lib/visionary/store/d1.rb +7 -0
- data/lib/visionary/store/sql.rb +19 -1
- data/lib/visionary/store.rb +1 -1
- data/lib/visionary/telemetry/sql_text.rb +1 -1
- data/lib/visionary/telemetry.rb +8 -4
- data/lib/visionary/version.rb +1 -1
- data/lib/visionary/writer.rb +11 -3
- data/lib/visionary.rb +26 -1
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a7d2e913847c653919ec1f345af3a86f40965707c686e25bd07045e76aff0161
|
|
4
|
+
data.tar.gz: 46ac3f49d03c1905f0a33c7befcb0f5b8e807b48f47b50629955a3b84eba28d8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2061e3a2855102aac59fb1b6b6e4c73f12afdb0aacee73132b4b6196f3264f119bd741a8ca02f68e3412c6466e893b4f836b5fa6142c34f07084f9a2349f739b
|
|
7
|
+
data.tar.gz: aeade39d1f5076bd7185e57ac1f0976cf776948526e454d61f88e5c1c28de6a2a323439470d324ea99583b3421cbb2be4f518adc85fb672dfefb9142f126d461
|
data/README.md
CHANGED
|
@@ -8,7 +8,9 @@ or Rails service and writes:
|
|
|
8
8
|
database and outbound HTTP calls the request made),
|
|
9
9
|
- `spans` for database statements and outbound HTTP calls of slow or failed
|
|
10
10
|
requests, `errors` for exceptions, and a `releases` row per deploy,
|
|
11
|
-
- a periodic host / process / runtime sample into `system_metrics
|
|
11
|
+
- a periodic host / process / runtime sample into `system_metrics`,
|
|
12
|
+
- CPU profiles into `profiles`, only while they are turned on in the
|
|
13
|
+
dashboard.
|
|
12
14
|
|
|
13
15
|
The Visionary server reads these tables for its dashboards. All slow work
|
|
14
16
|
happens on background threads, so requests are never blocked or slowed. If
|
|
@@ -87,6 +89,7 @@ Visionary.configure { |c| c.route = ->(env) { env["my.router.pattern"] } }
|
|
|
87
89
|
Visionary.set_user(id, handle: nil, role: nil) # the signed-in user of the current request
|
|
88
90
|
Visionary.set_route("/users/:id") # route pattern, for routers Visionary can't see
|
|
89
91
|
Visionary.span("charge card", kind: "http", target: "api.stripe.com") { ... } # a manual span
|
|
92
|
+
Visionary.background("send welcome email") { ... } # a thread that stays with the request
|
|
90
93
|
Visionary.capture_exception(e) # a handled error (handled = 1)
|
|
91
94
|
Visionary.start # start now instead of on the first request
|
|
92
95
|
Visionary.close(10) # flush; also runs at exit
|
|
@@ -100,6 +103,12 @@ user per batch, for users missing a handle or role.
|
|
|
100
103
|
|
|
101
104
|
## Spans and errors
|
|
102
105
|
|
|
106
|
+
The middleware records the request; the calls inside it are recorded only when
|
|
107
|
+
their client is hooked (see below). Wrap the rest (a Redis or queue client
|
|
108
|
+
without a hook, CPU work such as image processing) in `Visionary.span`.
|
|
109
|
+
Anything left shows as **Untracked** in the request's breakdown; see
|
|
110
|
+
[What the middleware sees](../../Readme.md#what-the-middleware-sees-and-what-needs-code).
|
|
111
|
+
|
|
103
112
|
- **Database:** every `sql.active_record` notification except `SCHEMA` and
|
|
104
113
|
cached queries becomes a `db` span named after the statement (`SELECT
|
|
105
114
|
users`, `INSERT bookings`, `BEGIN`), with the statement sanitized (literals
|
|
@@ -111,10 +120,9 @@ user per batch, for users missing a handle or role.
|
|
|
111
120
|
|
|
112
121
|
Every logged request gets `db_count`, `db_ms`, `http_count`, `http_ms`
|
|
113
122
|
(`db_*` are NULL when ActiveRecord isn't loaded and no `db` span was
|
|
114
|
-
recorded). Span rows are stored
|
|
115
|
-
`
|
|
116
|
-
|
|
117
|
-
threshold themselves. Visionary's own writes never become spans.
|
|
123
|
+
recorded). Span rows are stored for every request, at most
|
|
124
|
+
`VISIONARY_SPAN_MAX` per request; spans outside requests are stored when they
|
|
125
|
+
take at least `VISIONARY_SPAN_THRESHOLD_MS` themselves. Visionary's own writes never become spans.
|
|
118
126
|
|
|
119
127
|
Exceptions that escape the app are recorded and re-raised unchanged; the
|
|
120
128
|
middleware logs the request as 500. Errors are grouped by a fingerprint over
|
|
@@ -122,12 +130,95 @@ the exception class and the top five frames of your own code (not gems, the
|
|
|
122
130
|
standard library or this SDK), as `method@path` relative to the working
|
|
123
131
|
directory, so it is stable across deploys and line changes.
|
|
124
132
|
|
|
133
|
+
## Background work
|
|
134
|
+
|
|
135
|
+
Work a handler starts and doesn't wait for still belongs to its request. Start
|
|
136
|
+
it with `Visionary.background`, which runs the block in a new thread and
|
|
137
|
+
returns the `Thread`:
|
|
138
|
+
|
|
139
|
+
```ruby
|
|
140
|
+
Visionary.background("send welcome email") { UserMailer.welcome(user).deliver_now }
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
- The block is one span of kind `background` with the name you give, from
|
|
144
|
+
start to finish. If it raises, the span is marked `error` with the message
|
|
145
|
+
and the exception is re-raised in the thread as Ruby normally does (printed
|
|
146
|
+
when `Thread.report_on_exception` is on, raised again by `join`/`value`).
|
|
147
|
+
- Calls made inside it keep the request's `request_id`, and their `offset_ms`
|
|
148
|
+
counts from the request's start, even after the response was sent. Calls
|
|
149
|
+
that end after the request was logged don't change its `db_*`/`http_*`
|
|
150
|
+
totals. They are stored whatever their duration, up to
|
|
151
|
+
`VISIONARY_SPAN_MAX` of them per request; after that only those taking at
|
|
152
|
+
least `VISIONARY_SPAN_THRESHOLD_MS`. `background` spans never count in the
|
|
153
|
+
totals.
|
|
154
|
+
- Outside a request the `background` span has no `request_id` and is stored
|
|
155
|
+
only when it takes at least `VISIONARY_SPAN_THRESHOLD_MS`.
|
|
156
|
+
|
|
157
|
+
On Ruby 3.2+ any thread or fiber started during a request inherits it, so
|
|
158
|
+
calls from a plain `Thread.new` are attributed the same way; `background`
|
|
159
|
+
also carries the request over on 3.1 and adds the span around the work.
|
|
160
|
+
|
|
161
|
+
## CPU profiles
|
|
162
|
+
|
|
163
|
+
Profiling is **off until you turn it on in the dashboard** (Settings →
|
|
164
|
+
Profiling), and you can turn it off there again at any time. Neither needs a
|
|
165
|
+
redeploy. The SDK reads the switch from the `visionary_settings` table when it
|
|
166
|
+
starts and every 30 s after that, on its own thread. It logs one line when
|
|
167
|
+
profiling turns on or off. It follows the dashboard's service list
|
|
168
|
+
(`VISIONARY_SERVICE`), interval (default every 300 s) and duration (default
|
|
169
|
+
10 s), and "Profile now" takes one profile within 30 s. To keep a service out
|
|
170
|
+
of profiling whatever the dashboard says, set `VISIONARY_PROFILE=false`.
|
|
171
|
+
|
|
172
|
+
While a profile runs, a thread wakes about every 10 ms and takes the stack of
|
|
173
|
+
each app thread whose status is `run`. Threads that sleep, wait on a lock or
|
|
174
|
+
queue, or block on IO (a database or HTTP call) have the status `sleep`, so
|
|
175
|
+
they are left out, and so are Visionary's own threads. One row is written per
|
|
176
|
+
profile:
|
|
177
|
+
|
|
178
|
+
- `folded`: one `frame;frame;... count` line per distinct stack, outermost
|
|
179
|
+
call first. Each sample counts about 10 ms of CPU. Frames read
|
|
180
|
+
`Class#method file:line`. Your code's paths are relative to the working
|
|
181
|
+
directory. Gems are shown as `lib:<gem>/<path>`, e.g.
|
|
182
|
+
`lib:rack/rack/builder.rb`, and the standard library as `std:<path>`, e.g.
|
|
183
|
+
`std:json/common.rb` or `std:kernel.rb` for Ruby's built-in Ruby code.
|
|
184
|
+
Each row is capped at 90 000 bytes, and the lightest stacks are summed into
|
|
185
|
+
one `(other)` line.
|
|
186
|
+
- `routes`: the milliseconds that the requests finishing during the profile
|
|
187
|
+
took, per `METHOD route`.
|
|
188
|
+
|
|
189
|
+
A turn is skipped when no request arrived since the last one. A profile is
|
|
190
|
+
dropped when no request finished during it, or when it has fewer than 20
|
|
191
|
+
samples (about 200 ms of CPU).
|
|
192
|
+
|
|
193
|
+
**Cost:** nothing while profiling is off apart from the settings read every
|
|
194
|
+
30 s. While a profile runs, the sampler takes some tens of stack walks per
|
|
195
|
+
second, which costs a few percent of one CPU.
|
|
196
|
+
|
|
197
|
+
**How accurate it is.** Ruby has no built-in CPU sampler, so these are
|
|
198
|
+
wall-clock samples of threads that are runnable:
|
|
199
|
+
|
|
200
|
+
- Only the thread holding the GVL really runs. Several busy threads are each
|
|
201
|
+
counted, so time spent waiting for the GVL counts as CPU.
|
|
202
|
+
- A thread computing in Ruby keeps the GVL for up to 100 ms while the sampler
|
|
203
|
+
waits for it. A request that computes for 40 ms and then writes its answer
|
|
204
|
+
would rarely be caught computing. So while a profile runs, the SDK lowers
|
|
205
|
+
the app's threads to Ruby's lowest thread priority (`-3`, a 12.5 ms
|
|
206
|
+
quantum) and gives each thread its own priority back afterwards. In CRuby,
|
|
207
|
+
priority sets only this quantum, so no thread gains on another. A late
|
|
208
|
+
sample counts for the 10 ms ticks it missed. Code that gives up the GVL
|
|
209
|
+
within a few ms (short bursts between IO calls) is still under-counted.
|
|
210
|
+
- A C extension that releases the GVL to do work (some database drivers,
|
|
211
|
+
compression) keeps the status `run`. Its time shows on the Ruby frame that
|
|
212
|
+
called it.
|
|
213
|
+
- C methods (`Array#each`, `Time#strftime`) show the file and line of the Ruby
|
|
214
|
+
code that called them.
|
|
215
|
+
|
|
125
216
|
## Forking servers (Puma, Unicorn)
|
|
126
217
|
|
|
127
218
|
Threads don't survive `fork`. Visionary hooks `Process._fork` (Ruby 3.1+), so
|
|
128
219
|
when Puma or Unicorn forks workers after the agent started (for example with
|
|
129
220
|
`preload_app!` and an initializer that calls `Visionary.start`), each worker
|
|
130
|
-
restarts its writer, sampler and retention threads with fresh queues and
|
|
221
|
+
restarts its writer, sampler, profiler and retention threads with fresh queues and
|
|
131
222
|
connections. If you prefer to be explicit, Puma's hook does the same and is
|
|
132
223
|
safe to combine with it:
|
|
133
224
|
|
|
@@ -187,7 +278,7 @@ plus the hooks above. Set it before the first request.
|
|
|
187
278
|
| `VISIONARY_LOG_REQUESTS`, `VISIONARY_LOG_BODIES` | `true` |
|
|
188
279
|
| `VISIONARY_LOG_PATH_PREFIX` | everything |
|
|
189
280
|
| `VISIONARY_LOG_MAX_BODY_BYTES` | `8192` |
|
|
190
|
-
| `VISIONARY_LOG_RETENTION_DAYS`, `VISIONARY_SYSTEM_RETENTION_DAYS` | `
|
|
281
|
+
| `VISIONARY_LOG_RETENTION_DAYS`, `VISIONARY_SYSTEM_RETENTION_DAYS` | `0` (off; above 0 the SDK also deletes older rows) |
|
|
191
282
|
| `VISIONARY_LOG_REDACT_HEADERS`, `_REDACT_FIELDS`, `_REDACT_REQUEST_FIELDS` | added to the defaults |
|
|
192
283
|
| `VISIONARY_LOG_ERROR_CODE_FIELD`, `_ERROR_KEY_FIELD` | `error.code`, `error.key` |
|
|
193
284
|
| `VISIONARY_SYSTEM_METRICS` | `true` |
|
|
@@ -196,6 +287,11 @@ plus the hooks above. Set it before the first request.
|
|
|
196
287
|
| `VISIONARY_RELEASE` | the platform's commit variable, shortened to 12 characters |
|
|
197
288
|
| `VISIONARY_SPANS`, `VISIONARY_ERRORS` | `true` |
|
|
198
289
|
| `VISIONARY_SPAN_THRESHOLD_MS`, `VISIONARY_SPAN_MAX` | `100`, `100` |
|
|
290
|
+
| `VISIONARY_PROFILE` | `true`: follow the dashboard's Profiling setting; `false` never profiles |
|
|
291
|
+
|
|
292
|
+
Old rows are deleted by the Visionary server, following Settings → Retention in
|
|
293
|
+
the dashboard. The two `*_RETENTION_DAYS` variables only matter when the
|
|
294
|
+
service writes to a database no Visionary server reads.
|
|
199
295
|
|
|
200
296
|
Request bodies are captured as the app reads them, so a body the app never
|
|
201
297
|
reads is not stored (its size still is, from `Content-Length`).
|
data/lib/visionary/agent.rb
CHANGED
|
@@ -41,9 +41,10 @@ module Visionary
|
|
|
41
41
|
|
|
42
42
|
@instance = cfg.instance.to_s.empty? ? Socket.gethostname : cfg.instance
|
|
43
43
|
@schemas = Schemas.new(@store, @logger)
|
|
44
|
-
if cfg.log_requests || cfg.spans || cfg.errors || cfg.version
|
|
44
|
+
if cfg.log_requests || cfg.spans || cfg.errors || cfg.version || cfg.profile
|
|
45
45
|
@writer = Writer.new(cfg, @store, @schemas, @logger)
|
|
46
46
|
end
|
|
47
|
+
@profiler = Profiler.new(cfg, @instance, @store, @writer, @logger) if cfg.profile && @writer
|
|
47
48
|
@telemetry = Telemetry::Recorder.new(cfg, @instance, @writer) if cfg.spans || cfg.errors
|
|
48
49
|
if cfg.system_metrics && cfg.system_interval.to_f.positive?
|
|
49
50
|
@system = System::Recorder.new(cfg, @instance, @store, @schemas, @http, @logger)
|
|
@@ -55,6 +56,7 @@ module Visionary
|
|
|
55
56
|
tables["errors"] = cfg.log_retention_days
|
|
56
57
|
end
|
|
57
58
|
tables["system_metrics"] = cfg.system_retention_days if @system
|
|
59
|
+
tables["profiles"] = cfg.log_retention_days if @profiler
|
|
58
60
|
@retention = Retention.new(@store, @logger, tables)
|
|
59
61
|
end
|
|
60
62
|
|
|
@@ -93,6 +95,7 @@ module Visionary
|
|
|
93
95
|
@logger.info("visionary: recording system metrics every #{@config.system_interval}s as #{@instance}")
|
|
94
96
|
end
|
|
95
97
|
@retention.start
|
|
98
|
+
@profiler&.start
|
|
96
99
|
install_hooks
|
|
97
100
|
release
|
|
98
101
|
self
|
|
@@ -107,6 +110,7 @@ module Visionary
|
|
|
107
110
|
end
|
|
108
111
|
deadline = mono + timeout
|
|
109
112
|
@retention&.close
|
|
113
|
+
@profiler&.close
|
|
110
114
|
idle = @writer ? @writer.close(timeout) : true
|
|
111
115
|
idle = @system.close([deadline - mono, 0].max) && idle if @system
|
|
112
116
|
@store.close if idle && @store
|
|
@@ -127,6 +131,7 @@ module Visionary
|
|
|
127
131
|
@system&.after_fork
|
|
128
132
|
@writer&.after_fork
|
|
129
133
|
@retention&.after_fork if @retention&.needed?
|
|
134
|
+
@profiler&.after_fork
|
|
130
135
|
@logger.info("visionary: restarted in forked process #{@pid}")
|
|
131
136
|
end
|
|
132
137
|
|
data/lib/visionary/config.rb
CHANGED
|
@@ -21,7 +21,8 @@ module Visionary
|
|
|
21
21
|
:redact_headers, :redact_fields, :redact_request_fields,
|
|
22
22
|
:error_code_field, :error_key_field,
|
|
23
23
|
:system_metrics, :system_interval, :system_retention_days, :disk_path,
|
|
24
|
-
:service, :version, :spans, :span_threshold_ms, :span_max, :errors
|
|
24
|
+
:service, :version, :spans, :span_threshold_ms, :span_max, :errors,
|
|
25
|
+
:profile
|
|
25
26
|
|
|
26
27
|
# Runs on the writer thread, once per distinct user per batch, for users
|
|
27
28
|
# missing a handle or role: ->(id) { [handle, role] } or nil.
|
|
@@ -47,7 +48,7 @@ module Visionary
|
|
|
47
48
|
@queue_size = 5000
|
|
48
49
|
@batch_size = 50
|
|
49
50
|
@flush_interval = 3.0
|
|
50
|
-
@log_retention_days =
|
|
51
|
+
@log_retention_days = 0 # 0: off, the Visionary server applies Settings → Retention
|
|
51
52
|
@redact_headers = []
|
|
52
53
|
@redact_fields = []
|
|
53
54
|
@redact_request_fields = []
|
|
@@ -55,7 +56,7 @@ module Visionary
|
|
|
55
56
|
@error_key_field = "error.key"
|
|
56
57
|
@system_metrics = true
|
|
57
58
|
@system_interval = 15.0
|
|
58
|
-
@system_retention_days =
|
|
59
|
+
@system_retention_days = 0 # 0: off, as above
|
|
59
60
|
@disk_path = ""
|
|
60
61
|
@service = nil
|
|
61
62
|
@version = nil
|
|
@@ -63,6 +64,8 @@ module Visionary
|
|
|
63
64
|
@span_threshold_ms = 100.0
|
|
64
65
|
@span_max = 100
|
|
65
66
|
@errors = true
|
|
67
|
+
# false: never profile, whatever the dashboard says.
|
|
68
|
+
@profile = true
|
|
66
69
|
end
|
|
67
70
|
|
|
68
71
|
def self.from_env(env = ENV)
|
|
@@ -78,7 +81,7 @@ module Visionary
|
|
|
78
81
|
c.path_prefix = s.("VISIONARY_LOG_PATH_PREFIX")
|
|
79
82
|
c.capture_bodies = parse_bool(s.("VISIONARY_LOG_BODIES"), true)
|
|
80
83
|
c.max_body_bytes = parse_int(s.("VISIONARY_LOG_MAX_BODY_BYTES"), 8192)
|
|
81
|
-
c.log_retention_days = parse_int(s.("VISIONARY_LOG_RETENTION_DAYS"),
|
|
84
|
+
c.log_retention_days = parse_int(s.("VISIONARY_LOG_RETENTION_DAYS"), 0)
|
|
82
85
|
c.redact_headers = parse_list(s.("VISIONARY_LOG_REDACT_HEADERS"))
|
|
83
86
|
c.redact_fields = parse_list(s.("VISIONARY_LOG_REDACT_FIELDS"))
|
|
84
87
|
c.redact_request_fields = parse_list(s.("VISIONARY_LOG_REDACT_REQUEST_FIELDS"))
|
|
@@ -86,13 +89,14 @@ module Visionary
|
|
|
86
89
|
c.error_key_field = s.("VISIONARY_LOG_ERROR_KEY_FIELD").then { |v| v.empty? ? "error.key" : v }
|
|
87
90
|
c.system_metrics = parse_bool(s.("VISIONARY_SYSTEM_METRICS"), true)
|
|
88
91
|
c.system_interval = parse_duration(s.("VISIONARY_SYSTEM_INTERVAL"), 15.0)
|
|
89
|
-
c.system_retention_days = parse_int(s.("VISIONARY_SYSTEM_RETENTION_DAYS"),
|
|
92
|
+
c.system_retention_days = parse_int(s.("VISIONARY_SYSTEM_RETENTION_DAYS"), 0)
|
|
90
93
|
c.service = s.("VISIONARY_SERVICE").then { |v| v.empty? ? nil : v }
|
|
91
94
|
c.version = release_from(env)
|
|
92
95
|
c.spans = parse_bool(s.("VISIONARY_SPANS"), true)
|
|
93
96
|
c.span_threshold_ms = parse_float(s.("VISIONARY_SPAN_THRESHOLD_MS"), 100.0)
|
|
94
97
|
c.span_max = parse_int(s.("VISIONARY_SPAN_MAX"), 100)
|
|
95
98
|
c.errors = parse_bool(s.("VISIONARY_ERRORS"), true)
|
|
99
|
+
c.profile = parse_bool(s.("VISIONARY_PROFILE"), true)
|
|
96
100
|
c
|
|
97
101
|
end
|
|
98
102
|
|
data/lib/visionary/context.rb
CHANGED
|
@@ -28,6 +28,8 @@ module Visionary
|
|
|
28
28
|
@errors = []
|
|
29
29
|
@db_count = @http_count = 0
|
|
30
30
|
@db_ms = @http_ms = 0.0
|
|
31
|
+
@logged = false
|
|
32
|
+
@late = 0
|
|
31
33
|
end
|
|
32
34
|
|
|
33
35
|
def set_user(id, handle, role)
|
|
@@ -47,8 +49,14 @@ module Visionary
|
|
|
47
49
|
end
|
|
48
50
|
|
|
49
51
|
# Database and HTTP calls always count; rows are kept up to the cap.
|
|
52
|
+
# Returns nil when the span joined the request, or, once the request has
|
|
53
|
+
# been logged (work it started in the background), how many such late
|
|
54
|
+
# spans the request has had, this one included: the caller writes it on
|
|
55
|
+
# its own and it never touches the request's totals.
|
|
50
56
|
def add_span(span)
|
|
51
57
|
@mutex.synchronize do
|
|
58
|
+
return @late += 1 if @logged
|
|
59
|
+
|
|
52
60
|
case span.kind
|
|
53
61
|
when "db"
|
|
54
62
|
@db_count += 1
|
|
@@ -59,9 +67,16 @@ module Visionary
|
|
|
59
67
|
@http_ms += span.duration_ms
|
|
60
68
|
end
|
|
61
69
|
@spans << span if @spans.size < @span_max
|
|
70
|
+
nil
|
|
62
71
|
end
|
|
63
72
|
end
|
|
64
73
|
|
|
74
|
+
# Called when the request's row is being built: spans that end from now
|
|
75
|
+
# on are late and no longer change totals or spans.
|
|
76
|
+
def logged!
|
|
77
|
+
@mutex.synchronize { @logged = true }
|
|
78
|
+
end
|
|
79
|
+
|
|
65
80
|
def add_error(row)
|
|
66
81
|
@mutex.synchronize { @errors << row }
|
|
67
82
|
end
|
|
@@ -91,7 +106,7 @@ module Visionary
|
|
|
91
106
|
end
|
|
92
107
|
end
|
|
93
108
|
|
|
94
|
-
# One database, HTTP, cache, queue or
|
|
109
|
+
# One database, HTTP, cache, queue, custom, graphql or background call.
|
|
95
110
|
Span = Struct.new(:kind, :name, :target, :detail, :started_at, :offset_ms, :duration_ms,
|
|
96
111
|
:status, :error, :row_count, :request_id, keyword_init: true)
|
|
97
112
|
end
|
data/lib/visionary/middleware.rb
CHANGED
|
@@ -71,6 +71,8 @@ module Visionary
|
|
|
71
71
|
# Not logged: count it, and still record what escapes.
|
|
72
72
|
def passthrough(agent, env)
|
|
73
73
|
agent.http.started
|
|
74
|
+
Profiler::Tally.started!
|
|
75
|
+
t0 = Telemetry.now_ms
|
|
74
76
|
status = 500
|
|
75
77
|
begin
|
|
76
78
|
res = @app.call(env)
|
|
@@ -81,6 +83,11 @@ module Visionary
|
|
|
81
83
|
raise
|
|
82
84
|
ensure
|
|
83
85
|
agent.http.finished(status)
|
|
86
|
+
if Profiler::Tally.busy?
|
|
87
|
+
route = agent.route_for(env)
|
|
88
|
+
route = "#{env['SCRIPT_NAME']}#{env['PATH_INFO']}" if route.empty?
|
|
89
|
+
Profiler::Tally.finished(env["REQUEST_METHOD"], route, Telemetry.now_ms - t0)
|
|
90
|
+
end
|
|
84
91
|
end
|
|
85
92
|
end
|
|
86
93
|
|
|
@@ -102,6 +109,7 @@ module Visionary
|
|
|
102
109
|
started_mono: t0, span_max: cfg.span_max.to_i, db_visible: Telemetry::ActiveRecordHook.installed?
|
|
103
110
|
)
|
|
104
111
|
agent.http.started
|
|
112
|
+
Profiler::Tally.started!
|
|
105
113
|
outer = RequestContext.current
|
|
106
114
|
RequestContext.current = ctx
|
|
107
115
|
begin
|
|
@@ -133,9 +141,11 @@ module Visionary
|
|
|
133
141
|
|
|
134
142
|
def finish(agent, env, ctx, headers, req_capture, status, res_headers, resp_capture, t0)
|
|
135
143
|
agent.http.finished(status)
|
|
144
|
+
ctx.logged!
|
|
136
145
|
duration = (Telemetry.now_ms - t0).round(3)
|
|
137
146
|
route = ctx.route
|
|
138
147
|
route = agent.route_for(env) if route.empty?
|
|
148
|
+
Profiler::Tally.finished(ctx.method, route.empty? ? ctx.path : route, duration)
|
|
139
149
|
user_id, handle, role = ctx.user
|
|
140
150
|
tel = agent.telemetry
|
|
141
151
|
entry = Entry.new(
|
|
@@ -150,7 +160,7 @@ module Visionary
|
|
|
150
160
|
response_body: resp_capture.bytes, response_truncated: resp_capture.truncated?,
|
|
151
161
|
response_size: [self.class.header(res_headers, "content-length").to_i, resp_capture.size].max,
|
|
152
162
|
totals: tel&.spans_on ? ctx.totals : [nil, nil, nil, nil],
|
|
153
|
-
span_rows: tel ? tel.request_span_rows(ctx
|
|
163
|
+
span_rows: tel ? tel.request_span_rows(ctx) : [],
|
|
154
164
|
error_rows: tel ? ctx.errors.map { |row| tel.error_row(row, ctx, route, user_id) } : []
|
|
155
165
|
)
|
|
156
166
|
agent.enqueue([:request, entry])
|
|
@@ -0,0 +1,522 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "rbconfig"
|
|
5
|
+
|
|
6
|
+
module Visionary
|
|
7
|
+
# CPU profiles, switched on and off from the dashboard (Settings ->
|
|
8
|
+
# Profiling, kept in visionary_settings). One background thread reads the
|
|
9
|
+
# settings every 30 s and, while profiling is on, samples the stacks of the
|
|
10
|
+
# threads that are running every 10 ms for profile.duration_ms every
|
|
11
|
+
# profile.interval_s. Each profile is one profiles row: folded stacks plus
|
|
12
|
+
# the time each route's requests took while it ran.
|
|
13
|
+
#
|
|
14
|
+
# Ruby has no CPU-time sampler without a native extension, so this walks
|
|
15
|
+
# Thread.list and keeps the threads whose status is "run". Threads that
|
|
16
|
+
# sleep, wait on a lock or queue, or block on IO report "sleep" and are
|
|
17
|
+
# left out. Of the "run" threads only one holds the GVL at a time; the
|
|
18
|
+
# others are waiting for it (or inside a C call that released it), so with
|
|
19
|
+
# several busy threads each is counted, which over-counts waiting time.
|
|
20
|
+
# Each sample stands for about 10 ms: a tick that comes late because a
|
|
21
|
+
# busy thread held the GVL counts once per 10 ms it covers.
|
|
22
|
+
class Profiler
|
|
23
|
+
POLL_S = 30
|
|
24
|
+
MAX_FOLDED = 90_000
|
|
25
|
+
KEEP_WHOLE = 72_000 # the heaviest stacks kept whole; the rest roll up (see fold)
|
|
26
|
+
MIN_SAMPLES = 20
|
|
27
|
+
TICK_S = 0.01
|
|
28
|
+
NOW_FRESH_S = 300
|
|
29
|
+
DEFAULT_INTERVAL_S = 300.0
|
|
30
|
+
DEFAULT_DURATION_MS = 10_000.0
|
|
31
|
+
MAX_DURATION_MS = 120_000.0
|
|
32
|
+
SETTINGS_SQL = "SELECT name, value FROM visionary_settings WHERE name LIKE 'profile.%'"
|
|
33
|
+
|
|
34
|
+
# Requests seen by the middleware: how many started (a turn with none
|
|
35
|
+
# new is skipped) and, while a profile runs, the ms each "METHOD route"
|
|
36
|
+
# spent on requests that finished.
|
|
37
|
+
module Tally
|
|
38
|
+
@mutex = Mutex.new
|
|
39
|
+
@started = 0
|
|
40
|
+
@busy = nil
|
|
41
|
+
|
|
42
|
+
class << self
|
|
43
|
+
attr_reader :started
|
|
44
|
+
|
|
45
|
+
# Under the GVL a lost increment is possible but harmless: only
|
|
46
|
+
# "did it change" matters.
|
|
47
|
+
def started!
|
|
48
|
+
@started += 1
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def busy? = !@busy.nil?
|
|
52
|
+
|
|
53
|
+
def finished(method, route, ms)
|
|
54
|
+
return unless @busy
|
|
55
|
+
key = "#{method} #{route}"
|
|
56
|
+
@mutex.synchronize { @busy[key] = (@busy[key] || 0.0) + ms.to_f if @busy }
|
|
57
|
+
rescue StandardError
|
|
58
|
+
nil
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def begin!
|
|
62
|
+
@mutex.synchronize { @busy = {} }
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
# The busy ms since begin!, and stops counting.
|
|
66
|
+
def finish!
|
|
67
|
+
@mutex.synchronize do
|
|
68
|
+
out = @busy || {}
|
|
69
|
+
@busy = nil
|
|
70
|
+
out
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def after_fork
|
|
75
|
+
@mutex = Mutex.new
|
|
76
|
+
@busy = nil
|
|
77
|
+
end
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
Settings = Struct.new(:enabled, :services, :interval_s, :duration_ms, :now, keyword_init: true)
|
|
82
|
+
|
|
83
|
+
class << self
|
|
84
|
+
# visionary_settings rows ([[name, value], ...]) as Settings, with the
|
|
85
|
+
# spec's defaults for missing or unreadable values.
|
|
86
|
+
def parse_settings(rows)
|
|
87
|
+
h = {}
|
|
88
|
+
rows.each { |name, value| h[name.to_s] = value.to_s.strip }
|
|
89
|
+
services = begin
|
|
90
|
+
list = h["profile.services"].to_s.empty? ? nil : JSON.parse(h["profile.services"])
|
|
91
|
+
list.is_a?(Array) ? list.map(&:to_s) : nil
|
|
92
|
+
rescue JSON::ParserError
|
|
93
|
+
nil
|
|
94
|
+
end
|
|
95
|
+
Settings.new(
|
|
96
|
+
enabled: %w[true 1].include?(h["profile.enabled"].to_s.downcase),
|
|
97
|
+
services: services,
|
|
98
|
+
interval_s: positive(h["profile.interval_s"], DEFAULT_INTERVAL_S),
|
|
99
|
+
duration_ms: [positive(h["profile.duration_ms"], DEFAULT_DURATION_MS), MAX_DURATION_MS].min,
|
|
100
|
+
now: h["profile.now"].to_s.empty? ? nil : h["profile.now"]
|
|
101
|
+
)
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
# A missing or empty list means every service; a service with no name
|
|
105
|
+
# matches only that.
|
|
106
|
+
def participates?(services, service)
|
|
107
|
+
return true if services.nil? || services.empty?
|
|
108
|
+
return false if service.to_s.empty?
|
|
109
|
+
services.include?(service.to_s)
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
# profile.now asks for one profile when it is newer than this process's
|
|
113
|
+
# start and less than five minutes old.
|
|
114
|
+
def now_due?(value, started_at, now = Time.now)
|
|
115
|
+
t = parse_time(value)
|
|
116
|
+
!t.nil? && t > started_at && now - t < NOW_FRESH_S
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
# created_at text ("2026-10-22T10:00:00.000Z", or with a space and no Z)
|
|
120
|
+
# as a UTC Time.
|
|
121
|
+
def parse_time(s)
|
|
122
|
+
m = s.to_s.strip.match(/\A(\d{4})-(\d\d)-(\d\d)[T ](\d\d):(\d\d):(\d\d)(\.\d+)?(Z|[+-]00:?00)?\z/)
|
|
123
|
+
return nil unless m
|
|
124
|
+
Time.utc(m[1].to_i, m[2].to_i, m[3].to_i, m[4].to_i, m[5].to_i, m[6].to_i + m[7].to_f)
|
|
125
|
+
rescue ArgumentError
|
|
126
|
+
nil
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# Stack counts ({"a;b;c" => n}) as folded text, heaviest first within
|
|
130
|
+
# MAX_FOLDED bytes (sdks/SPEC.md, "CPU profiles"): whole lines up to
|
|
131
|
+
# KEEP_WHOLE, then the rest cut after the deepest frame they share with
|
|
132
|
+
# a kept stack and ending in "(other)", then one "(other) n" for
|
|
133
|
+
# whatever is left.
|
|
134
|
+
def fold(counts)
|
|
135
|
+
samples = counts.values.sum
|
|
136
|
+
sorted = counts.sort_by { |stack, n| [-n, stack] }
|
|
137
|
+
out = +""
|
|
138
|
+
# Whole lines first, leaving room for the rolled-up ones.
|
|
139
|
+
i = 0
|
|
140
|
+
while i < sorted.size
|
|
141
|
+
line = "#{sorted[i][0]} #{sorted[i][1]}\n"
|
|
142
|
+
break if out.bytesize + line.bytesize > KEEP_WHOLE
|
|
143
|
+
out << line
|
|
144
|
+
i += 1
|
|
145
|
+
end
|
|
146
|
+
return [out, samples] if i == sorted.size
|
|
147
|
+
|
|
148
|
+
# The rest are cut after the deepest frame they share with a kept
|
|
149
|
+
# stack and end in "(other)", so their time stays under the right caller.
|
|
150
|
+
prefixes = {}
|
|
151
|
+
sorted[0...i].each do |stack, _|
|
|
152
|
+
j = -1
|
|
153
|
+
prefixes[stack[0...j]] = true while (j = stack.index(";", j + 1))
|
|
154
|
+
prefixes[stack] = true
|
|
155
|
+
end
|
|
156
|
+
rolled = Hash.new(0)
|
|
157
|
+
other = 0
|
|
158
|
+
sorted[i..].each do |stack, n|
|
|
159
|
+
cut = ""
|
|
160
|
+
j = -1
|
|
161
|
+
while (j = stack.index(";", j + 1))
|
|
162
|
+
cut = stack[0...j] if prefixes[stack[0...j]]
|
|
163
|
+
end
|
|
164
|
+
cut = stack if prefixes[stack]
|
|
165
|
+
if cut.empty?
|
|
166
|
+
other += n
|
|
167
|
+
elsif cut == stack # the whole stack leads into a kept one: nothing to cut
|
|
168
|
+
rolled[stack] += n
|
|
169
|
+
else
|
|
170
|
+
rolled["#{cut};(other)"] += n
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
rolled.sort_by { |stack, n| [-n, stack] }.each do |stack, n|
|
|
174
|
+
line = "#{stack} #{n}\n"
|
|
175
|
+
if out.bytesize + line.bytesize > MAX_FOLDED - 32 # room for the last "(other) n"
|
|
176
|
+
other += n
|
|
177
|
+
else
|
|
178
|
+
out << line
|
|
179
|
+
end
|
|
180
|
+
end
|
|
181
|
+
out << "(other) #{other}\n" if other.positive?
|
|
182
|
+
[out, samples]
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
# Whether the sampler takes a stack of this thread: it is using the
|
|
186
|
+
# CPU (or waiting for the GVL) and is not one of Visionary's own.
|
|
187
|
+
def sample?(thread, current = Thread.current)
|
|
188
|
+
thread.status == "run" && !own?(thread, current)
|
|
189
|
+
rescue StandardError
|
|
190
|
+
false
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
# The sampler itself and Visionary's background threads.
|
|
194
|
+
def own?(thread, current = Thread.current)
|
|
195
|
+
thread.equal?(current) || thread.name.to_s.start_with?("visionary-") ||
|
|
196
|
+
thread[Telemetry::INTERNAL] == true
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
private
|
|
200
|
+
|
|
201
|
+
def positive(v, default)
|
|
202
|
+
n = Float(v.to_s)
|
|
203
|
+
n.positive? && n.finite? ? n : default
|
|
204
|
+
rescue ArgumentError
|
|
205
|
+
default
|
|
206
|
+
end
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
# Turns backtrace locations into "label file:line" frames, where file
|
|
210
|
+
# says whose code it is: relative to the working directory for the app,
|
|
211
|
+
# lib:<gem>/<path> for gems, std:<path> for Ruby's own library.
|
|
212
|
+
class Frames
|
|
213
|
+
def initialize(root: Dir.pwd, gem_dirs: Frames.gem_dirs, std_dirs: Frames.std_dirs,
|
|
214
|
+
fold_case: Gem.win_platform?)
|
|
215
|
+
@fold_case = fold_case
|
|
216
|
+
@root = dir(root)
|
|
217
|
+
@gem_dirs = gem_dirs.map { |d| dir(d) }.uniq.sort_by { |d| -d.length }
|
|
218
|
+
@std_dirs = std_dirs.map { |d| dir(d) }.uniq.sort_by { |d| -d.length }
|
|
219
|
+
@files = {}
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def self.gem_dirs
|
|
223
|
+
dirs = Gem.path.flat_map { |p| ["#{p}/gems", "#{p}/bundler/gems"] }
|
|
224
|
+
if defined?(::Bundler) && ::Bundler.respond_to?(:bundle_path)
|
|
225
|
+
begin
|
|
226
|
+
bp = ::Bundler.bundle_path.to_s
|
|
227
|
+
dirs.push("#{bp}/gems", "#{bp}/bundler/gems")
|
|
228
|
+
rescue StandardError
|
|
229
|
+
nil
|
|
230
|
+
end
|
|
231
|
+
end
|
|
232
|
+
dirs
|
|
233
|
+
end
|
|
234
|
+
|
|
235
|
+
def self.std_dirs
|
|
236
|
+
RbConfig::CONFIG.values_at("rubyarchdir", "rubylibdir").compact.reject(&:empty?)
|
|
237
|
+
end
|
|
238
|
+
|
|
239
|
+
# Outermost caller first. Only the sampled frame keeps its line: callers
|
|
240
|
+
# are one frame per function, whichever line they called from.
|
|
241
|
+
def stack(locations)
|
|
242
|
+
leaf = locations.first
|
|
243
|
+
locations.reverse_each.map { |loc| frame(loc, loc.equal?(leaf)) }.join(";")
|
|
244
|
+
end
|
|
245
|
+
|
|
246
|
+
def frame(loc, line = true)
|
|
247
|
+
label = loc.label || loc.base_label || "(unknown)"
|
|
248
|
+
file = file(loc.absolute_path || loc.path)
|
|
249
|
+
name = if file
|
|
250
|
+
line ? "#{label} #{file}:#{loc.lineno}" : "#{label} #{file}"
|
|
251
|
+
else
|
|
252
|
+
label.to_s
|
|
253
|
+
end
|
|
254
|
+
name.include?(";") ? name.tr(";", ",") : name
|
|
255
|
+
end
|
|
256
|
+
|
|
257
|
+
def file(path)
|
|
258
|
+
return nil if path.nil? || path.empty?
|
|
259
|
+
@files.fetch(path) { @files[path] = classify(path) }
|
|
260
|
+
end
|
|
261
|
+
|
|
262
|
+
private
|
|
263
|
+
|
|
264
|
+
def classify(path)
|
|
265
|
+
# Ruby's own library compiled in (Kernel#loop, Integer#times, ...):
|
|
266
|
+
# <internal:kernel> is kernel.rb in Ruby's source.
|
|
267
|
+
return "std:#{path[10..-2]}.rb" if path.start_with?("<internal:") && path.end_with?(">")
|
|
268
|
+
return nil if path.start_with?("(", "<")
|
|
269
|
+
|
|
270
|
+
p = path.tr("\\", "/")
|
|
271
|
+
key = @fold_case ? p.downcase : p
|
|
272
|
+
@gem_dirs.each do |d|
|
|
273
|
+
next unless key.start_with?(d)
|
|
274
|
+
return gem_file(p[d.length..])
|
|
275
|
+
end
|
|
276
|
+
if (m = p.match(%r{/vendor/bundle/[^/]+/[^/]+/(?:bundler/)?gems/(.+)\z}))
|
|
277
|
+
return gem_file(m[1])
|
|
278
|
+
end
|
|
279
|
+
@std_dirs.each do |d|
|
|
280
|
+
return "std:#{p[d.length..]}" if key.start_with?(d)
|
|
281
|
+
end
|
|
282
|
+
return p[@root.length..] if key.start_with?(@root)
|
|
283
|
+
p
|
|
284
|
+
end
|
|
285
|
+
|
|
286
|
+
# "rack-3.2.7/lib/rack/builder.rb" -> "lib:rack/rack/builder.rb".
|
|
287
|
+
def gem_file(rest)
|
|
288
|
+
dir, inner = rest.split("/", 2)
|
|
289
|
+
name = dir.sub(/-\d[^-]*(?:-.+)?\z/, "").sub(/-\h{7,40}\z/, "")
|
|
290
|
+
name = dir if name.empty?
|
|
291
|
+
"lib:#{name}/#{inner.to_s.delete_prefix('lib/')}"
|
|
292
|
+
end
|
|
293
|
+
|
|
294
|
+
# With a trailing slash, case-folded where the file system is.
|
|
295
|
+
def dir(d)
|
|
296
|
+
s = d.to_s.tr("\\", "/").chomp("/") + "/"
|
|
297
|
+
@fold_case ? s.downcase : s
|
|
298
|
+
end
|
|
299
|
+
end
|
|
300
|
+
|
|
301
|
+
# CRuby lets a thread that computes keep the GVL for 100 ms (shifted by
|
|
302
|
+
# its priority) while another waits for it. A request that computes for
|
|
303
|
+
# 40 ms and then writes its response gives the GVL up at the write, when
|
|
304
|
+
# it no longer runs, so a sampler waiting for the GVL would rarely catch
|
|
305
|
+
# it computing. While a profile runs, the app's threads get the lowest
|
|
306
|
+
# priority (-3: a 12.5 ms quantum) and get their own back afterwards. In
|
|
307
|
+
# CRuby the priority only sets that quantum; all threads drop together,
|
|
308
|
+
# so none gains on another.
|
|
309
|
+
class Quantum
|
|
310
|
+
LOWEST = -3
|
|
311
|
+
|
|
312
|
+
def initialize(enabled: RUBY_ENGINE == "ruby")
|
|
313
|
+
@enabled = enabled
|
|
314
|
+
@saved = {}
|
|
315
|
+
@first = true
|
|
316
|
+
end
|
|
317
|
+
|
|
318
|
+
def shorten(thread)
|
|
319
|
+
return if !@enabled || @saved.key?(thread)
|
|
320
|
+
prio = thread.priority
|
|
321
|
+
# A thread started during the profile inherited the lowered priority.
|
|
322
|
+
@saved[thread] = !@first && prio == LOWEST ? 0 : prio
|
|
323
|
+
thread.priority = LOWEST if prio > LOWEST
|
|
324
|
+
rescue StandardError
|
|
325
|
+
nil
|
|
326
|
+
end
|
|
327
|
+
|
|
328
|
+
def first_pass_done
|
|
329
|
+
@first = false
|
|
330
|
+
end
|
|
331
|
+
|
|
332
|
+
def restore
|
|
333
|
+
return unless @enabled
|
|
334
|
+
@saved.each do |t, prio|
|
|
335
|
+
t.priority = prio if t.alive? && t.priority != prio
|
|
336
|
+
rescue StandardError
|
|
337
|
+
nil
|
|
338
|
+
end
|
|
339
|
+
Thread.list.each do |t|
|
|
340
|
+
next if @saved.key?(t) || Profiler.own?(t)
|
|
341
|
+
t.priority = 0 if t.priority == LOWEST
|
|
342
|
+
rescue StandardError
|
|
343
|
+
nil
|
|
344
|
+
end
|
|
345
|
+
@saved.clear
|
|
346
|
+
end
|
|
347
|
+
end
|
|
348
|
+
|
|
349
|
+
attr_reader :settings
|
|
350
|
+
|
|
351
|
+
def initialize(cfg, instance, store, writer, logger, started_at: Time.now)
|
|
352
|
+
@cfg = cfg
|
|
353
|
+
@instance = instance
|
|
354
|
+
@store = store
|
|
355
|
+
@writer = writer
|
|
356
|
+
@logger = logger
|
|
357
|
+
@started_at = started_at
|
|
358
|
+
@frames = Frames.new
|
|
359
|
+
@settings = Settings.new(enabled: false, services: nil, interval_s: DEFAULT_INTERVAL_S,
|
|
360
|
+
duration_ms: DEFAULT_DURATION_MS, now: nil)
|
|
361
|
+
reset
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
def start
|
|
365
|
+
@thread = Thread.new { run }
|
|
366
|
+
@thread.name = "visionary-profiler" if @thread.respond_to?(:name=)
|
|
367
|
+
self
|
|
368
|
+
end
|
|
369
|
+
|
|
370
|
+
def after_fork
|
|
371
|
+
Tally.after_fork
|
|
372
|
+
reset
|
|
373
|
+
start
|
|
374
|
+
end
|
|
375
|
+
|
|
376
|
+
def close(timeout = 1)
|
|
377
|
+
@mutex.synchronize do
|
|
378
|
+
@stopped = true
|
|
379
|
+
@cond.broadcast
|
|
380
|
+
end
|
|
381
|
+
@thread&.join(timeout)
|
|
382
|
+
end
|
|
383
|
+
|
|
384
|
+
# Whether profiles are taken on the schedule: enabled in the dashboard
|
|
385
|
+
# for this service.
|
|
386
|
+
def active?
|
|
387
|
+
@settings.enabled && Profiler.participates?(@settings.services, @cfg.service)
|
|
388
|
+
end
|
|
389
|
+
|
|
390
|
+
# Reads the settings; true when profile.now asks for a profile now.
|
|
391
|
+
def poll
|
|
392
|
+
rows = @store.rows(SETTINGS_SQL)
|
|
393
|
+
@read_failed = false
|
|
394
|
+
was = active?
|
|
395
|
+
@settings = Profiler.parse_settings(rows)
|
|
396
|
+
now_value = @settings.now
|
|
397
|
+
due = !now_value.nil? && now_value != @last_now &&
|
|
398
|
+
Profiler.now_due?(now_value, @started_at) &&
|
|
399
|
+
Profiler.participates?(@settings.services, @cfg.service)
|
|
400
|
+
@last_now = now_value
|
|
401
|
+
log_switch(was)
|
|
402
|
+
due
|
|
403
|
+
rescue StandardError => e
|
|
404
|
+
unless @read_failed
|
|
405
|
+
@read_failed = true
|
|
406
|
+
@logger.info("visionary: profiling settings unreadable (#{e.message.lines.first.to_s.strip}); profiling stays off")
|
|
407
|
+
end
|
|
408
|
+
was = active?
|
|
409
|
+
@settings = Profiler.parse_settings([])
|
|
410
|
+
log_switch(was)
|
|
411
|
+
false
|
|
412
|
+
end
|
|
413
|
+
|
|
414
|
+
# Samples for duration_ms and queues the profile. Returns the row, or
|
|
415
|
+
# nil when it was dropped (too few samples, no finished request).
|
|
416
|
+
def capture
|
|
417
|
+
@last_started = Tally.started
|
|
418
|
+
duration_s = @settings.duration_ms / 1000.0
|
|
419
|
+
counts = Hash.new(0)
|
|
420
|
+
started_at = Time.now
|
|
421
|
+
t0 = mono
|
|
422
|
+
deadline = t0 + duration_s
|
|
423
|
+
quantum = Quantum.new
|
|
424
|
+
Tally.begin!
|
|
425
|
+
last = mono
|
|
426
|
+
begin
|
|
427
|
+
loop do
|
|
428
|
+
tick = mono
|
|
429
|
+
# A thread busy in Ruby code keeps the GVL for a quantum before the
|
|
430
|
+
# sampler gets it back, so a late tick stands for the ticks it
|
|
431
|
+
# missed: the stack it finds is the one that held the GVL.
|
|
432
|
+
weight = [((tick - last) / TICK_S).round, 1].max
|
|
433
|
+
last = tick
|
|
434
|
+
Thread.list.each do |t|
|
|
435
|
+
next if Profiler.own?(t)
|
|
436
|
+
quantum.shorten(t)
|
|
437
|
+
next unless t.status == "run"
|
|
438
|
+
locs = t.backtrace_locations
|
|
439
|
+
counts[@frames.stack(locs)] += weight if locs && !locs.empty?
|
|
440
|
+
end
|
|
441
|
+
quantum.first_pass_done
|
|
442
|
+
break if mono >= deadline || wait(TICK_S - (mono - tick))
|
|
443
|
+
end
|
|
444
|
+
ensure
|
|
445
|
+
quantum&.restore
|
|
446
|
+
busy = Tally.finish!
|
|
447
|
+
end
|
|
448
|
+
elapsed_ms = ((mono - t0) * 1000).round(3)
|
|
449
|
+
folded, samples = Profiler.fold(counts)
|
|
450
|
+
routes = busy.transform_values { |ms| [ms.round, 1].max }
|
|
451
|
+
return nil if samples < MIN_SAMPLES || routes.empty?
|
|
452
|
+
|
|
453
|
+
row = [Visionary.timestamp(started_at), @cfg.service, @instance, @cfg.version, "ruby",
|
|
454
|
+
elapsed_ms, samples, JSON.generate(routes), folded]
|
|
455
|
+
@writer.enqueue([:profile, row])
|
|
456
|
+
row
|
|
457
|
+
rescue StandardError => e
|
|
458
|
+
@logger.warn("visionary: CPU profile failed: #{e.class}: #{e.message}")
|
|
459
|
+
nil
|
|
460
|
+
end
|
|
461
|
+
|
|
462
|
+
private
|
|
463
|
+
|
|
464
|
+
def reset
|
|
465
|
+
@mutex = Mutex.new
|
|
466
|
+
@cond = ConditionVariable.new
|
|
467
|
+
@stopped = false
|
|
468
|
+
@thread = nil
|
|
469
|
+
@last_started = Tally.started
|
|
470
|
+
end
|
|
471
|
+
|
|
472
|
+
def run
|
|
473
|
+
Telemetry.internal!
|
|
474
|
+
next_poll = mono
|
|
475
|
+
next_profile = nil
|
|
476
|
+
until stopped?
|
|
477
|
+
if mono >= next_poll
|
|
478
|
+
next_poll = mono + POLL_S
|
|
479
|
+
if poll
|
|
480
|
+
@logger.info("visionary: taking a CPU profile now (asked for in the dashboard)")
|
|
481
|
+
capture
|
|
482
|
+
next_profile = mono + @settings.interval_s if next_profile
|
|
483
|
+
end
|
|
484
|
+
end
|
|
485
|
+
if active?
|
|
486
|
+
next_profile ||= mono + @settings.interval_s
|
|
487
|
+
if mono >= next_profile
|
|
488
|
+
capture unless Tally.started == @last_started
|
|
489
|
+
next_profile = mono + @settings.interval_s
|
|
490
|
+
end
|
|
491
|
+
else
|
|
492
|
+
next_profile = nil
|
|
493
|
+
end
|
|
494
|
+
break if wait([next_poll, next_profile].compact.min - mono)
|
|
495
|
+
end
|
|
496
|
+
rescue StandardError => e
|
|
497
|
+
@logger.error("visionary: profiler stopped: #{e.class}: #{e.message}")
|
|
498
|
+
end
|
|
499
|
+
|
|
500
|
+
def log_switch(was)
|
|
501
|
+
now = active?
|
|
502
|
+
return if now == was
|
|
503
|
+
if now
|
|
504
|
+
@logger.info("visionary: CPU profiling on (#{@settings.duration_ms.round} ms every #{@settings.interval_s}s)")
|
|
505
|
+
else
|
|
506
|
+
@logger.info("visionary: CPU profiling off")
|
|
507
|
+
end
|
|
508
|
+
end
|
|
509
|
+
|
|
510
|
+
# Waits up to secs; true when stopped.
|
|
511
|
+
def wait(secs)
|
|
512
|
+
@mutex.synchronize do
|
|
513
|
+
@cond.wait(@mutex, secs) if !@stopped && secs.positive?
|
|
514
|
+
@stopped
|
|
515
|
+
end
|
|
516
|
+
end
|
|
517
|
+
|
|
518
|
+
def stopped? = @mutex.synchronize { @stopped }
|
|
519
|
+
|
|
520
|
+
def mono = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
521
|
+
end
|
|
522
|
+
end
|
data/lib/visionary/retention.rb
CHANGED
|
@@ -8,7 +8,8 @@ module Visionary
|
|
|
8
8
|
EVERY = 24 * 3600
|
|
9
9
|
MAX_ROUNDS = 200
|
|
10
10
|
|
|
11
|
-
# tables: { "api_logs" => days, ... }; 0 days
|
|
11
|
+
# tables: { "api_logs" => days, ... }; 0 days (the default) purges nothing
|
|
12
|
+
# and starts no thread: the Visionary server applies Settings → Retention.
|
|
12
13
|
def initialize(store, logger, tables)
|
|
13
14
|
@store = store
|
|
14
15
|
@logger = logger
|
|
@@ -46,6 +47,8 @@ module Visionary
|
|
|
46
47
|
n = begin
|
|
47
48
|
@store.changes(sql, [cutoff])
|
|
48
49
|
rescue StandardError => e
|
|
50
|
+
# The server creates profiles; before it has, there is nothing to purge.
|
|
51
|
+
return total if table == "profiles" && e.message.match?(/no such table|does(?:n't| not) exist/i)
|
|
49
52
|
@logger.error("visionary: retention purge of #{table} failed after #{total} rows: #{e.message}")
|
|
50
53
|
return total
|
|
51
54
|
end
|
data/lib/visionary/schema.rb
CHANGED
|
@@ -64,12 +64,16 @@ module Visionary
|
|
|
64
64
|
|
|
65
65
|
RELEASE_COLUMNS = %w[created_at service instance version sdk sdk_version runtime].freeze
|
|
66
66
|
|
|
67
|
+
# Created by the Visionary server, never by the SDK.
|
|
68
|
+
PROFILE_COLUMNS = %w[created_at service instance version sdk duration_ms samples routes folded].freeze
|
|
69
|
+
|
|
67
70
|
TABLES = {
|
|
68
71
|
"api_logs" => REQUEST_COLUMNS + ADDED_REQUEST_COLUMNS.map(&:first),
|
|
69
72
|
"system_metrics" => SYSTEM_BASE_COLUMNS + ADDED_SYSTEM_COLUMNS.map(&:first),
|
|
70
73
|
"spans" => SPAN_COLUMNS,
|
|
71
74
|
"errors" => ERROR_COLUMNS,
|
|
72
|
-
"releases" => RELEASE_COLUMNS
|
|
75
|
+
"releases" => RELEASE_COLUMNS,
|
|
76
|
+
"profiles" => PROFILE_COLUMNS
|
|
73
77
|
}.freeze
|
|
74
78
|
|
|
75
79
|
# Which script creates a table, and the columns older copies may lack.
|
|
@@ -40,6 +40,11 @@ module Visionary
|
|
|
40
40
|
with { |c| c.select_values(sql).map(&:to_s) }
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
+
# Every row of a query, as arrays of values.
|
|
44
|
+
def rows(sql)
|
|
45
|
+
with { |c| c.select_rows(sql) }
|
|
46
|
+
end
|
|
47
|
+
|
|
43
48
|
def batch(stmts)
|
|
44
49
|
with do |c|
|
|
45
50
|
c.transaction(requires_new: true) do
|
data/lib/visionary/store/d1.rb
CHANGED
|
@@ -42,6 +42,13 @@ module Visionary
|
|
|
42
42
|
Array(rows).map { |r| r.is_a?(Array) ? r[0].to_s : nil }.compact
|
|
43
43
|
end
|
|
44
44
|
|
|
45
|
+
# Every row of a query, as arrays of values.
|
|
46
|
+
def rows(sql)
|
|
47
|
+
result = post(@raw_uri, { sql: sql })
|
|
48
|
+
rows = result.is_a?(Array) && result.dig(0, "results", "rows")
|
|
49
|
+
Array(rows).select { |r| r.is_a?(Array) }
|
|
50
|
+
end
|
|
51
|
+
|
|
45
52
|
def batch(stmts)
|
|
46
53
|
stmts.each_slice(MAX_BATCH) do |chunk|
|
|
47
54
|
post(@query_uri, { batch: chunk.map { |sql, args| { sql: sql, params: args.map { |a| self.class.param(a) } } } })
|
data/lib/visionary/store/sql.rb
CHANGED
|
@@ -39,6 +39,11 @@ module Visionary
|
|
|
39
39
|
run { |c| c.column(sql) }
|
|
40
40
|
end
|
|
41
41
|
|
|
42
|
+
# Every row of a query, as arrays of values.
|
|
43
|
+
def rows(sql)
|
|
44
|
+
run { |c| c.rows(sql) }
|
|
45
|
+
end
|
|
46
|
+
|
|
42
47
|
# One transaction for the whole batch.
|
|
43
48
|
def batch(stmts)
|
|
44
49
|
run do |c|
|
|
@@ -157,7 +162,7 @@ module Visionary
|
|
|
157
162
|
end
|
|
158
163
|
|
|
159
164
|
# Thin driver adapters: execute returns affected rows, column returns the
|
|
160
|
-
# first column of every row.
|
|
165
|
+
# first column of every row, rows every row as an array.
|
|
161
166
|
class PgConn
|
|
162
167
|
def initialize(conn)
|
|
163
168
|
@conn = conn
|
|
@@ -177,6 +182,13 @@ module Visionary
|
|
|
177
182
|
res&.clear
|
|
178
183
|
end
|
|
179
184
|
|
|
185
|
+
def rows(sql)
|
|
186
|
+
res = @conn.exec(sql)
|
|
187
|
+
res.values
|
|
188
|
+
ensure
|
|
189
|
+
res&.clear
|
|
190
|
+
end
|
|
191
|
+
|
|
180
192
|
def broken?
|
|
181
193
|
@conn.finished? || @conn.status != PG::CONNECTION_OK
|
|
182
194
|
rescue StandardError
|
|
@@ -231,6 +243,10 @@ module Visionary
|
|
|
231
243
|
@client.query(sql, as: :array).map { |r| r[0].to_s }
|
|
232
244
|
end
|
|
233
245
|
|
|
246
|
+
def rows(sql)
|
|
247
|
+
@client.query(sql, as: :array).to_a
|
|
248
|
+
end
|
|
249
|
+
|
|
234
250
|
def broken?
|
|
235
251
|
!@client.ping
|
|
236
252
|
rescue StandardError
|
|
@@ -262,6 +278,8 @@ module Visionary
|
|
|
262
278
|
@db.execute(sql).map { |r| r[0].to_s }
|
|
263
279
|
end
|
|
264
280
|
|
|
281
|
+
def rows(sql) = @db.execute(sql)
|
|
282
|
+
|
|
265
283
|
def broken? = @db.closed?
|
|
266
284
|
|
|
267
285
|
def close
|
data/lib/visionary/store.rb
CHANGED
|
@@ -7,7 +7,7 @@ require_relative "store/active_record"
|
|
|
7
7
|
module Visionary
|
|
8
8
|
# Where rows go: D1 over REST, a connection opened from a database URL, or
|
|
9
9
|
# the app's ActiveRecord pool. Every store answers the same calls:
|
|
10
|
-
# exec_script, exec, column_names, batch([[sql, args], ...]), changes,
|
|
10
|
+
# exec_script, exec, column_names, rows, batch([[sql, args], ...]), changes,
|
|
11
11
|
# after_fork and close.
|
|
12
12
|
module Store
|
|
13
13
|
# Picks the store from the config, or nil when none is configured.
|
data/lib/visionary/telemetry.rb
CHANGED
|
@@ -37,6 +37,7 @@ module Visionary
|
|
|
37
37
|
@spans_on = cfg.spans ? true : false
|
|
38
38
|
@errors_on = cfg.errors ? true : false
|
|
39
39
|
@threshold = cfg.span_threshold_ms.to_f
|
|
40
|
+
@span_max = cfg.span_max.to_i
|
|
40
41
|
@limiter = Errors::Limiter.new
|
|
41
42
|
end
|
|
42
43
|
|
|
@@ -56,7 +57,10 @@ module Visionary
|
|
|
56
57
|
row_count: row_count, request_id: ctx&.request_id
|
|
57
58
|
)
|
|
58
59
|
if ctx
|
|
59
|
-
|
|
60
|
+
# A late span (its request was already logged) is written on its
|
|
61
|
+
# own: always up to span_max of them per request, then only slow ones.
|
|
62
|
+
late = ctx.add_span(span)
|
|
63
|
+
@writer.enqueue([:span, span_row(span)]) if late && (late <= @span_max || duration_ms >= @threshold)
|
|
60
64
|
elsif duration_ms >= @threshold
|
|
61
65
|
@writer.enqueue([:span, span_row(span)])
|
|
62
66
|
end
|
|
@@ -69,9 +73,9 @@ module Visionary
|
|
|
69
73
|
span.target, span.detail, span.offset_ms, span.duration_ms, span.status, span.error, span.row_count]
|
|
70
74
|
end
|
|
71
75
|
|
|
72
|
-
# Rows for a finished request's spans
|
|
73
|
-
def request_span_rows(ctx
|
|
74
|
-
return [] unless @spans_on
|
|
76
|
+
# Rows for a finished request's spans, kept for every request.
|
|
77
|
+
def request_span_rows(ctx)
|
|
78
|
+
return [] unless @spans_on
|
|
75
79
|
ctx.spans.map { |s| span_row(s) }
|
|
76
80
|
end
|
|
77
81
|
|
data/lib/visionary/version.rb
CHANGED
data/lib/visionary/writer.rb
CHANGED
|
@@ -12,10 +12,10 @@ module Visionary
|
|
|
12
12
|
)
|
|
13
13
|
|
|
14
14
|
# The single background thread that batches api_logs, spans, errors and
|
|
15
|
-
# releases rows. Queue items are [:request, Entry],
|
|
16
|
-
# [:error, row] or [:
|
|
15
|
+
# releases rows, and CPU profiles. Queue items are [:request, Entry],
|
|
16
|
+
# [:span, row], [:error, row], [:release, row] or [:profile, row].
|
|
17
17
|
class Writer
|
|
18
|
-
INSERTS = { request: "api_logs", span: "spans", error: "errors", release: "releases" }.freeze
|
|
18
|
+
INSERTS = { request: "api_logs", span: "spans", error: "errors", release: "releases", profile: "profiles" }.freeze
|
|
19
19
|
|
|
20
20
|
def initialize(cfg, store, schemas, logger)
|
|
21
21
|
@cfg = cfg
|
|
@@ -93,7 +93,15 @@ module Visionary
|
|
|
93
93
|
@logger.warn("visionary: queue full; #{n} entries dropped") if n.positive?
|
|
94
94
|
end
|
|
95
95
|
|
|
96
|
+
# A profile goes in its own transaction: the server creates its table, so
|
|
97
|
+
# if that is missing only the profile is lost.
|
|
96
98
|
def write(batch)
|
|
99
|
+
profiles, rest = batch.partition { |kind, _| kind == :profile }
|
|
100
|
+
commit(rest) unless rest.empty?
|
|
101
|
+
profiles.each { |item| commit([item]) }
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
def commit(batch)
|
|
97
105
|
stmts = begin
|
|
98
106
|
users = lookup_users(batch)
|
|
99
107
|
batch.flat_map { |kind, item| statements(kind, item, users) }
|
data/lib/visionary.rb
CHANGED
|
@@ -15,6 +15,7 @@ require_relative "visionary/system/cgroup"
|
|
|
15
15
|
require_relative "visionary/system/recorder"
|
|
16
16
|
require_relative "visionary/telemetry"
|
|
17
17
|
require_relative "visionary/telemetry/hooks"
|
|
18
|
+
require_relative "visionary/profiler"
|
|
18
19
|
require_relative "visionary/body"
|
|
19
20
|
require_relative "visionary/middleware"
|
|
20
21
|
require_relative "visionary/agent"
|
|
@@ -96,7 +97,7 @@ module Visionary
|
|
|
96
97
|
end
|
|
97
98
|
|
|
98
99
|
# Measures the block as a span and returns its value. kind is "custom",
|
|
99
|
-
# "cache", "queue", "db" or "
|
|
100
|
+
# "cache", "queue", "db", "http", "graphql" or "background"; name should be low-cardinality (the
|
|
100
101
|
# operation, not its arguments).
|
|
101
102
|
#
|
|
102
103
|
# Visionary.span("charge card", kind: "http", target: "api.stripe.com") { stripe.charge(...) }
|
|
@@ -119,6 +120,30 @@ module Visionary
|
|
|
119
120
|
end
|
|
120
121
|
end
|
|
121
122
|
|
|
123
|
+
# Runs the block in a new Thread that keeps belonging to the current
|
|
124
|
+
# request, and records it as one span of kind "background" named name,
|
|
125
|
+
# from start to finish. Calls made inside it carry the request's
|
|
126
|
+
# request_id and offset_ms even after the response was sent; those ending
|
|
127
|
+
# after the request was logged are written on their own and never change
|
|
128
|
+
# its totals. Outside a request the span is a lone span (kept when it
|
|
129
|
+
# takes at least VISIONARY_SPAN_THRESHOLD_MS).
|
|
130
|
+
#
|
|
131
|
+
# If the block raises, the span gets status "error" and the message, and
|
|
132
|
+
# the exception is re-raised in the thread as Ruby normally does: printed
|
|
133
|
+
# when Thread.report_on_exception is on, raised again by join/value.
|
|
134
|
+
# Returns the Thread.
|
|
135
|
+
#
|
|
136
|
+
# Visionary.background("send welcome email") { Mailer.welcome(user).deliver_now }
|
|
137
|
+
def background(name, &block)
|
|
138
|
+
raise ArgumentError, "Visionary.background needs a block" unless block
|
|
139
|
+
|
|
140
|
+
ctx = RequestContext.current # thread locals don't carry over on Ruby 3.1
|
|
141
|
+
Thread.new do
|
|
142
|
+
RequestContext.current = ctx
|
|
143
|
+
span(name, kind: "background", &block)
|
|
144
|
+
end
|
|
145
|
+
end
|
|
146
|
+
|
|
122
147
|
# Records a handled error (handled = 1). Inside a request it links to it.
|
|
123
148
|
#
|
|
124
149
|
# rescue PaymentError => e
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: visionary-sdk
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- styropyr0
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-10-
|
|
11
|
+
date: 2026-10-22 00:00:00.000000000 Z
|
|
12
12
|
dependencies: []
|
|
13
13
|
description: |
|
|
14
14
|
The Ruby collector for Visionary. A Rack middleware (with a Railtie for Rails)
|
|
@@ -30,6 +30,7 @@ files:
|
|
|
30
30
|
- lib/visionary/context.rb
|
|
31
31
|
- lib/visionary/fork.rb
|
|
32
32
|
- lib/visionary/middleware.rb
|
|
33
|
+
- lib/visionary/profiler.rb
|
|
33
34
|
- lib/visionary/queue.rb
|
|
34
35
|
- lib/visionary/railtie.rb
|
|
35
36
|
- lib/visionary/redactor.rb
|