visionary-sdk 0.5.2 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 7219c9b0987df17cbcb27fc9f0f584a77175ce5767b51236142d2c0c13084442
4
- data.tar.gz: aae551a0b45c8b888edf8451eb1914a2a07ffe45ae6dde4b15cae3d7de4385ee
3
+ metadata.gz: a7d2e913847c653919ec1f345af3a86f40965707c686e25bd07045e76aff0161
4
+ data.tar.gz: 46ac3f49d03c1905f0a33c7befcb0f5b8e807b48f47b50629955a3b84eba28d8
5
5
  SHA512:
6
- metadata.gz: 9ad5629d6fe0f7c49b6f55eea196259a3a050be902fbad8729b5af5b1e2ba24fabe18a051e01fc8a02a1a26923fb8efc9d1e4c69f3703d1b98cdf28f5056978e
7
- data.tar.gz: 931de79643682b1f62c729f008ae217e8a16d5464bba0f804763a374e77c4eab6bdb3aaabead6e79d191d0e8cecb6159481bcce1dc80247c1cca77ae7cd5fde1
6
+ metadata.gz: 2061e3a2855102aac59fb1b6b6e4c73f12afdb0aacee73132b4b6196f3264f119bd741a8ca02f68e3412c6466e893b4f836b5fa6142c34f07084f9a2349f739b
7
+ data.tar.gz: aeade39d1f5076bd7185e57ac1f0976cf776948526e454d61f88e5c1c28de6a2a323439470d324ea99583b3421cbb2be4f518adc85fb672dfefb9142f126d461
data/README.md CHANGED
@@ -8,7 +8,9 @@ or Rails service and writes:
8
8
  database and outbound HTTP calls the request made),
9
9
  - `spans` for database statements and outbound HTTP calls of slow or failed
10
10
  requests, `errors` for exceptions, and a `releases` row per deploy,
11
- - a periodic host / process / runtime sample into `system_metrics`.
11
+ - a periodic host / process / runtime sample into `system_metrics`,
12
+ - CPU profiles into `profiles`, only while they are turned on in the
13
+ dashboard.
12
14
 
13
15
  The Visionary server reads these tables for its dashboards. All slow work
14
16
  happens on background threads, so requests are never blocked or slowed. If
@@ -87,6 +89,7 @@ Visionary.configure { |c| c.route = ->(env) { env["my.router.pattern"] } }
87
89
  Visionary.set_user(id, handle: nil, role: nil) # the signed-in user of the current request
88
90
  Visionary.set_route("/users/:id") # route pattern, for routers Visionary can't see
89
91
  Visionary.span("charge card", kind: "http", target: "api.stripe.com") { ... } # a manual span
92
+ Visionary.background("send welcome email") { ... } # a thread that stays with the request
90
93
  Visionary.capture_exception(e) # a handled error (handled = 1)
91
94
  Visionary.start # start now instead of on the first request
92
95
  Visionary.close(10) # flush; also runs at exit
@@ -100,6 +103,12 @@ user per batch, for users missing a handle or role.
100
103
 
101
104
  ## Spans and errors
102
105
 
106
+ The middleware records the request; the calls inside it are recorded only when
107
+ their client is hooked (see below). Wrap the rest (a Redis or queue client
108
+ without a hook, CPU work such as image processing) in `Visionary.span`.
109
+ Anything left shows as **Untracked** in the request's breakdown; see
110
+ [What the middleware sees](../../Readme.md#what-the-middleware-sees-and-what-needs-code).
111
+
103
112
  - **Database:** every `sql.active_record` notification except `SCHEMA` and
104
113
  cached queries becomes a `db` span named after the statement (`SELECT
105
114
  users`, `INSERT bookings`, `BEGIN`), with the statement sanitized (literals
@@ -111,10 +120,9 @@ user per batch, for users missing a handle or role.
111
120
 
112
121
  Every logged request gets `db_count`, `db_ms`, `http_count`, `http_ms`
113
122
  (`db_*` are NULL when ActiveRecord isn't loaded and no `db` span was
114
- recorded). Span rows are stored only for requests that took at least
115
- `VISIONARY_SPAN_THRESHOLD_MS` or answered 5xx, at most `VISIONARY_SPAN_MAX`
116
- per request; spans outside requests are stored when they take at least the
117
- threshold themselves. Visionary's own writes never become spans.
123
+ recorded). Span rows are stored for every request, at most
124
+ `VISIONARY_SPAN_MAX` per request; spans outside requests are stored when they
125
+ take at least `VISIONARY_SPAN_THRESHOLD_MS` themselves. Visionary's own writes never become spans.
118
126
 
119
127
  Exceptions that escape the app are recorded and re-raised unchanged; the
120
128
  middleware logs the request as 500. Errors are grouped by a fingerprint over
@@ -122,12 +130,95 @@ the exception class and the top five frames of your own code (not gems, the
122
130
  standard library or this SDK), as `method@path` relative to the working
123
131
  directory, so it is stable across deploys and line changes.
124
132
 
133
+ ## Background work
134
+
135
+ Work a handler starts and doesn't wait for still belongs to its request. Start
136
+ it with `Visionary.background`, which runs the block in a new thread and
137
+ returns the `Thread`:
138
+
139
+ ```ruby
140
+ Visionary.background("send welcome email") { UserMailer.welcome(user).deliver_now }
141
+ ```
142
+
143
+ - The block is one span of kind `background` with the name you give, from
144
+ start to finish. If it raises, the span is marked `error` with the message
145
+ and the exception is re-raised in the thread as Ruby normally does (printed
146
+ when `Thread.report_on_exception` is on, raised again by `join`/`value`).
147
+ - Calls made inside it keep the request's `request_id`, and their `offset_ms`
148
+ counts from the request's start, even after the response was sent. Calls
149
+ that end after the request was logged don't change its `db_*`/`http_*`
150
+ totals. They are stored whatever their duration, up to
151
+ `VISIONARY_SPAN_MAX` of them per request; after that only those taking at
152
+ least `VISIONARY_SPAN_THRESHOLD_MS`. `background` spans never count in the
153
+ totals.
154
+ - Outside a request the `background` span has no `request_id` and is stored
155
+ only when it takes at least `VISIONARY_SPAN_THRESHOLD_MS`.
156
+
157
+ On Ruby 3.2+ any thread or fiber started during a request inherits it, so
158
+ calls from a plain `Thread.new` are attributed the same way; `background`
159
+ also carries the request over on 3.1 and adds the span around the work.
160
+
161
+ ## CPU profiles
162
+
163
+ Profiling is **off until you turn it on in the dashboard** (Settings →
164
+ Profiling), and you can turn it off there again at any time. Neither needs a
165
+ redeploy. The SDK reads the switch from the `visionary_settings` table when it
166
+ starts and every 30 s after that, on its own thread. It logs one line when
167
+ profiling turns on or off. It follows the dashboard's service list
168
+ (`VISIONARY_SERVICE`), interval (default every 300 s) and duration (default
169
+ 10 s), and "Profile now" takes one profile within 30 s. To keep a service out
170
+ of profiling whatever the dashboard says, set `VISIONARY_PROFILE=false`.
171
+
172
+ While a profile runs, a thread wakes about every 10 ms and takes the stack of
173
+ each app thread whose status is `run`. Threads that sleep, wait on a lock or
174
+ queue, or block on IO (a database or HTTP call) have the status `sleep`, so
175
+ they are left out, and so are Visionary's own threads. One row is written per
176
+ profile:
177
+
178
+ - `folded`: one `frame;frame;... count` line per distinct stack, outermost
179
+ call first. Each sample counts about 10 ms of CPU. Frames read
180
+ `Class#method file:line`. Your code's paths are relative to the working
181
+ directory. Gems are shown as `lib:<gem>/<path>`, e.g.
182
+ `lib:rack/rack/builder.rb`, and the standard library as `std:<path>`, e.g.
183
+ `std:json/common.rb` or `std:kernel.rb` for Ruby's built-in Ruby code.
184
+ Each row is capped at 90 000 bytes, and the lightest stacks are summed into
185
+ one `(other)` line.
186
+ - `routes`: the milliseconds that the requests finishing during the profile
187
+ took, per `METHOD route`.
188
+
189
+ A turn is skipped when no request arrived since the last one. A profile is
190
+ dropped when no request finished during it, or when it has fewer than 20
191
+ samples (about 200 ms of CPU).
192
+
193
+ **Cost:** nothing while profiling is off apart from the settings read every
194
+ 30 s. While a profile runs, the sampler takes some tens of stack walks per
195
+ second, which costs a few percent of one CPU.
196
+
197
+ **How accurate it is.** Ruby has no built-in CPU sampler, so these are
198
+ wall-clock samples of threads that are runnable:
199
+
200
+ - Only the thread holding the GVL really runs. Several busy threads are each
201
+ counted, so time spent waiting for the GVL counts as CPU.
202
+ - A thread computing in Ruby keeps the GVL for up to 100 ms while the sampler
203
+ waits for it. A request that computes for 40 ms and then writes its answer
204
+ would rarely be caught computing. So while a profile runs, the SDK lowers
205
+ the app's threads to Ruby's lowest thread priority (`-3`, a 12.5 ms
206
+ quantum) and gives each thread its own priority back afterwards. In CRuby,
207
+ priority sets only this quantum, so no thread gains on another. A late
208
+ sample counts for the 10 ms ticks it missed. Code that gives up the GVL
209
+ within a few ms (short bursts between IO calls) is still under-counted.
210
+ - A C extension that releases the GVL to do work (some database drivers,
211
+ compression) keeps the status `run`. Its time shows on the Ruby frame that
212
+ called it.
213
+ - C methods (`Array#each`, `Time#strftime`) show the file and line of the Ruby
214
+ code that called them.
215
+
125
216
  ## Forking servers (Puma, Unicorn)
126
217
 
127
218
  Threads don't survive `fork`. Visionary hooks `Process._fork` (Ruby 3.1+), so
128
219
  when Puma or Unicorn forks workers after the agent started (for example with
129
220
  `preload_app!` and an initializer that calls `Visionary.start`), each worker
130
- restarts its writer, sampler and retention threads with fresh queues and
221
+ restarts its writer, sampler, profiler and retention threads with fresh queues and
131
222
  connections. If you prefer to be explicit, Puma's hook does the same and is
132
223
  safe to combine with it:
133
224
 
@@ -187,7 +278,7 @@ plus the hooks above. Set it before the first request.
187
278
  | `VISIONARY_LOG_REQUESTS`, `VISIONARY_LOG_BODIES` | `true` |
188
279
  | `VISIONARY_LOG_PATH_PREFIX` | everything |
189
280
  | `VISIONARY_LOG_MAX_BODY_BYTES` | `8192` |
190
- | `VISIONARY_LOG_RETENTION_DAYS`, `VISIONARY_SYSTEM_RETENTION_DAYS` | `30` (0 keeps forever) |
281
+ | `VISIONARY_LOG_RETENTION_DAYS`, `VISIONARY_SYSTEM_RETENTION_DAYS` | `0` (off; above 0 the SDK also deletes older rows) |
191
282
  | `VISIONARY_LOG_REDACT_HEADERS`, `_REDACT_FIELDS`, `_REDACT_REQUEST_FIELDS` | added to the defaults |
192
283
  | `VISIONARY_LOG_ERROR_CODE_FIELD`, `_ERROR_KEY_FIELD` | `error.code`, `error.key` |
193
284
  | `VISIONARY_SYSTEM_METRICS` | `true` |
@@ -196,6 +287,11 @@ plus the hooks above. Set it before the first request.
196
287
  | `VISIONARY_RELEASE` | the platform's commit variable, shortened to 12 characters |
197
288
  | `VISIONARY_SPANS`, `VISIONARY_ERRORS` | `true` |
198
289
  | `VISIONARY_SPAN_THRESHOLD_MS`, `VISIONARY_SPAN_MAX` | `100`, `100` |
290
+ | `VISIONARY_PROFILE` | `true`: follow the dashboard's Profiling setting; `false` never profiles |
291
+
292
+ Old rows are deleted by the Visionary server, following Settings → Retention in
293
+ the dashboard. The two `*_RETENTION_DAYS` variables only matter when the
294
+ service writes to a database no Visionary server reads.
199
295
 
200
296
  Request bodies are captured as the app reads them, so a body the app never
201
297
  reads is not stored (its size still is, from `Content-Length`).
@@ -41,9 +41,10 @@ module Visionary
41
41
 
42
42
  @instance = cfg.instance.to_s.empty? ? Socket.gethostname : cfg.instance
43
43
  @schemas = Schemas.new(@store, @logger)
44
- if cfg.log_requests || cfg.spans || cfg.errors || cfg.version
44
+ if cfg.log_requests || cfg.spans || cfg.errors || cfg.version || cfg.profile
45
45
  @writer = Writer.new(cfg, @store, @schemas, @logger)
46
46
  end
47
+ @profiler = Profiler.new(cfg, @instance, @store, @writer, @logger) if cfg.profile && @writer
47
48
  @telemetry = Telemetry::Recorder.new(cfg, @instance, @writer) if cfg.spans || cfg.errors
48
49
  if cfg.system_metrics && cfg.system_interval.to_f.positive?
49
50
  @system = System::Recorder.new(cfg, @instance, @store, @schemas, @http, @logger)
@@ -55,6 +56,7 @@ module Visionary
55
56
  tables["errors"] = cfg.log_retention_days
56
57
  end
57
58
  tables["system_metrics"] = cfg.system_retention_days if @system
59
+ tables["profiles"] = cfg.log_retention_days if @profiler
58
60
  @retention = Retention.new(@store, @logger, tables)
59
61
  end
60
62
 
@@ -93,6 +95,7 @@ module Visionary
93
95
  @logger.info("visionary: recording system metrics every #{@config.system_interval}s as #{@instance}")
94
96
  end
95
97
  @retention.start
98
+ @profiler&.start
96
99
  install_hooks
97
100
  release
98
101
  self
@@ -107,6 +110,7 @@ module Visionary
107
110
  end
108
111
  deadline = mono + timeout
109
112
  @retention&.close
113
+ @profiler&.close
110
114
  idle = @writer ? @writer.close(timeout) : true
111
115
  idle = @system.close([deadline - mono, 0].max) && idle if @system
112
116
  @store.close if idle && @store
@@ -127,6 +131,7 @@ module Visionary
127
131
  @system&.after_fork
128
132
  @writer&.after_fork
129
133
  @retention&.after_fork if @retention&.needed?
134
+ @profiler&.after_fork
130
135
  @logger.info("visionary: restarted in forked process #{@pid}")
131
136
  end
132
137
 
@@ -21,7 +21,8 @@ module Visionary
21
21
  :redact_headers, :redact_fields, :redact_request_fields,
22
22
  :error_code_field, :error_key_field,
23
23
  :system_metrics, :system_interval, :system_retention_days, :disk_path,
24
- :service, :version, :spans, :span_threshold_ms, :span_max, :errors
24
+ :service, :version, :spans, :span_threshold_ms, :span_max, :errors,
25
+ :profile
25
26
 
26
27
  # Runs on the writer thread, once per distinct user per batch, for users
27
28
  # missing a handle or role: ->(id) { [handle, role] } or nil.
@@ -47,7 +48,7 @@ module Visionary
47
48
  @queue_size = 5000
48
49
  @batch_size = 50
49
50
  @flush_interval = 3.0
50
- @log_retention_days = 30
51
+ @log_retention_days = 0 # 0: off, the Visionary server applies Settings → Retention
51
52
  @redact_headers = []
52
53
  @redact_fields = []
53
54
  @redact_request_fields = []
@@ -55,7 +56,7 @@ module Visionary
55
56
  @error_key_field = "error.key"
56
57
  @system_metrics = true
57
58
  @system_interval = 15.0
58
- @system_retention_days = 30
59
+ @system_retention_days = 0 # 0: off, as above
59
60
  @disk_path = ""
60
61
  @service = nil
61
62
  @version = nil
@@ -63,6 +64,8 @@ module Visionary
63
64
  @span_threshold_ms = 100.0
64
65
  @span_max = 100
65
66
  @errors = true
67
+ # false: never profile, whatever the dashboard says.
68
+ @profile = true
66
69
  end
67
70
 
68
71
  def self.from_env(env = ENV)
@@ -78,7 +81,7 @@ module Visionary
78
81
  c.path_prefix = s.("VISIONARY_LOG_PATH_PREFIX")
79
82
  c.capture_bodies = parse_bool(s.("VISIONARY_LOG_BODIES"), true)
80
83
  c.max_body_bytes = parse_int(s.("VISIONARY_LOG_MAX_BODY_BYTES"), 8192)
81
- c.log_retention_days = parse_int(s.("VISIONARY_LOG_RETENTION_DAYS"), 30)
84
+ c.log_retention_days = parse_int(s.("VISIONARY_LOG_RETENTION_DAYS"), 0)
82
85
  c.redact_headers = parse_list(s.("VISIONARY_LOG_REDACT_HEADERS"))
83
86
  c.redact_fields = parse_list(s.("VISIONARY_LOG_REDACT_FIELDS"))
84
87
  c.redact_request_fields = parse_list(s.("VISIONARY_LOG_REDACT_REQUEST_FIELDS"))
@@ -86,13 +89,14 @@ module Visionary
86
89
  c.error_key_field = s.("VISIONARY_LOG_ERROR_KEY_FIELD").then { |v| v.empty? ? "error.key" : v }
87
90
  c.system_metrics = parse_bool(s.("VISIONARY_SYSTEM_METRICS"), true)
88
91
  c.system_interval = parse_duration(s.("VISIONARY_SYSTEM_INTERVAL"), 15.0)
89
- c.system_retention_days = parse_int(s.("VISIONARY_SYSTEM_RETENTION_DAYS"), 30)
92
+ c.system_retention_days = parse_int(s.("VISIONARY_SYSTEM_RETENTION_DAYS"), 0)
90
93
  c.service = s.("VISIONARY_SERVICE").then { |v| v.empty? ? nil : v }
91
94
  c.version = release_from(env)
92
95
  c.spans = parse_bool(s.("VISIONARY_SPANS"), true)
93
96
  c.span_threshold_ms = parse_float(s.("VISIONARY_SPAN_THRESHOLD_MS"), 100.0)
94
97
  c.span_max = parse_int(s.("VISIONARY_SPAN_MAX"), 100)
95
98
  c.errors = parse_bool(s.("VISIONARY_ERRORS"), true)
99
+ c.profile = parse_bool(s.("VISIONARY_PROFILE"), true)
96
100
  c
97
101
  end
98
102
 
@@ -28,6 +28,8 @@ module Visionary
28
28
  @errors = []
29
29
  @db_count = @http_count = 0
30
30
  @db_ms = @http_ms = 0.0
31
+ @logged = false
32
+ @late = 0
31
33
  end
32
34
 
33
35
  def set_user(id, handle, role)
@@ -47,8 +49,14 @@ module Visionary
47
49
  end
48
50
 
49
51
  # Database and HTTP calls always count; rows are kept up to the cap.
52
+ # Returns nil when the span joined the request, or, once the request has
53
+ # been logged (work it started in the background), how many such late
54
+ # spans the request has had, this one included: the caller writes it on
55
+ # its own and it never touches the request's totals.
50
56
  def add_span(span)
51
57
  @mutex.synchronize do
58
+ return @late += 1 if @logged
59
+
52
60
  case span.kind
53
61
  when "db"
54
62
  @db_count += 1
@@ -59,9 +67,16 @@ module Visionary
59
67
  @http_ms += span.duration_ms
60
68
  end
61
69
  @spans << span if @spans.size < @span_max
70
+ nil
62
71
  end
63
72
  end
64
73
 
74
+ # Called when the request's row is being built: spans that end from now
75
+ # on are late and no longer change totals or spans.
76
+ def logged!
77
+ @mutex.synchronize { @logged = true }
78
+ end
79
+
65
80
  def add_error(row)
66
81
  @mutex.synchronize { @errors << row }
67
82
  end
@@ -91,7 +106,7 @@ module Visionary
91
106
  end
92
107
  end
93
108
 
94
- # One database, HTTP, cache, queue or custom call.
109
+ # One database, HTTP, cache, queue, custom, graphql or background call.
95
110
  Span = Struct.new(:kind, :name, :target, :detail, :started_at, :offset_ms, :duration_ms,
96
111
  :status, :error, :row_count, :request_id, keyword_init: true)
97
112
  end
@@ -71,6 +71,8 @@ module Visionary
71
71
  # Not logged: count it, and still record what escapes.
72
72
  def passthrough(agent, env)
73
73
  agent.http.started
74
+ Profiler::Tally.started!
75
+ t0 = Telemetry.now_ms
74
76
  status = 500
75
77
  begin
76
78
  res = @app.call(env)
@@ -81,6 +83,11 @@ module Visionary
81
83
  raise
82
84
  ensure
83
85
  agent.http.finished(status)
86
+ if Profiler::Tally.busy?
87
+ route = agent.route_for(env)
88
+ route = "#{env['SCRIPT_NAME']}#{env['PATH_INFO']}" if route.empty?
89
+ Profiler::Tally.finished(env["REQUEST_METHOD"], route, Telemetry.now_ms - t0)
90
+ end
84
91
  end
85
92
  end
86
93
 
@@ -102,6 +109,7 @@ module Visionary
102
109
  started_mono: t0, span_max: cfg.span_max.to_i, db_visible: Telemetry::ActiveRecordHook.installed?
103
110
  )
104
111
  agent.http.started
112
+ Profiler::Tally.started!
105
113
  outer = RequestContext.current
106
114
  RequestContext.current = ctx
107
115
  begin
@@ -133,9 +141,11 @@ module Visionary
133
141
 
134
142
  def finish(agent, env, ctx, headers, req_capture, status, res_headers, resp_capture, t0)
135
143
  agent.http.finished(status)
144
+ ctx.logged!
136
145
  duration = (Telemetry.now_ms - t0).round(3)
137
146
  route = ctx.route
138
147
  route = agent.route_for(env) if route.empty?
148
+ Profiler::Tally.finished(ctx.method, route.empty? ? ctx.path : route, duration)
139
149
  user_id, handle, role = ctx.user
140
150
  tel = agent.telemetry
141
151
  entry = Entry.new(
@@ -150,7 +160,7 @@ module Visionary
150
160
  response_body: resp_capture.bytes, response_truncated: resp_capture.truncated?,
151
161
  response_size: [self.class.header(res_headers, "content-length").to_i, resp_capture.size].max,
152
162
  totals: tel&.spans_on ? ctx.totals : [nil, nil, nil, nil],
153
- span_rows: tel ? tel.request_span_rows(ctx, duration, status) : [],
163
+ span_rows: tel ? tel.request_span_rows(ctx) : [],
154
164
  error_rows: tel ? ctx.errors.map { |row| tel.error_row(row, ctx, route, user_id) } : []
155
165
  )
156
166
  agent.enqueue([:request, entry])
@@ -0,0 +1,522 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "json"
4
+ require "rbconfig"
5
+
6
+ module Visionary
7
+ # CPU profiles, switched on and off from the dashboard (Settings ->
8
+ # Profiling, kept in visionary_settings). One background thread reads the
9
+ # settings every 30 s and, while profiling is on, samples the stacks of the
10
+ # threads that are running every 10 ms for profile.duration_ms every
11
+ # profile.interval_s. Each profile is one profiles row: folded stacks plus
12
+ # the time each route's requests took while it ran.
13
+ #
14
+ # Ruby has no CPU-time sampler without a native extension, so this walks
15
+ # Thread.list and keeps the threads whose status is "run". Threads that
16
+ # sleep, wait on a lock or queue, or block on IO report "sleep" and are
17
+ # left out. Of the "run" threads only one holds the GVL at a time; the
18
+ # others are waiting for it (or inside a C call that released it), so with
19
+ # several busy threads each is counted, which over-counts waiting time.
20
+ # Each sample stands for about 10 ms: a tick that comes late because a
21
+ # busy thread held the GVL counts once per 10 ms it covers.
22
+ class Profiler
23
+ POLL_S = 30
24
+ MAX_FOLDED = 90_000
25
+ KEEP_WHOLE = 72_000 # the heaviest stacks kept whole; the rest roll up (see fold)
26
+ MIN_SAMPLES = 20
27
+ TICK_S = 0.01
28
+ NOW_FRESH_S = 300
29
+ DEFAULT_INTERVAL_S = 300.0
30
+ DEFAULT_DURATION_MS = 10_000.0
31
+ MAX_DURATION_MS = 120_000.0
32
+ SETTINGS_SQL = "SELECT name, value FROM visionary_settings WHERE name LIKE 'profile.%'"
33
+
34
+ # Requests seen by the middleware: how many started (a turn with none
35
+ # new is skipped) and, while a profile runs, the ms each "METHOD route"
36
+ # spent on requests that finished.
37
+ module Tally
38
+ @mutex = Mutex.new
39
+ @started = 0
40
+ @busy = nil
41
+
42
+ class << self
43
+ attr_reader :started
44
+
45
+ # Under the GVL a lost increment is possible but harmless: only
46
+ # "did it change" matters.
47
+ def started!
48
+ @started += 1
49
+ end
50
+
51
+ def busy? = !@busy.nil?
52
+
53
+ def finished(method, route, ms)
54
+ return unless @busy
55
+ key = "#{method} #{route}"
56
+ @mutex.synchronize { @busy[key] = (@busy[key] || 0.0) + ms.to_f if @busy }
57
+ rescue StandardError
58
+ nil
59
+ end
60
+
61
+ def begin!
62
+ @mutex.synchronize { @busy = {} }
63
+ end
64
+
65
+ # The busy ms since begin!, and stops counting.
66
+ def finish!
67
+ @mutex.synchronize do
68
+ out = @busy || {}
69
+ @busy = nil
70
+ out
71
+ end
72
+ end
73
+
74
+ def after_fork
75
+ @mutex = Mutex.new
76
+ @busy = nil
77
+ end
78
+ end
79
+ end
80
+
81
+ Settings = Struct.new(:enabled, :services, :interval_s, :duration_ms, :now, keyword_init: true)
82
+
83
+ class << self
84
+ # visionary_settings rows ([[name, value], ...]) as Settings, with the
85
+ # spec's defaults for missing or unreadable values.
86
+ def parse_settings(rows)
87
+ h = {}
88
+ rows.each { |name, value| h[name.to_s] = value.to_s.strip }
89
+ services = begin
90
+ list = h["profile.services"].to_s.empty? ? nil : JSON.parse(h["profile.services"])
91
+ list.is_a?(Array) ? list.map(&:to_s) : nil
92
+ rescue JSON::ParserError
93
+ nil
94
+ end
95
+ Settings.new(
96
+ enabled: %w[true 1].include?(h["profile.enabled"].to_s.downcase),
97
+ services: services,
98
+ interval_s: positive(h["profile.interval_s"], DEFAULT_INTERVAL_S),
99
+ duration_ms: [positive(h["profile.duration_ms"], DEFAULT_DURATION_MS), MAX_DURATION_MS].min,
100
+ now: h["profile.now"].to_s.empty? ? nil : h["profile.now"]
101
+ )
102
+ end
103
+
104
+ # A missing or empty list means every service; a service with no name
105
+ # matches only that.
106
+ def participates?(services, service)
107
+ return true if services.nil? || services.empty?
108
+ return false if service.to_s.empty?
109
+ services.include?(service.to_s)
110
+ end
111
+
112
+ # profile.now asks for one profile when it is newer than this process's
113
+ # start and less than five minutes old.
114
+ def now_due?(value, started_at, now = Time.now)
115
+ t = parse_time(value)
116
+ !t.nil? && t > started_at && now - t < NOW_FRESH_S
117
+ end
118
+
119
+ # created_at text ("2026-10-22T10:00:00.000Z", or with a space and no Z)
120
+ # as a UTC Time.
121
+ def parse_time(s)
122
+ m = s.to_s.strip.match(/\A(\d{4})-(\d\d)-(\d\d)[T ](\d\d):(\d\d):(\d\d)(\.\d+)?(Z|[+-]00:?00)?\z/)
123
+ return nil unless m
124
+ Time.utc(m[1].to_i, m[2].to_i, m[3].to_i, m[4].to_i, m[5].to_i, m[6].to_i + m[7].to_f)
125
+ rescue ArgumentError
126
+ nil
127
+ end
128
+
129
+ # Stack counts ({"a;b;c" => n}) as folded text, heaviest first within
130
+ # MAX_FOLDED bytes (sdks/SPEC.md, "CPU profiles"): whole lines up to
131
+ # KEEP_WHOLE, then the rest cut after the deepest frame they share with
132
+ # a kept stack and ending in "(other)", then one "(other) n" for
133
+ # whatever is left.
134
+ def fold(counts)
135
+ samples = counts.values.sum
136
+ sorted = counts.sort_by { |stack, n| [-n, stack] }
137
+ out = +""
138
+ # Whole lines first, leaving room for the rolled-up ones.
139
+ i = 0
140
+ while i < sorted.size
141
+ line = "#{sorted[i][0]} #{sorted[i][1]}\n"
142
+ break if out.bytesize + line.bytesize > KEEP_WHOLE
143
+ out << line
144
+ i += 1
145
+ end
146
+ return [out, samples] if i == sorted.size
147
+
148
+ # The rest are cut after the deepest frame they share with a kept
149
+ # stack and end in "(other)", so their time stays under the right caller.
150
+ prefixes = {}
151
+ sorted[0...i].each do |stack, _|
152
+ j = -1
153
+ prefixes[stack[0...j]] = true while (j = stack.index(";", j + 1))
154
+ prefixes[stack] = true
155
+ end
156
+ rolled = Hash.new(0)
157
+ other = 0
158
+ sorted[i..].each do |stack, n|
159
+ cut = ""
160
+ j = -1
161
+ while (j = stack.index(";", j + 1))
162
+ cut = stack[0...j] if prefixes[stack[0...j]]
163
+ end
164
+ cut = stack if prefixes[stack]
165
+ if cut.empty?
166
+ other += n
167
+ elsif cut == stack # the whole stack leads into a kept one: nothing to cut
168
+ rolled[stack] += n
169
+ else
170
+ rolled["#{cut};(other)"] += n
171
+ end
172
+ end
173
+ rolled.sort_by { |stack, n| [-n, stack] }.each do |stack, n|
174
+ line = "#{stack} #{n}\n"
175
+ if out.bytesize + line.bytesize > MAX_FOLDED - 32 # room for the last "(other) n"
176
+ other += n
177
+ else
178
+ out << line
179
+ end
180
+ end
181
+ out << "(other) #{other}\n" if other.positive?
182
+ [out, samples]
183
+ end
184
+
185
+ # Whether the sampler takes a stack of this thread: it is using the
186
+ # CPU (or waiting for the GVL) and is not one of Visionary's own.
187
+ def sample?(thread, current = Thread.current)
188
+ thread.status == "run" && !own?(thread, current)
189
+ rescue StandardError
190
+ false
191
+ end
192
+
193
+ # The sampler itself and Visionary's background threads.
194
+ def own?(thread, current = Thread.current)
195
+ thread.equal?(current) || thread.name.to_s.start_with?("visionary-") ||
196
+ thread[Telemetry::INTERNAL] == true
197
+ end
198
+
199
+ private
200
+
201
+ def positive(v, default)
202
+ n = Float(v.to_s)
203
+ n.positive? && n.finite? ? n : default
204
+ rescue ArgumentError
205
+ default
206
+ end
207
+ end
208
+
209
+ # Turns backtrace locations into "label file:line" frames, where file
210
+ # says whose code it is: relative to the working directory for the app,
211
+ # lib:<gem>/<path> for gems, std:<path> for Ruby's own library.
212
+ class Frames
213
+ def initialize(root: Dir.pwd, gem_dirs: Frames.gem_dirs, std_dirs: Frames.std_dirs,
214
+ fold_case: Gem.win_platform?)
215
+ @fold_case = fold_case
216
+ @root = dir(root)
217
+ @gem_dirs = gem_dirs.map { |d| dir(d) }.uniq.sort_by { |d| -d.length }
218
+ @std_dirs = std_dirs.map { |d| dir(d) }.uniq.sort_by { |d| -d.length }
219
+ @files = {}
220
+ end
221
+
222
+ def self.gem_dirs
223
+ dirs = Gem.path.flat_map { |p| ["#{p}/gems", "#{p}/bundler/gems"] }
224
+ if defined?(::Bundler) && ::Bundler.respond_to?(:bundle_path)
225
+ begin
226
+ bp = ::Bundler.bundle_path.to_s
227
+ dirs.push("#{bp}/gems", "#{bp}/bundler/gems")
228
+ rescue StandardError
229
+ nil
230
+ end
231
+ end
232
+ dirs
233
+ end
234
+
235
+ def self.std_dirs
236
+ RbConfig::CONFIG.values_at("rubyarchdir", "rubylibdir").compact.reject(&:empty?)
237
+ end
238
+
239
+ # Outermost caller first. Only the sampled frame keeps its line: callers
240
+ # are one frame per function, whichever line they called from.
241
+ def stack(locations)
242
+ leaf = locations.first
243
+ locations.reverse_each.map { |loc| frame(loc, loc.equal?(leaf)) }.join(";")
244
+ end
245
+
246
+ def frame(loc, line = true)
247
+ label = loc.label || loc.base_label || "(unknown)"
248
+ file = file(loc.absolute_path || loc.path)
249
+ name = if file
250
+ line ? "#{label} #{file}:#{loc.lineno}" : "#{label} #{file}"
251
+ else
252
+ label.to_s
253
+ end
254
+ name.include?(";") ? name.tr(";", ",") : name
255
+ end
256
+
257
+ def file(path)
258
+ return nil if path.nil? || path.empty?
259
+ @files.fetch(path) { @files[path] = classify(path) }
260
+ end
261
+
262
+ private
263
+
264
+ def classify(path)
265
+ # Ruby's own library compiled in (Kernel#loop, Integer#times, ...):
266
+ # <internal:kernel> is kernel.rb in Ruby's source.
267
+ return "std:#{path[10..-2]}.rb" if path.start_with?("<internal:") && path.end_with?(">")
268
+ return nil if path.start_with?("(", "<")
269
+
270
+ p = path.tr("\\", "/")
271
+ key = @fold_case ? p.downcase : p
272
+ @gem_dirs.each do |d|
273
+ next unless key.start_with?(d)
274
+ return gem_file(p[d.length..])
275
+ end
276
+ if (m = p.match(%r{/vendor/bundle/[^/]+/[^/]+/(?:bundler/)?gems/(.+)\z}))
277
+ return gem_file(m[1])
278
+ end
279
+ @std_dirs.each do |d|
280
+ return "std:#{p[d.length..]}" if key.start_with?(d)
281
+ end
282
+ return p[@root.length..] if key.start_with?(@root)
283
+ p
284
+ end
285
+
286
+ # "rack-3.2.7/lib/rack/builder.rb" -> "lib:rack/rack/builder.rb".
287
+ def gem_file(rest)
288
+ dir, inner = rest.split("/", 2)
289
+ name = dir.sub(/-\d[^-]*(?:-.+)?\z/, "").sub(/-\h{7,40}\z/, "")
290
+ name = dir if name.empty?
291
+ "lib:#{name}/#{inner.to_s.delete_prefix('lib/')}"
292
+ end
293
+
294
+ # With a trailing slash, case-folded where the file system is.
295
+ def dir(d)
296
+ s = d.to_s.tr("\\", "/").chomp("/") + "/"
297
+ @fold_case ? s.downcase : s
298
+ end
299
+ end
300
+
301
+ # CRuby lets a thread that computes keep the GVL for 100 ms (shifted by
302
+ # its priority) while another waits for it. A request that computes for
303
+ # 40 ms and then writes its response gives the GVL up at the write, when
304
+ # it no longer runs, so a sampler waiting for the GVL would rarely catch
305
+ # it computing. While a profile runs, the app's threads get the lowest
306
+ # priority (-3: a 12.5 ms quantum) and get their own back afterwards. In
307
+ # CRuby the priority only sets that quantum; all threads drop together,
308
+ # so none gains on another.
309
+ class Quantum
310
+ LOWEST = -3
311
+
312
+ def initialize(enabled: RUBY_ENGINE == "ruby")
313
+ @enabled = enabled
314
+ @saved = {}
315
+ @first = true
316
+ end
317
+
318
+ def shorten(thread)
319
+ return if !@enabled || @saved.key?(thread)
320
+ prio = thread.priority
321
+ # A thread started during the profile inherited the lowered priority.
322
+ @saved[thread] = !@first && prio == LOWEST ? 0 : prio
323
+ thread.priority = LOWEST if prio > LOWEST
324
+ rescue StandardError
325
+ nil
326
+ end
327
+
328
+ def first_pass_done
329
+ @first = false
330
+ end
331
+
332
+ def restore
333
+ return unless @enabled
334
+ @saved.each do |t, prio|
335
+ t.priority = prio if t.alive? && t.priority != prio
336
+ rescue StandardError
337
+ nil
338
+ end
339
+ Thread.list.each do |t|
340
+ next if @saved.key?(t) || Profiler.own?(t)
341
+ t.priority = 0 if t.priority == LOWEST
342
+ rescue StandardError
343
+ nil
344
+ end
345
+ @saved.clear
346
+ end
347
+ end
348
+
349
+ attr_reader :settings
350
+
351
+ def initialize(cfg, instance, store, writer, logger, started_at: Time.now)
352
+ @cfg = cfg
353
+ @instance = instance
354
+ @store = store
355
+ @writer = writer
356
+ @logger = logger
357
+ @started_at = started_at
358
+ @frames = Frames.new
359
+ @settings = Settings.new(enabled: false, services: nil, interval_s: DEFAULT_INTERVAL_S,
360
+ duration_ms: DEFAULT_DURATION_MS, now: nil)
361
+ reset
362
+ end
363
+
364
+ def start
365
+ @thread = Thread.new { run }
366
+ @thread.name = "visionary-profiler" if @thread.respond_to?(:name=)
367
+ self
368
+ end
369
+
370
+ def after_fork
371
+ Tally.after_fork
372
+ reset
373
+ start
374
+ end
375
+
376
+ def close(timeout = 1)
377
+ @mutex.synchronize do
378
+ @stopped = true
379
+ @cond.broadcast
380
+ end
381
+ @thread&.join(timeout)
382
+ end
383
+
384
+ # Whether profiles are taken on the schedule: enabled in the dashboard
385
+ # for this service.
386
+ def active?
387
+ @settings.enabled && Profiler.participates?(@settings.services, @cfg.service)
388
+ end
389
+
390
+ # Reads the settings; true when profile.now asks for a profile now.
391
+ def poll
392
+ rows = @store.rows(SETTINGS_SQL)
393
+ @read_failed = false
394
+ was = active?
395
+ @settings = Profiler.parse_settings(rows)
396
+ now_value = @settings.now
397
+ due = !now_value.nil? && now_value != @last_now &&
398
+ Profiler.now_due?(now_value, @started_at) &&
399
+ Profiler.participates?(@settings.services, @cfg.service)
400
+ @last_now = now_value
401
+ log_switch(was)
402
+ due
403
+ rescue StandardError => e
404
+ unless @read_failed
405
+ @read_failed = true
406
+ @logger.info("visionary: profiling settings unreadable (#{e.message.lines.first.to_s.strip}); profiling stays off")
407
+ end
408
+ was = active?
409
+ @settings = Profiler.parse_settings([])
410
+ log_switch(was)
411
+ false
412
+ end
413
+
414
+ # Samples for duration_ms and queues the profile. Returns the row, or
415
+ # nil when it was dropped (too few samples, no finished request).
416
+ def capture
417
+ @last_started = Tally.started
418
+ duration_s = @settings.duration_ms / 1000.0
419
+ counts = Hash.new(0)
420
+ started_at = Time.now
421
+ t0 = mono
422
+ deadline = t0 + duration_s
423
+ quantum = Quantum.new
424
+ Tally.begin!
425
+ last = mono
426
+ begin
427
+ loop do
428
+ tick = mono
429
+ # A thread busy in Ruby code keeps the GVL for a quantum before the
430
+ # sampler gets it back, so a late tick stands for the ticks it
431
+ # missed: the stack it finds is the one that held the GVL.
432
+ weight = [((tick - last) / TICK_S).round, 1].max
433
+ last = tick
434
+ Thread.list.each do |t|
435
+ next if Profiler.own?(t)
436
+ quantum.shorten(t)
437
+ next unless t.status == "run"
438
+ locs = t.backtrace_locations
439
+ counts[@frames.stack(locs)] += weight if locs && !locs.empty?
440
+ end
441
+ quantum.first_pass_done
442
+ break if mono >= deadline || wait(TICK_S - (mono - tick))
443
+ end
444
+ ensure
445
+ quantum&.restore
446
+ busy = Tally.finish!
447
+ end
448
+ elapsed_ms = ((mono - t0) * 1000).round(3)
449
+ folded, samples = Profiler.fold(counts)
450
+ routes = busy.transform_values { |ms| [ms.round, 1].max }
451
+ return nil if samples < MIN_SAMPLES || routes.empty?
452
+
453
+ row = [Visionary.timestamp(started_at), @cfg.service, @instance, @cfg.version, "ruby",
454
+ elapsed_ms, samples, JSON.generate(routes), folded]
455
+ @writer.enqueue([:profile, row])
456
+ row
457
+ rescue StandardError => e
458
+ @logger.warn("visionary: CPU profile failed: #{e.class}: #{e.message}")
459
+ nil
460
+ end
461
+
462
+ private
463
+
464
+ def reset
465
+ @mutex = Mutex.new
466
+ @cond = ConditionVariable.new
467
+ @stopped = false
468
+ @thread = nil
469
+ @last_started = Tally.started
470
+ end
471
+
472
+ def run
473
+ Telemetry.internal!
474
+ next_poll = mono
475
+ next_profile = nil
476
+ until stopped?
477
+ if mono >= next_poll
478
+ next_poll = mono + POLL_S
479
+ if poll
480
+ @logger.info("visionary: taking a CPU profile now (asked for in the dashboard)")
481
+ capture
482
+ next_profile = mono + @settings.interval_s if next_profile
483
+ end
484
+ end
485
+ if active?
486
+ next_profile ||= mono + @settings.interval_s
487
+ if mono >= next_profile
488
+ capture unless Tally.started == @last_started
489
+ next_profile = mono + @settings.interval_s
490
+ end
491
+ else
492
+ next_profile = nil
493
+ end
494
+ break if wait([next_poll, next_profile].compact.min - mono)
495
+ end
496
+ rescue StandardError => e
497
+ @logger.error("visionary: profiler stopped: #{e.class}: #{e.message}")
498
+ end
499
+
500
+ def log_switch(was)
501
+ now = active?
502
+ return if now == was
503
+ if now
504
+ @logger.info("visionary: CPU profiling on (#{@settings.duration_ms.round} ms every #{@settings.interval_s}s)")
505
+ else
506
+ @logger.info("visionary: CPU profiling off")
507
+ end
508
+ end
509
+
510
+ # Waits up to secs; true when stopped.
511
+ def wait(secs)
512
+ @mutex.synchronize do
513
+ @cond.wait(@mutex, secs) if !@stopped && secs.positive?
514
+ @stopped
515
+ end
516
+ end
517
+
518
+ def stopped? = @mutex.synchronize { @stopped }
519
+
520
+ def mono = Process.clock_gettime(Process::CLOCK_MONOTONIC)
521
+ end
522
+ end
@@ -8,7 +8,8 @@ module Visionary
8
8
  EVERY = 24 * 3600
9
9
  MAX_ROUNDS = 200
10
10
 
11
- # tables: { "api_logs" => days, ... }; 0 days keeps forever.
11
+ # tables: { "api_logs" => days, ... }; 0 days (the default) purges nothing
12
+ # and starts no thread: the Visionary server applies Settings → Retention.
12
13
  def initialize(store, logger, tables)
13
14
  @store = store
14
15
  @logger = logger
@@ -46,6 +47,8 @@ module Visionary
46
47
  n = begin
47
48
  @store.changes(sql, [cutoff])
48
49
  rescue StandardError => e
50
+ # The server creates profiles; before it has, there is nothing to purge.
51
+ return total if table == "profiles" && e.message.match?(/no such table|does(?:n't| not) exist/i)
49
52
  @logger.error("visionary: retention purge of #{table} failed after #{total} rows: #{e.message}")
50
53
  return total
51
54
  end
@@ -64,12 +64,16 @@ module Visionary
64
64
 
65
65
  RELEASE_COLUMNS = %w[created_at service instance version sdk sdk_version runtime].freeze
66
66
 
67
+ # Created by the Visionary server, never by the SDK.
68
+ PROFILE_COLUMNS = %w[created_at service instance version sdk duration_ms samples routes folded].freeze
69
+
67
70
  TABLES = {
68
71
  "api_logs" => REQUEST_COLUMNS + ADDED_REQUEST_COLUMNS.map(&:first),
69
72
  "system_metrics" => SYSTEM_BASE_COLUMNS + ADDED_SYSTEM_COLUMNS.map(&:first),
70
73
  "spans" => SPAN_COLUMNS,
71
74
  "errors" => ERROR_COLUMNS,
72
- "releases" => RELEASE_COLUMNS
75
+ "releases" => RELEASE_COLUMNS,
76
+ "profiles" => PROFILE_COLUMNS
73
77
  }.freeze
74
78
 
75
79
  # Which script creates a table, and the columns older copies may lack.
@@ -40,6 +40,11 @@ module Visionary
40
40
  with { |c| c.select_values(sql).map(&:to_s) }
41
41
  end
42
42
 
43
+ # Every row of a query, as arrays of values.
44
+ def rows(sql)
45
+ with { |c| c.select_rows(sql) }
46
+ end
47
+
43
48
  def batch(stmts)
44
49
  with do |c|
45
50
  c.transaction(requires_new: true) do
@@ -42,6 +42,13 @@ module Visionary
42
42
  Array(rows).map { |r| r.is_a?(Array) ? r[0].to_s : nil }.compact
43
43
  end
44
44
 
45
+ # Every row of a query, as arrays of values.
46
+ def rows(sql)
47
+ result = post(@raw_uri, { sql: sql })
48
+ rows = result.is_a?(Array) && result.dig(0, "results", "rows")
49
+ Array(rows).select { |r| r.is_a?(Array) }
50
+ end
51
+
45
52
  def batch(stmts)
46
53
  stmts.each_slice(MAX_BATCH) do |chunk|
47
54
  post(@query_uri, { batch: chunk.map { |sql, args| { sql: sql, params: args.map { |a| self.class.param(a) } } } })
@@ -39,6 +39,11 @@ module Visionary
39
39
  run { |c| c.column(sql) }
40
40
  end
41
41
 
42
+ # Every row of a query, as arrays of values.
43
+ def rows(sql)
44
+ run { |c| c.rows(sql) }
45
+ end
46
+
42
47
  # One transaction for the whole batch.
43
48
  def batch(stmts)
44
49
  run do |c|
@@ -157,7 +162,7 @@ module Visionary
157
162
  end
158
163
 
159
164
  # Thin driver adapters: execute returns affected rows, column returns the
160
- # first column of every row.
165
+ # first column of every row, rows every row as an array.
161
166
  class PgConn
162
167
  def initialize(conn)
163
168
  @conn = conn
@@ -177,6 +182,13 @@ module Visionary
177
182
  res&.clear
178
183
  end
179
184
 
185
+ def rows(sql)
186
+ res = @conn.exec(sql)
187
+ res.values
188
+ ensure
189
+ res&.clear
190
+ end
191
+
180
192
  def broken?
181
193
  @conn.finished? || @conn.status != PG::CONNECTION_OK
182
194
  rescue StandardError
@@ -231,6 +243,10 @@ module Visionary
231
243
  @client.query(sql, as: :array).map { |r| r[0].to_s }
232
244
  end
233
245
 
246
+ def rows(sql)
247
+ @client.query(sql, as: :array).to_a
248
+ end
249
+
234
250
  def broken?
235
251
  !@client.ping
236
252
  rescue StandardError
@@ -262,6 +278,8 @@ module Visionary
262
278
  @db.execute(sql).map { |r| r[0].to_s }
263
279
  end
264
280
 
281
+ def rows(sql) = @db.execute(sql)
282
+
265
283
  def broken? = @db.closed?
266
284
 
267
285
  def close
@@ -7,7 +7,7 @@ require_relative "store/active_record"
7
7
  module Visionary
8
8
  # Where rows go: D1 over REST, a connection opened from a database URL, or
9
9
  # the app's ActiveRecord pool. Every store answers the same calls:
10
- # exec_script, exec, column_names, batch([[sql, args], ...]), changes,
10
+ # exec_script, exec, column_names, rows, batch([[sql, args], ...]), changes,
11
11
  # after_fork and close.
12
12
  module Store
13
13
  # Picks the store from the config, or nil when none is configured.
@@ -15,7 +15,7 @@ module Visionary
15
15
  "REPLACE" => /\bINTO\s+([\w."`\[\]]+)/i,
16
16
  "UPDATE" => /\AUPDATE\s+(?:ONLY\s+)?([\w."`\[\]]+)/i
17
17
  }.freeze
18
- MAX_DETAIL = 2000
18
+ MAX_DETAIL = 8000
19
19
 
20
20
  module_function
21
21
 
@@ -37,6 +37,7 @@ module Visionary
37
37
  @spans_on = cfg.spans ? true : false
38
38
  @errors_on = cfg.errors ? true : false
39
39
  @threshold = cfg.span_threshold_ms.to_f
40
+ @span_max = cfg.span_max.to_i
40
41
  @limiter = Errors::Limiter.new
41
42
  end
42
43
 
@@ -56,7 +57,10 @@ module Visionary
56
57
  row_count: row_count, request_id: ctx&.request_id
57
58
  )
58
59
  if ctx
59
- ctx.add_span(span)
60
+ # A late span (its request was already logged) is written on its
61
+ # own: always up to span_max of them per request, then only slow ones.
62
+ late = ctx.add_span(span)
63
+ @writer.enqueue([:span, span_row(span)]) if late && (late <= @span_max || duration_ms >= @threshold)
60
64
  elsif duration_ms >= @threshold
61
65
  @writer.enqueue([:span, span_row(span)])
62
66
  end
@@ -69,9 +73,9 @@ module Visionary
69
73
  span.target, span.detail, span.offset_ms, span.duration_ms, span.status, span.error, span.row_count]
70
74
  end
71
75
 
72
- # Rows for a finished request's spans when it was slow or failed.
73
- def request_span_rows(ctx, duration_ms, status)
74
- return [] unless @spans_on && (duration_ms >= @threshold || status >= 500)
76
+ # Rows for a finished request's spans, kept for every request.
77
+ def request_span_rows(ctx)
78
+ return [] unless @spans_on
75
79
  ctx.spans.map { |s| span_row(s) }
76
80
  end
77
81
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Visionary
4
- VERSION = "0.5.2"
4
+ VERSION = "0.6.0"
5
5
  end
@@ -12,10 +12,10 @@ module Visionary
12
12
  )
13
13
 
14
14
  # The single background thread that batches api_logs, spans, errors and
15
- # releases rows. Queue items are [:request, Entry], [:span, row],
16
- # [:error, row] or [:release, row].
15
+ # releases rows, and CPU profiles. Queue items are [:request, Entry],
16
+ # [:span, row], [:error, row], [:release, row] or [:profile, row].
17
17
  class Writer
18
- INSERTS = { request: "api_logs", span: "spans", error: "errors", release: "releases" }.freeze
18
+ INSERTS = { request: "api_logs", span: "spans", error: "errors", release: "releases", profile: "profiles" }.freeze
19
19
 
20
20
  def initialize(cfg, store, schemas, logger)
21
21
  @cfg = cfg
@@ -93,7 +93,15 @@ module Visionary
93
93
  @logger.warn("visionary: queue full; #{n} entries dropped") if n.positive?
94
94
  end
95
95
 
96
+ # A profile goes in its own transaction: the server creates its table, so
97
+ # if that is missing only the profile is lost.
96
98
  def write(batch)
99
+ profiles, rest = batch.partition { |kind, _| kind == :profile }
100
+ commit(rest) unless rest.empty?
101
+ profiles.each { |item| commit([item]) }
102
+ end
103
+
104
+ def commit(batch)
97
105
  stmts = begin
98
106
  users = lookup_users(batch)
99
107
  batch.flat_map { |kind, item| statements(kind, item, users) }
data/lib/visionary.rb CHANGED
@@ -15,6 +15,7 @@ require_relative "visionary/system/cgroup"
15
15
  require_relative "visionary/system/recorder"
16
16
  require_relative "visionary/telemetry"
17
17
  require_relative "visionary/telemetry/hooks"
18
+ require_relative "visionary/profiler"
18
19
  require_relative "visionary/body"
19
20
  require_relative "visionary/middleware"
20
21
  require_relative "visionary/agent"
@@ -96,7 +97,7 @@ module Visionary
96
97
  end
97
98
 
98
99
  # Measures the block as a span and returns its value. kind is "custom",
99
- # "cache", "queue", "db" or "http"; name should be low-cardinality (the
100
+ # "cache", "queue", "db", "http", "graphql" or "background"; name should be low-cardinality (the
100
101
  # operation, not its arguments).
101
102
  #
102
103
  # Visionary.span("charge card", kind: "http", target: "api.stripe.com") { stripe.charge(...) }
@@ -119,6 +120,30 @@ module Visionary
119
120
  end
120
121
  end
121
122
 
123
+ # Runs the block in a new Thread that keeps belonging to the current
124
+ # request, and records it as one span of kind "background" named name,
125
+ # from start to finish. Calls made inside it carry the request's
126
+ # request_id and offset_ms even after the response was sent; those ending
127
+ # after the request was logged are written on their own and never change
128
+ # its totals. Outside a request the span is a lone span (kept when it
129
+ # takes at least VISIONARY_SPAN_THRESHOLD_MS).
130
+ #
131
+ # If the block raises, the span gets status "error" and the message, and
132
+ # the exception is re-raised in the thread as Ruby normally does: printed
133
+ # when Thread.report_on_exception is on, raised again by join/value.
134
+ # Returns the Thread.
135
+ #
136
+ # Visionary.background("send welcome email") { Mailer.welcome(user).deliver_now }
137
+ def background(name, &block)
138
+ raise ArgumentError, "Visionary.background needs a block" unless block
139
+
140
+ ctx = RequestContext.current # thread locals don't carry over on Ruby 3.1
141
+ Thread.new do
142
+ RequestContext.current = ctx
143
+ span(name, kind: "background", &block)
144
+ end
145
+ end
146
+
122
147
  # Records a handled error (handled = 1). Inside a request it links to it.
123
148
  #
124
149
  # rescue PaymentError => e
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: visionary-sdk
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.2
4
+ version: 0.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - styropyr0
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2026-10-19 00:00:00.000000000 Z
11
+ date: 2026-10-22 00:00:00.000000000 Z
12
12
  dependencies: []
13
13
  description: |
14
14
  The Ruby collector for Visionary. A Rack middleware (with a Railtie for Rails)
@@ -30,6 +30,7 @@ files:
30
30
  - lib/visionary/context.rb
31
31
  - lib/visionary/fork.rb
32
32
  - lib/visionary/middleware.rb
33
+ - lib/visionary/profiler.rb
33
34
  - lib/visionary/queue.rb
34
35
  - lib/visionary/railtie.rb
35
36
  - lib/visionary/redactor.rb