forge_ops_tracker 0.5.0 → 0.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f15d33570709adad8c061ee0c432d9a6484fc50881076fae1493b892562188e7
-  data.tar.gz: f5fada725654a4006a6a44ee0680fb5f7f34cb808087305596892101c743a0ef
+  metadata.gz: 8366b59c60be4a6766a7d7775e9eaeed1e184645e620351e81aa97bc88c0cacd
+  data.tar.gz: a971669ad792a6893bdcddb577050883cb3d8f175778d817f02174c9428b5acc
 SHA512:
-  metadata.gz: 66513f4aebaeb443198a73bad548faaab86653841d213c37000bd343de406926053faa58b65be68934d90c8e12cc1393061ee763e7b5541c4545e90713910fb9
-  data.tar.gz: 86eb61aca41a42a55a6df37df0d8ef7789b5342055239640a593310da438cda20e3d6e959a0b229c4dbcbe137089e31d8bd537b1eb8d4e91c72bec4acfd6ec8c
+  metadata.gz: 03c78e9de07b46238931804ce2743c5ce659efbfdb5ae86151ab64a355c31a9e2fc142f07e22ad15d2e12bff02cb53fddaebd419f3c889ca9bd325adc3205b54
+  data.tar.gz: b882fd3008d11185934f8905ad23261bedf5c5290902da1da0a789042f829f117b9e7d2fa66361a79a8d802a1d8561e700b1af81caec8bf4fb41cbbc974eb180

data/CHANGELOG.md

@@ -1,5 +1,77 @@
 # Changelog
 
+## 0.10.1
+
+- Sidekiq job failures now report the real error. Sidekiq raises its own control-flow exception
+  (`Sidekiq::JobRetry::Handled`, or `Skip` for a job with retries turned off) once it has scheduled
+  a failed job's retry, and that wrapper, not the error inside it, was what reached ForgeOps, so
+  every failing Sidekiq job, whatever went wrong, landed in the same opaque issue. The subscriber
+  now unwraps it to its `#cause`. Confirmed against a real Sidekiq worker on 7.3 and 8.1. Among
+  other things, this is what lets ForgeOps recognize a job class that was renamed or removed while
+  jobs for it were still queued (an ActiveJob `UnknownJobClassError`, or a plain Sidekiq job's
+  `NameError`), which previously arrived only as the wrapper.
+
+## 0.10.0
+
+- Real per-occurrence job/dependency failure tracking, feeding ForgeOps's own new cross-type
+  project timeline: a job whose `perform` raised (detected via the existing `perform.active_job`
+  subscription reading ActiveSupport::Notifications' own automatic `exception_object` payload key,
+  no new hook needed) or an outbound `Net::HTTP` call that raised or came back 5xx (not 4xx) is
+  now recorded individually, not just as an aggregate failure count. New `config.track_failures`
+  (default true, independent of `track_performance`/`track_breadcrumbs`) and
+  `config.failure_event_flush_interval` (default 60s). Known, documented gap: a connection-refused
+  or DNS-failure error raised during `Net::HTTP.start`/`Net::HTTP.get_response`'s own connection
+  setup (before `#request` itself ever runs) is invisible to this detection, the same pre-existing
+  limitation that already silently applied to this SDK's outbound-HTTP breadcrumbs and timing;
+  `Net::HTTP.new(...).request(...)` (no explicit `#start`) is unaffected, since Net::HTTP connects
+  lazily inside `#request` itself in that case.
+
+## 0.9.0
+
+- Performance percentiles: every `PerformanceSample` (controller/query/http timing) now carries a
+  small latency histogram alongside its existing count/sum/max, so ForgeOps can compute approximate
+  p50/p95/p99 per transaction instead of only a weighted average. No new config; this rides the
+  existing `track_performance` flag and flush interval.
+- Background job queue latency: the existing `perform.active_job` subscription now also records how
+  long a job actually waited in its queue before starting (not how long it took to run), as a new
+  `job_latency` performance-sample kind grouped by queue name. Nil for a job run via `perform_now`,
+  which never goes through `enqueue` at all.
+- Database connection pool utilization: a new automatic reporter (matching the existing Sidekiq/
+  Solid Queue/Puma stats reporters' own shape) periodically captures this process's own ActiveRecord
+  connection pool size/busy/waiting counts as infrastructure metrics, polled on the same
+  `config.gauge_poll_interval` those reporters already use.
+
+## 0.8.0
+
+- Breadcrumbs: a trail of SQL queries, controller actions, and outbound `Net::HTTP` calls leading
+  up to an error, recorded automatically (on by default, same as every other automatic
+  instrumentation this gem does; opt out with `config.track_breadcrumbs = false`, or tune
+  `config.max_breadcrumbs`, default 30). `ForgeOpsTracker.add_breadcrumb(message, category:,
+  level:, data:)` adds your own, regardless of whether the automatic sources are on, and works
+  outside a request entirely too (a background job, a console session). Unlike the affected user,
+  a breadcrumb's message/data is scrubbed for likely PII.
+
+## 0.7.0
+
+- Automatically associates a reported error with the current signed-in user, for any Warden-based
+  auth setup (Devise included; Devise just mounts Warden automatically, this isn't Devise-
+  specific). On by default, same as every other automatic instrumentation this gem does; opt out
+  with `config.track_current_user = false`. `ForgeOpsTracker.set_user(id:, email:, username:)`
+  manually sets or overrides it, for a custom auth setup or outside a request entirely (a
+  background job, a console session).
+
+## 0.6.0
+
+- Performance monitoring now also covers database queries (`sql.active_record`, bucketed by
+  Rails' own query name, never raw SQL text), background jobs (`perform.active_job`, any backend,
+  Solid Queue included, plus raw Sidekiq workers via a server middleware, deduplicated against
+  jobs dispatched through Active Job), and outbound `Net::HTTP` calls, alongside the existing
+  controller/action timing. Each shows up as its own `kind` on the same `performance` dashboard
+  dataset. Also reports Sidekiq's, Solid Queue's, and Puma's own operational gauges (queue depth,
+  worker/dispatcher counts, Puma's thread pool) as infrastructure metrics on a periodic timer
+  (`config.gauge_poll_interval`, default 60 seconds), whenever the corresponding library is
+  already loaded. All on the same `track_performance` flag as before; no new flags to opt into.
+
 ## 0.5.0
 
 - Every reported event now carries `sdk_name` ("ruby"), so a project's language on the ForgeOps

data/README.md

@@ -67,13 +67,75 @@ Bottom line: if an exception would otherwise crash something, you're already cov
 code already catches and handles it, route that specific `rescue` through `Rails.error.handle`/
 `.record` instead of a bare one wherever you want ForgeOps to know about it.
 
+## Identifying users
+
+If you're using Devise, or any other Warden-based authentication, the currently signed-in user is
+already associated with every error automatically, on by default, no setup needed:
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_current_user = false # opt out entirely
+end
+```
+
+For a custom auth setup this can't detect, or to override its guess, call
+`ForgeOpsTracker.set_user` yourself:
+
+```ruby
+before_action do
+  ForgeOpsTracker.set_user(id: current_user&.id, email: current_user&.email)
+end
+```
+
+Also usable outside a request entirely, a background job or a console session, for the rest of
+whichever thread calls it. `id`/`email`/`username` are all independently optional; call it with
+none of them (or with a signed-out `current_user`, as the `&.` above already handles) to clear
+whatever was set. Shows up on an issue's own detail page, and as its own `affected_users_count`
+alongside the regular event count.
+
+## Breadcrumbs
+
+A trail of what happened right before an error, on by default, no setup needed: every SQL query,
+controller action, and outbound `Net::HTTP` call during a request is recorded automatically, and
+shows up alongside the error on an issue's own detail page.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_breadcrumbs = false # opt out of the automatic sources entirely
+  config.max_breadcrumbs = 30 # oldest entry dropped once this many have accumulated in one request
+end
+```
+
+Add your own by hand, regardless of whether the automatic sources are on:
+
+```ruby
+ForgeOpsTracker.add_breadcrumb("charged card", category: "billing", data: { order_id: order.id })
+```
+
+`category` defaults to `"custom"`, `level` to `"info"` (`"debug"`/`"info"`/`"warning"`/`"error"`
+are the four levels the automatic sources themselves use too), and `data` to `{}`. Works outside a
+request entirely too (a background job, a console session): the buffer it adds to is created
+lazily on whatever thread calls it, the same "works standalone, no specific setup required" shape
+`ForgeOpsTracker.set_user` already has, rather than silently doing nothing with no
+`ForgeOpsTracker::Middleware::BreadcrumbContext` around it.
+
+Each request gets its own fresh, bounded trail (a ring buffer capped at `max_breadcrumbs`, oldest
+entry dropped once full): scoped to the current thread the same way `ForgeOpsTracker.set_user`'s
+own thread-local is, so one request's trail never bleeds into another's on a reused Puma thread.
+Unlike the affected user above, a breadcrumb's `message`/`data` **is** scrubbed for likely PII:
+console-style/query/request trail entries are exactly the kind of free text (a bind parameter
+showing up in a message, a URL with a token in it) the scrubber exists to catch, not a
+deliberately-structured field the way `user` is.
+
 ## PII scrubbing
 
 By default, the message, backtrace, and any context/tags you attach are scanned for likely
 personal data: email addresses, formatted SSNs/credit cards, known API key/token formats, and
 anything under a suspiciously-named key (`password`, `api_key`, `ssn`, and similar); redacted
 before the payload ever leaves this process. ForgeOps itself scrubs again on arrival regardless, so
-this is a second, earlier layer, not the only one.
+this is a second, earlier layer, not the only one. The user identified via automatic Warden
+detection or `ForgeOpsTracker.set_user` above is a deliberate exception: it's never scrubbed,
+since redacting it would defeat the whole point of identifying users in the first place.
 
 To disable it (e.g. if your app already scrubs its own error context, or you have your own reasons
 to want the raw payload):
@@ -124,3 +186,154 @@ end
 
 Requires a ForgeOps plan that includes release health; on a plan that doesn't, the periodic
 flushes are simply rejected server-side and dropped, exactly like any other delivery failure.
+
+## Performance monitoring
+
+By default, every request's controller/action duration is timed (via Rails' own
+`process_action.action_controller` instrumentation, no extra middleware needed) so a dashboard
+widget on ForgeOps can show which parts of your app are actually slow, not just which ones raise.
+Bucketed by transaction ("PostsController#show") and flushed as a small periodic aggregate per
+transaction on a background thread, the same delivery philosophy as session tracking above: a
+broken or unreachable tracker never affects the host app either way.
+
+Each aggregate carries a small latency histogram alongside its count/sum/max, so ForgeOps can show
+an approximate p50/p95/p99 per transaction, not just an average: accurate to the width of whichever
+latency bucket a duration falls into (50/100/250/500/1000/2500/5000/10000ms), the standard
+histogram-quantile trade-off (the same one Prometheus's own `histogram_quantile` makes) rather than
+storing every individual request's own duration.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_performance = false   # opt out entirely
+  config.performance_flush_interval = 30 # seconds; default 60
+end
+```
+
+Requires a ForgeOps plan that includes performance monitoring; on a plan that doesn't, the
+periodic flushes are simply rejected server-side and dropped, exactly like any other delivery
+failure.
+
+### Database queries, background jobs, and outbound HTTP calls
+
+The same automatic instrumentation, on the same `track_performance` flag, also covers:
+
+- **Database queries**, via `sql.active_record`: bucketed by Rails' own auto-generated query name
+  ("User Load", "Order Create"), not the raw SQL text. Internal schema-introspection queries
+  ("SCHEMA") and cached reads (never a real round trip to the database) are skipped.
+- **Background jobs**, via `perform.active_job`: bucketed by job class. Covers any Active Job
+  backend, Solid Queue included, with no backend-specific code needed. A second, independent
+  sample, `kind: "job_latency"`, bucketed by queue name instead of job class, also records how long
+  the job actually waited in its queue before this run started: `nil`, and skipped, for a job run
+  via `perform_now`, which never goes through `enqueue` at all and so has no queue wait to report.
+- **Raw Sidekiq workers** (a `Sidekiq::Worker`/`Sidekiq::Job` not dispatched through Active Job):
+  a server middleware, registered automatically when Sidekiq is already loaded. A job dispatched
+  through Active Job and run on Sidekiq is still only counted once, by the Active Job hook above,
+  not twice.
+- **Outbound `Net::HTTP` calls**, bucketed by `"<method> <host>"`, never the full URL (a path or
+  query string could carry an id or a token). Most other Ruby HTTP client libraries (Faraday's own
+  `net_http` adapter, HTTParty, RestClient) ultimately call through `Net::HTTP`, so this covers
+  those too as a side effect.
+
+Every one of these shows up as its own `kind` ("controller", "job", "job_latency", "query", "http")
+on the same `performance` dashboard dataset, so "slowest jobs" and "slowest queries" are just a
+filtered version of the same widget builder "slowest transactions" already uses.
+
+### Sidekiq, Solid Queue, Puma, and ActiveRecord gauges
+
+Also on by default whenever the corresponding library is already loaded: Sidekiq's own aggregate
+stats (`Sidekiq::Stats`: processed/failed/scheduled/retry/dead counts, plus a queue-depth
+reading per queue), Solid Queue's own state (active workers, active dispatchers, failed/scheduled/
+blocked counts, a queue-depth reading per queue), Puma's own thread pool (backlog, running threads,
+pool capacity), and this process's own ActiveRecord connection pool (size, busy count, waiting
+count, and a derived utilization percent). Reported as ordinary infrastructure metrics
+(`sidekiq.queue_depth.default`, `puma.backlog`, `active_record_pool.utilization_pct`, and so on) via
+the same `capture_infrastructure_metric` call your own scripts use, on a periodic timer:
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.gauge_poll_interval = 30 # seconds; default 60
+end
+```
+
+No separate opt-out: these are gated by the same `track_performance` flag as everything else in
+this section, not a flag of their own.
+
+## Failure tracking
+
+On by default, from the same two instrumentation points performance monitoring already uses:
+
+- **Background jobs**: a job whose `perform` raised, via the existing `perform.active_job`
+  subscription. Not just an aggregate failure count; each occurrence carries the job class, the
+  raised exception's class name, and its message.
+- **Outbound `Net::HTTP` calls**: a request that raised, or came back with a 5xx status (a 4xx is
+  a response the dependency actually gave, not a failure of the dependency itself).
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_failures = false        # opt out entirely
+  config.failure_event_flush_interval = 30 # seconds; default 60
+end
+```
+
+**Known, honest gap**: `Net::HTTP.get_response` and the `Net::HTTP.start { |http| ... }` block
+form both open the TCP connection before the block (and so before `#request`, the only method
+this gem wraps) ever runs, so a connection-refused or DNS-failure error raised during that setup
+is invisible here, for the same reason it's already invisible to this gem's own outbound-HTTP
+breadcrumbs and timing. `Net::HTTP.new(host, port).request(...)` (no explicit `#start`) is
+unaffected: confirmed directly that Net::HTTP connects lazily, inside `#request` itself, the first
+time it's called on a not-yet-started instance.
+
+Requires a ForgeOps plan that includes performance monitoring, the same plan feature performance
+monitoring itself already requires; on a plan that doesn't, the periodic flushes are simply
+rejected server-side and dropped, exactly like any other delivery failure.
+
+## Custom metrics
+
+Unlike session/performance tracking above, there's no automatic instrumentation here at all: a
+signup or a payment isn't something this gem could ever detect on its own, so this is an explicit
+call your own code makes. `value` defaults to `1.0` so a bare counter-style call needs no
+argument; pass one for a metric with a real amount. Buffered and flushed as a batch on a
+background thread, same delivery philosophy as everything else here, so this is safe to call from
+inside a request (right after a signup completes, say) without adding network latency there.
+
+```ruby
+ForgeOpsTracker.capture_metric("signups")
+ForgeOpsTracker.capture_metric("revenue", value: 49.00)
+
+ForgeOpsTracker.configure do |config|
+  config.metric_flush_interval = 30 # seconds; default 60
+end
+```
+
+Requires a ForgeOps plan that includes custom metrics; on a plan that doesn't, the periodic
+flushes are simply rejected server-side and dropped, exactly like any other delivery failure.
+
+## Infrastructure monitoring
+
+A deliberately different shape from everything above: there's no automatic instrumentation and no
+ForgeOps-built agent. Run your own short-lived script on a cron entry or a systemd timer, reading
+your own host's own stats; `hostname` defaults to the box the script is actually running on.
+Buffered and flushed on exit, so a handful of capture calls in one short-lived process still cost
+one network request, not several.
+
+```ruby
+# A cron entry or systemd timer runs this periodically, not your web app itself.
+require "forge_ops_tracker"
+ForgeOpsTracker.configure { |config| config.dsn = ENV["FORGE_OPS_DSN"] }
+
+load_average = File.read("/proc/loadavg").split.first.to_f
+ForgeOpsTracker.capture_infrastructure_metric("load_average", value: load_average)
+
+meminfo = File.read("/proc/meminfo").lines.to_h { |line| line.split(":").map(&:strip) }
+total_kb, available_kb = meminfo["MemTotal"].to_i, meminfo["MemAvailable"].to_i
+ForgeOpsTracker.capture_infrastructure_metric("memory_used_percent",
+  value: 100.0 * (total_kb - available_kb) / total_kb)
+
+disk_used_percent = `df --output=pcent / | tail -1`.strip.delete("%").to_f
+ForgeOpsTracker.capture_infrastructure_metric("disk_used_percent", value: disk_used_percent)
+```
+
+The example above is Linux-specific (`/proc/loadavg`, `/proc/meminfo`); on another OS, read that
+platform's own equivalents instead. Requires a ForgeOps plan that includes infrastructure
+monitoring; on a plan that doesn't, the periodic flushes are simply rejected server-side and
+dropped, exactly like any other delivery failure.

data/lib/forge_ops_tracker/breadcrumb_buffer.rb

@@ -0,0 +1,45 @@
+module ForgeOpsTracker
+  # A bounded, in-order trail of whatever happened recently on this thread: SQL queries,
+  # controller actions, outgoing HTTP requests, and anything added by hand via
+  # ForgeOpsTracker.add_breadcrumb, all recorded automatically once ForgeOpsTracker::Railtie's own
+  # subscriptions are installed. The direct Ruby analog to sdks/typescript's own BreadcrumbBuffer
+  # (a ring buffer capped at Configuration#max_breadcrumbs, oldest entry dropped once full), but
+  # per-thread rather than per-page-lifetime: a browser tab has one continuous session to trail,
+  # but a Rails server handles many concurrent, unrelated requests on a thread pool, the same
+  # "Thread.current, not a plain global" reasoning ForgeOpsTracker.set_user's own thread-local
+  # already documents. ForgeOpsTracker::Middleware::BreadcrumbContext resets this to a fresh, empty
+  # buffer at the start of every request, so one request's trail never bleeds into another's on a
+  # reused Puma thread, and clears it again in an ensure so it doesn't leak into whatever runs next
+  # on that thread outside a request either (a console session, a background job).
+  class BreadcrumbBuffer
+    def initialize(configuration)
+      @configuration = configuration
+      @entries = []
+    end
+
+    def add(category:, message:, level: "info", data: {})
+      # Read fresh on every add, not captured once at construction: the same reasoning
+      # DeliveryQueue re-reads Configuration#queue_size on every push, so a config change from
+      # ForgeOpsTracker.configure takes effect on whatever's added next, not just a buffer created
+      # afterward.
+      max_size = [ configuration.max_breadcrumbs, 0 ].max
+      return if max_size.zero?
+
+      entries << {
+        category: category.to_s,
+        message: message.to_s,
+        level: level.to_s,
+        timestamp: Time.now.utc.iso8601,
+        data: data || {}
+      }
+      entries.shift while entries.length > max_size
+    end
+
+    def all
+      entries.dup
+    end
+
+    private
+      attr_reader :configuration, :entries
+  end
+end

data/lib/forge_ops_tracker/client.rb

@@ -23,6 +23,43 @@ module ForgeOpsTracker
       post(configuration.session_checkins_uri, payload)
     end
 
+    # Same delivery contract again; see Configuration#performance_samples_uri. payload here is a
+    # batch (one entry per distinct transaction a flush interval saw), not a single aggregate the
+    # way deliver_session_checkin's own payload is, so this posts { samples: [...] } rather than
+    # the array bare, matching what Api::V1::PerformanceSamplesController expects.
+    def deliver_performance_samples(samples)
+      post(configuration.performance_samples_uri, { samples: samples })
+    end
+
+    # Same delivery contract again; see Configuration#custom_metrics_uri. payload is a batch of
+    # individual capture_metric calls (see MetricBuffer), matching what
+    # Api::V1::CustomMetricsController expects.
+    def deliver_metrics(entries)
+      post(configuration.custom_metrics_uri, { metrics: entries })
+    end
+
+    # Same again; see Configuration#infrastructure_metrics_uri and InfrastructureMetricBuffer.
+    def deliver_infrastructure_metrics(entries)
+      post(configuration.infrastructure_metrics_uri, { metrics: entries })
+    end
+
+    # Same again; see Configuration#failure_events_uri and FailureEventBuffer. payload is a batch
+    # of individual real failures (matching Api::V1::FailureEventsController's own { failures:
+    # [...] } expectation), the same "a list of individually-meaningful entries, not an aggregate"
+    # shape deliver_metrics above already has, not deliver_performance_samples' own bucketed one.
+    def deliver_failure_events(entries)
+      post(configuration.failure_events_uri, { failures: entries })
+    end
+
+    # Same delivery contract again; see Configuration#spans_uri and SpanQueue. payload is one
+    # whole captured trace (a trace_id plus every span belonging to it), matching what
+    # Api::V1::SpansController expects - unlike every other deliver_* method above, this is never
+    # a batch of several distinct traces at once; SpanQueue pushes (and this posts) one trace per
+    # call, the moment it's ready, rather than accumulating several over a flush interval.
+    def deliver_spans(trace_id:, spans:)
+      post(configuration.spans_uri, { trace_id: trace_id, spans: spans })
+    end
+
     private
       attr_reader :configuration
 

data/lib/forge_ops_tracker/configuration.rb

@@ -9,6 +9,13 @@ module ForgeOpsTracker
     attr_accessor :enabled_environments, :queue_size, :open_timeout, :read_timeout, :scrub_pii
     attr_accessor :capture_source_context
     attr_accessor :track_sessions, :session_flush_interval
+    attr_accessor :track_performance, :performance_flush_interval
+    attr_accessor :track_current_user
+    attr_accessor :track_breadcrumbs, :max_breadcrumbs
+    attr_accessor :track_failures, :failure_event_flush_interval
+    attr_accessor :metric_flush_interval, :infrastructure_metric_flush_interval
+    attr_accessor :gauge_poll_interval
+    attr_accessor :track_tracing, :trace_capture_threshold_ms
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -43,6 +50,70 @@ module ForgeOpsTracker
       # ForgeOpsTracker::Middleware::SessionTracking for what this actually wraps.
       @track_sessions = true
       @session_flush_interval = 60
+      # Auto-instruments every request the same "on unless you turn it off" default as
+      # track_sessions above; see Railtie's own comment for exactly what this subscribes to.
+      @track_performance = true
+      @performance_flush_interval = 60
+      # Auto-detects the current user via Warden (env["warden"].user; Devise mounts Warden
+      # automatically, so this covers Devise apps too, but works for any Warden-based auth, not
+      # just Devise specifically) the same "on unless you turn it off" default every other
+      # automatic instrumentation flag above already has. See
+      # ForgeOpsTracker::Middleware::UserContext for how, and ForgeOpsTracker.set_user for the
+      # manual override/fallback when there's no Warden at all or its guess isn't right.
+      @track_current_user = true
+      # Auto-instruments SQL queries, controller actions, and outgoing Net::HTTP requests as
+      # breadcrumbs the same "on unless you turn it off" default every other automatic
+      # instrumentation flag above already has. See ForgeOpsTracker::Middleware::BreadcrumbContext
+      # for the per-request buffer this gates, and ForgeOpsTracker.add_breadcrumb for adding one by
+      # hand regardless of this flag (the manual API isn't gated by it: an app that wants only its
+      # own hand-added breadcrumbs, with none of the automatic ones, turns this off and still gets
+      # add_breadcrumb).
+      @track_breadcrumbs = true
+      # Oldest entry dropped once this many have accumulated in a single request: the same
+      # "bounded ring buffer, not an unbounded log" reasoning sdks/typescript's own
+      # Configuration#maxBreadcrumbs already documents, so a request that runs a very large number
+      # of queries doesn't grow the trail (and the payload it rides in) without bound.
+      @max_breadcrumbs = 30
+      # Recorded from the same two instrumentation points that already exist for other reasons
+      # (the perform.active_job subscription, Net::HTTP's own Timing#request): a job whose perform
+      # raised, or an outbound call that raised or came back 5xx. On by default, independently of
+      # track_performance/track_breadcrumbs, the same "several genuinely independent mechanisms
+      # recorded from the same instrumentation point" pattern breadcrumbs already established at
+      # both of those exact call sites.
+      @track_failures = true
+      @failure_event_flush_interval = 60
+      # No track_metrics/track_infrastructure boolean the way track_sessions/track_performance
+      # each have one: those gate automatic instrumentation that's on unless you turn it off;
+      # capture_metric/capture_infrastructure_metric are explicit calls the customer's own code
+      # chooses to make at all, so there's no "automatic behavior" for a flag to disable. Only
+      # the flush interval needs a knob.
+      @metric_flush_interval = 60
+      @infrastructure_metric_flush_interval = 60
+      # How often the Sidekiq/Solid Queue/Puma stats reporters poll and report their own gauge
+      # readings (queue depth, worker counts, Puma backlog/threads/pool capacity). Gated by
+      # track_performance, same as controller/job/query/http timing: one flag for "automatically
+      # instrument this app's own operational data," reused rather than adding a separate boolean
+      # per new automatic instrumentation source this gem grows.
+      @gauge_poll_interval = 60
+      # Auto-instruments controller actions, SQL queries, outgoing Net::HTTP requests, and Redis
+      # calls into a nested per-request span tree, the same "on unless you turn it off" default
+      # every other automatic instrumentation flag above already has. See
+      # ForgeOpsTracker::Middleware::SpanTracing for the per-request buffer this gates, and
+      # ForgeOpsTracker.span for wrapping a customer's own service-level code by hand so it shows
+      # up as a real nested layer (e.g. "PaymentService#charge") rather than its database/HTTP
+      # calls appearing to hang directly off the controller root. Whether this ever actually
+      # reaches the server is a completely separate question from whether it's on: distributed
+      # tracing is a plan-gated feature (see Api::V1::SpansController), enforced server-side the
+      # same way every other ingestion endpoint already is, not by this client-side flag.
+      @track_tracing = true
+      # Only a request whose own total duration is at least this slow ever gets sent at all: see
+      # SpanBuffer#slow?/Middleware::SpanTracing's own comment for why that decision happens
+      # entirely client-side, before a single byte goes over the wire, rather than sending every
+      # trace and letting the server decide. 1 second is a deliberately conservative default (most
+      # web apps consider anything near that already a bad user experience), not tied to any
+      # PerformanceSample-derived threshold elsewhere in this gem: those describe a trend across
+      # many requests, this describes a single one being outright slow.
+      @trace_capture_threshold_ms = 1_000
     end
 
     def api_key
@@ -72,6 +143,52 @@ module ForgeOpsTracker
       uri
     end
 
+    # Same substitution as session_checkins_uri above, its own sibling path under the same DSN.
+    def performance_samples_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/performance_samples")
+      uri
+    end
+
+    def custom_metrics_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/custom_metrics")
+      uri
+    end
+
+    def infrastructure_metrics_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/infrastructure_metrics")
+      uri
+    end
+
+    def failure_events_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/failure_events")
+      uri
+    end
+
+    def spans_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/spans")
+      uri
+    end
+
     def enabled?
       !blank?(dsn) && !blank?(api_key) && enabled_environments.map(&:to_s).include?(environment.to_s)
     end

data/lib/forge_ops_tracker/error_subscriber.rb

@@ -5,6 +5,16 @@ module ForgeOpsTracker
   # wrapped to guarantee this never propagates an exception back into the
   # host app's error-handling cycle.
   class ErrorSubscriber
+    # Sidekiq raises its own control-flow exception once it has already scheduled a failed job's
+    # retry (Sidekiq::JobRetry::Handled, and Sidekiq::JobRetry::Skip, a subclass, for a job that
+    # opted out of retries), and Rails' executor, which Sidekiq's Rails integration wraps every
+    # job in, reports whatever escapes that wrap. Left alone, every failing Sidekiq job, whatever
+    # actually went wrong inside it, reaches ForgeOps as the same opaque "Sidekiq::JobRetry::
+    # Handled" issue with the real error nowhere in it. The real error is that exception's #cause
+    # (Sidekiq raises it from inside its own rescue), so it's unwrapped here, by class name
+    # rather than constant since Sidekiq is an optional dependency this gem never requires.
+    RETRY_CONTROL_FLOW_CLASS = "Sidekiq::JobRetry::Handled".freeze
+
     def initialize(configuration, delivery_queue: DeliveryQueue.new(configuration), event_builder: EventBuilder.new(configuration))
       @configuration = configuration
       @delivery_queue = delivery_queue
@@ -14,7 +24,15 @@ module ForgeOpsTracker
     def report(error, handled: true, severity: nil, context: {}, source: nil)
       return unless configuration.enabled?
 
-      delivery_queue.push(event_builder.build(error, context: context))
+      # Read here, not inside EventBuilder: Rails.error.subscribe's own #report interface never
+      # hands this class the Rack env, so ForgeOpsTracker::Middleware::UserContext (Warden
+      # auto-detection) and ForgeOpsTracker.set_user (the manual override/fallback) both
+      # communicate with this exact same thread-local instead, the only channel available here.
+      # ForgeOpsTracker::Middleware::BreadcrumbContext's own per-request buffer works the same way.
+      error = unwrap_retry_control_flow(error)
+      user = Thread.current[:forge_ops_tracker_current_user]
+      breadcrumbs = Thread.current[:forge_ops_tracker_breadcrumbs]&.all || []
+      delivery_queue.push(event_builder.build(error, context: context, user: user, breadcrumbs: breadcrumbs))
       nil
     rescue StandardError => e
       configuration.logger&.debug { "[ForgeOpsTracker] report failed: #{e.class}: #{e.message}" }
@@ -23,5 +41,14 @@ module ForgeOpsTracker
 
     private
       attr_reader :configuration, :delivery_queue, :event_builder
+
+      # Only ever unwraps when there is a real #cause to unwrap to; a Handled with no cause (not
+      # something Sidekiq itself does) is reported as-is rather than dropped.
+      def unwrap_retry_control_flow(error)
+        while error.cause && error.class.ancestors.any? { |ancestor| ancestor.name == RETRY_CONTROL_FLOW_CLASS }
+          error = error.cause
+        end
+        error
+      end
   end
 end

data/lib/forge_ops_tracker/event_builder.rb

@@ -27,7 +27,7 @@ module ForgeOpsTracker
       @configuration = configuration
     end
 
-    def build(error, context: {})
+    def build(error, context: {}, user: nil, breadcrumbs: [])
       payload = {
         exception_class: error.class.name,
         message: error.message.to_s,
@@ -40,25 +40,34 @@ module ForgeOpsTracker
         tags: {},
         sdk_name: SDK_NAME
       }
+      payload[:user] = user if user && !user.empty?
+      payload[:breadcrumbs] = breadcrumbs if breadcrumbs && !breadcrumbs.empty?
       scrub(payload)
     end
 
     private
       attr_reader :configuration
 
-      # exception_class/occurred_at/environment/release/server_name/sdk_name are
-      # left alone; structured fields this gem or the host app sets
-      # deliberately, not free text an exception or its context could
-      # accidentally spill sensitive data into.
+      # exception_class/occurred_at/environment/release/server_name/sdk_name/user are left alone;
+      # structured fields this gem or the host app sets deliberately, not free text an exception
+      # or its context could accidentally spill sensitive data into. user specifically is a
+      # deliberate exemption, not an oversight: the server's own PiiScrubber-equivalent would
+      # otherwise redact the exact email address this field exists to carry. breadcrumbs is *not*
+      # exempt, unlike user: console-style/query/request trail entries are exactly the kind of free
+      # text (a query's bind params showing up in a message, a URL with a token in it) the scrubber
+      # exists to catch, matching the server's own api/v1/events_controller.rb treatment of this
+      # field.
       def scrub(payload)
         return payload unless configuration.scrub_pii
 
-        payload.merge(
+        scrubbed = payload.merge(
           message: PiiScrubber.scrub(payload[:message]),
           backtrace: PiiScrubber.scrub(payload[:backtrace]),
           context: PiiScrubber.scrub(payload[:context]),
           tags: PiiScrubber.scrub(payload[:tags])
         )
+        scrubbed[:breadcrumbs] = PiiScrubber.scrub(payload[:breadcrumbs]) if payload.key?(:breadcrumbs)
+        scrubbed
       end
 
       def backtrace_frames(error)