forge_ops_tracker 0.7.0 → 0.10.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 72692319cfdf068129d7d005773e21a3b6b78cb9f7816cf0e7c8e6409c9704f1
-  data.tar.gz: 713f244a7ab71a005e66b7f00d1dff3e1e3ffc44e304d5a36f79c911e508c74b
+  metadata.gz: 9b32a031e24fb613c219aee3bf01f0517043b05b4d1788d7ece456096d7561a0
+  data.tar.gz: 39c7f16920425d01862607b6d3f85b957c694ab009ce8ffd010f33714632a3a9
 SHA512:
-  metadata.gz: 57f5372fc1aa6ac8f8830804003c00d68d0633d4f330dc8f6c678faa1c1dbc3f09ad1da0f29c180c63a90dc41f2d0f6acb008e4a726d59ba6b806298af3d6d8f
-  data.tar.gz: 2b21f12200f99202c5f26cc739104021d5eaf1cfe16a01b075fe4bc3418ada28cf7befb91470fb1ebd8bb18865cd52e61a705decc5f511c2c497a79c74c36016
+  metadata.gz: 315fb4765179a298da43dd2beabe68970f257dee7b4b1f3938bcc6efa72ac403b8dbb0e39b8d16dfcaf19b76696aed961270dc88fd92731b05ae1610b3d4a223
+  data.tar.gz: f2907bae8015b4093346a30369cc3da38c8d36f4198ae4935cb1e2d2cea2e6baed44d5fcab4a8278029f228353d991275316fd54a9e08bd3116519b529f09159

data/CHANGELOG.md

@@ -1,5 +1,60 @@
 # Changelog
 
+## 0.10.2
+
+- Documentation only: the README and package description now describe ForgeOps as a hosted service, link to getforgeops.net, and show the real host in the connection string example instead of a placeholder. No code changes.
+
+## 0.10.1
+
+- Sidekiq job failures now report the real error. Sidekiq raises its own control-flow exception
+  (`Sidekiq::JobRetry::Handled`, or `Skip` for a job with retries turned off) once it has scheduled
+  a failed job's retry, and that wrapper, not the error inside it, was what reached ForgeOps, so
+  every failing Sidekiq job, whatever went wrong, landed in the same opaque issue. The subscriber
+  now unwraps it to its `#cause`. Confirmed against a real Sidekiq worker on 7.3 and 8.1. Among
+  other things, this is what lets ForgeOps recognize a job class that was renamed or removed while
+  jobs for it were still queued (an ActiveJob `UnknownJobClassError`, or a plain Sidekiq job's
+  `NameError`), which previously arrived only as the wrapper.
+
+## 0.10.0
+
+- Real per-occurrence job/dependency failure tracking, feeding ForgeOps's own new cross-type
+  project timeline: a job whose `perform` raised (detected via the existing `perform.active_job`
+  subscription reading ActiveSupport::Notifications' own automatic `exception_object` payload key,
+  no new hook needed) or an outbound `Net::HTTP` call that raised or came back 5xx (not 4xx) is
+  now recorded individually, not just as an aggregate failure count. New `config.track_failures`
+  (default true, independent of `track_performance`/`track_breadcrumbs`) and
+  `config.failure_event_flush_interval` (default 60s). Known, documented gap: a connection-refused
+  or DNS-failure error raised during `Net::HTTP.start`/`Net::HTTP.get_response`'s own connection
+  setup (before `#request` itself ever runs) is invisible to this detection, the same pre-existing
+  limitation that already silently applied to this SDK's outbound-HTTP breadcrumbs and timing;
+  `Net::HTTP.new(...).request(...)` (no explicit `#start`) is unaffected, since Net::HTTP connects
+  lazily inside `#request` itself in that case.
+
+## 0.9.0
+
+- Performance percentiles: every `PerformanceSample` (controller/query/http timing) now carries a
+  small latency histogram alongside its existing count/sum/max, so ForgeOps can compute approximate
+  p50/p95/p99 per transaction instead of only a weighted average. No new config; this rides the
+  existing `track_performance` flag and flush interval.
+- Background job queue latency: the existing `perform.active_job` subscription now also records how
+  long a job actually waited in its queue before starting (not how long it took to run), as a new
+  `job_latency` performance-sample kind grouped by queue name. Nil for a job run via `perform_now`,
+  which never goes through `enqueue` at all.
+- Database connection pool utilization: a new automatic reporter (matching the existing Sidekiq/
+  Solid Queue/Puma stats reporters' own shape) periodically captures this process's own ActiveRecord
+  connection pool size/busy/waiting counts as infrastructure metrics, polled on the same
+  `config.gauge_poll_interval` those reporters already use.
+
+## 0.8.0
+
+- Breadcrumbs: a trail of SQL queries, controller actions, and outbound `Net::HTTP` calls leading
+  up to an error, recorded automatically (on by default, same as every other automatic
+  instrumentation this gem does; opt out with `config.track_breadcrumbs = false`, or tune
+  `config.max_breadcrumbs`, default 30). `ForgeOpsTracker.add_breadcrumb(message, category:,
+  level:, data:)` adds your own, regardless of whether the automatic sources are on, and works
+  outside a request entirely too (a background job, a console session). Unlike the affected user,
+  a breadcrumb's message/data is scrubbed for likely PII.
+
 ## 0.7.0
 
 - Automatically associates a reported error with the current signed-in user, for any Warden-based

data/README.md

@@ -1,6 +1,6 @@
 # ForgeOpsTracker
 
-Rails exception reporting client for a [ForgeOps](../../) instance.
+Rails exception reporting client for [ForgeOps](https://getforgeops.net).
 
 ## Installation
 
@@ -15,7 +15,7 @@ Set a DSN (from a project's settings page in ForgeOps) via an initializer or env
 ```ruby
 # config/initializers/forge_ops_tracker.rb
 ForgeOpsTracker.configure do |config|
-  config.dsn = ENV["FORGE_OPS_DSN"] # "https://<api_key>@forgeops.example.com/api/v1/events"
+  config.dsn = ENV["FORGE_OPS_DSN"] # "https://<api_key>@getforgeops.net/api/v1/events"
   config.release = ENV["HEROKU_SLUG_COMMIT"] || `git rev-parse HEAD`.strip
   config.enabled_environments = %w[production staging] # default; reporting is a no-op elsewhere
 end
@@ -67,6 +67,25 @@ Bottom line: if an exception would otherwise crash something, you're already cov
 code already catches and handles it, route that specific `rescue` through `Rails.error.handle`/
 `.record` instead of a bare one wherever you want ForgeOps to know about it.
 
+### Sidekiq job failures and renamed job classes
+
+A failing Sidekiq job is reported through `Rails.error` like any other error. Since 0.10.1 the issue
+carries the real error, not Sidekiq's internal `Sidekiq::JobRetry::Handled` retry wrapper (which is
+what every failing job used to arrive as, whatever went wrong inside it).
+
+Renaming or removing a job class breaks every job still queued under the old name, since Sidekiq
+stores the class name as a string, and no test can catch it because those jobs live in Redis. ForgeOps
+recognizes that failure (an Active Job `UnknownJobClassError`, or a plain Sidekiq job's `NameError`
+from constantizing the class), names the class that no longer exists, and shows how soon after which
+release it first appeared. Keep the old name as an alias until the queue, retry set and scheduled set
+have drained:
+
+```ruby
+# app/jobs/send_invoice_job.rb
+# Kept only until jobs queued under the old name have drained, then delete this file.
+SendInvoiceJob = DeliverInvoiceJob
+```
+
 ## Identifying users
 
 If you're using Devise, or any other Warden-based authentication, the currently signed-in user is
@@ -93,6 +112,40 @@ none of them (or with a signed-out `current_user`, as the `&.` above already han
 whatever was set. Shows up on an issue's own detail page, and as its own `affected_users_count`
 alongside the regular event count.
 
+## Breadcrumbs
+
+A trail of what happened right before an error, on by default, no setup needed: every SQL query,
+controller action, and outbound `Net::HTTP` call during a request is recorded automatically, and
+shows up alongside the error on an issue's own detail page.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_breadcrumbs = false # opt out of the automatic sources entirely
+  config.max_breadcrumbs = 30 # oldest entry dropped once this many have accumulated in one request
+end
+```
+
+Add your own by hand, regardless of whether the automatic sources are on:
+
+```ruby
+ForgeOpsTracker.add_breadcrumb("charged card", category: "billing", data: { order_id: order.id })
+```
+
+`category` defaults to `"custom"`, `level` to `"info"` (`"debug"`/`"info"`/`"warning"`/`"error"`
+are the four levels the automatic sources themselves use too), and `data` to `{}`. Works outside a
+request entirely too (a background job, a console session): the buffer it adds to is created
+lazily on whatever thread calls it, the same "works standalone, no specific setup required" shape
+`ForgeOpsTracker.set_user` already has, rather than silently doing nothing with no
+`ForgeOpsTracker::Middleware::BreadcrumbContext` around it.
+
+Each request gets its own fresh, bounded trail (a ring buffer capped at `max_breadcrumbs`, oldest
+entry dropped once full): scoped to the current thread the same way `ForgeOpsTracker.set_user`'s
+own thread-local is, so one request's trail never bleeds into another's on a reused Puma thread.
+Unlike the affected user above, a breadcrumb's `message`/`data` **is** scrubbed for likely PII:
+console-style/query/request trail entries are exactly the kind of free text (a bind parameter
+showing up in a message, a URL with a token in it) the scrubber exists to catch, not a
+deliberately-structured field the way `user` is.
+
 ## PII scrubbing
 
 By default, the message, backtrace, and any context/tags you attach are scanned for likely
@@ -162,6 +215,12 @@ Bucketed by transaction ("PostsController#show") and flushed as a small periodic
 transaction on a background thread, the same delivery philosophy as session tracking above: a
 broken or unreachable tracker never affects the host app either way.
 
+Each aggregate carries a small latency histogram alongside its count/sum/max, so ForgeOps can show
+an approximate p50/p95/p99 per transaction, not just an average: accurate to the width of whichever
+latency bucket a duration falls into (50/100/250/500/1000/2500/5000/10000ms), the standard
+histogram-quantile trade-off (the same one Prometheus's own `histogram_quantile` makes) rather than
+storing every individual request's own duration.
+
 ```ruby
 ForgeOpsTracker.configure do |config|
   config.track_performance = false   # opt out entirely
@@ -181,7 +240,10 @@ The same automatic instrumentation, on the same `track_performance` flag, also c
   ("User Load", "Order Create"), not the raw SQL text. Internal schema-introspection queries
   ("SCHEMA") and cached reads (never a real round trip to the database) are skipped.
 - **Background jobs**, via `perform.active_job`: bucketed by job class. Covers any Active Job
-  backend, Solid Queue included, with no backend-specific code needed.
+  backend, Solid Queue included, with no backend-specific code needed. A second, independent
+  sample, `kind: "job_latency"`, bucketed by queue name instead of job class, also records how long
+  the job actually waited in its queue before this run started: `nil`, and skipped, for a job run
+  via `perform_now`, which never goes through `enqueue` at all and so has no queue wait to report.
 - **Raw Sidekiq workers** (a `Sidekiq::Worker`/`Sidekiq::Job` not dispatched through Active Job):
   a server middleware, registered automatically when Sidekiq is already loaded. A job dispatched
   through Active Job and run on Sidekiq is still only counted once, by the Active Job hook above,
@@ -191,19 +253,20 @@ The same automatic instrumentation, on the same `track_performance` flag, also c
   `net_http` adapter, HTTParty, RestClient) ultimately call through `Net::HTTP`, so this covers
   those too as a side effect.
 
-Every one of these shows up as its own `kind` ("controller", "job", "query", "http") on the same
-`performance` dashboard dataset, so "slowest jobs" and "slowest queries" are just a filtered
-version of the same widget builder "slowest transactions" already uses.
+Every one of these shows up as its own `kind` ("controller", "job", "job_latency", "query", "http")
+on the same `performance` dashboard dataset, so "slowest jobs" and "slowest queries" are just a
+filtered version of the same widget builder "slowest transactions" already uses.
 
-### Sidekiq, Solid Queue, and Puma gauges
+### Sidekiq, Solid Queue, Puma, and ActiveRecord gauges
 
 Also on by default whenever the corresponding library is already loaded: Sidekiq's own aggregate
-stats (`Sidekiq::Stats`: processed/failed/scheduled/retry/dead counts, plus a queue-depth reading
-per queue), Solid Queue's own state (active workers, active dispatchers, failed/scheduled/blocked
-counts, a queue-depth reading per queue), and Puma's own thread pool (backlog, running threads,
-pool capacity). Reported as ordinary infrastructure metrics (`sidekiq.queue_depth.default`,
-`puma.backlog`, and so on) via the same `capture_infrastructure_metric` call your own scripts use,
-on a periodic timer:
+stats (`Sidekiq::Stats`: processed/failed/scheduled/retry/dead counts, plus a queue-depth
+reading per queue), Solid Queue's own state (active workers, active dispatchers, failed/scheduled/
+blocked counts, a queue-depth reading per queue), Puma's own thread pool (backlog, running threads,
+pool capacity), and this process's own ActiveRecord connection pool (size, busy count, waiting
+count, and a derived utilization percent). Reported as ordinary infrastructure metrics
+(`sidekiq.queue_depth.default`, `puma.backlog`, `active_record_pool.utilization_pct`, and so on) via
+the same `capture_infrastructure_metric` call your own scripts use, on a periodic timer:
 
 ```ruby
 ForgeOpsTracker.configure do |config|
@@ -214,6 +277,35 @@ end
 No separate opt-out: these are gated by the same `track_performance` flag as everything else in
 this section, not a flag of their own.
 
+## Failure tracking
+
+On by default, from the same two instrumentation points performance monitoring already uses:
+
+- **Background jobs**: a job whose `perform` raised, via the existing `perform.active_job`
+  subscription. Not just an aggregate failure count; each occurrence carries the job class, the
+  raised exception's class name, and its message.
+- **Outbound `Net::HTTP` calls**: a request that raised, or came back with a 5xx status (a 4xx is
+  a response the dependency actually gave, not a failure of the dependency itself).
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_failures = false        # opt out entirely
+  config.failure_event_flush_interval = 30 # seconds; default 60
+end
+```
+
+**Known, honest gap**: `Net::HTTP.get_response` and the `Net::HTTP.start { |http| ... }` block
+form both open the TCP connection before the block (and so before `#request`, the only method
+this gem wraps) ever runs, so a connection-refused or DNS-failure error raised during that setup
+is invisible here, for the same reason it's already invisible to this gem's own outbound-HTTP
+breadcrumbs and timing. `Net::HTTP.new(host, port).request(...)` (no explicit `#start`) is
+unaffected: confirmed directly that Net::HTTP connects lazily, inside `#request` itself, the first
+time it's called on a not-yet-started instance.
+
+Requires a ForgeOps plan that includes performance monitoring, the same plan feature performance
+monitoring itself already requires; on a plan that doesn't, the periodic flushes are simply
+rejected server-side and dropped, exactly like any other delivery failure.
+
 ## Custom metrics
 
 Unlike session/performance tracking above, there's no automatic instrumentation here at all: a

data/lib/forge_ops_tracker/breadcrumb_buffer.rb

@@ -0,0 +1,45 @@
+module ForgeOpsTracker
+  # A bounded, in-order trail of whatever happened recently on this thread: SQL queries,
+  # controller actions, outgoing HTTP requests, and anything added by hand via
+  # ForgeOpsTracker.add_breadcrumb, all recorded automatically once ForgeOpsTracker::Railtie's own
+  # subscriptions are installed. The direct Ruby analog to sdks/typescript's own BreadcrumbBuffer
+  # (a ring buffer capped at Configuration#max_breadcrumbs, oldest entry dropped once full), but
+  # per-thread rather than per-page-lifetime: a browser tab has one continuous session to trail,
+  # but a Rails server handles many concurrent, unrelated requests on a thread pool, the same
+  # "Thread.current, not a plain global" reasoning ForgeOpsTracker.set_user's own thread-local
+  # already documents. ForgeOpsTracker::Middleware::BreadcrumbContext resets this to a fresh, empty
+  # buffer at the start of every request, so one request's trail never bleeds into another's on a
+  # reused Puma thread, and clears it again in an ensure so it doesn't leak into whatever runs next
+  # on that thread outside a request either (a console session, a background job).
+  class BreadcrumbBuffer
+    def initialize(configuration)
+      @configuration = configuration
+      @entries = []
+    end
+
+    def add(category:, message:, level: "info", data: {})
+      # Read fresh on every add, not captured once at construction: the same reasoning
+      # DeliveryQueue re-reads Configuration#queue_size on every push, so a config change from
+      # ForgeOpsTracker.configure takes effect on whatever's added next, not just a buffer created
+      # afterward.
+      max_size = [ configuration.max_breadcrumbs, 0 ].max
+      return if max_size.zero?
+
+      entries << {
+        category: category.to_s,
+        message: message.to_s,
+        level: level.to_s,
+        timestamp: Time.now.utc.iso8601,
+        data: data || {}
+      }
+      entries.shift while entries.length > max_size
+    end
+
+    def all
+      entries.dup
+    end
+
+    private
+      attr_reader :configuration, :entries
+  end
+end

data/lib/forge_ops_tracker/client.rb

@@ -43,6 +43,23 @@ module ForgeOpsTracker
       post(configuration.infrastructure_metrics_uri, { metrics: entries })
     end
 
+    # Same again; see Configuration#failure_events_uri and FailureEventBuffer. payload is a batch
+    # of individual real failures (matching Api::V1::FailureEventsController's own { failures:
+    # [...] } expectation), the same "a list of individually-meaningful entries, not an aggregate"
+    # shape deliver_metrics above already has, not deliver_performance_samples' own bucketed one.
+    def deliver_failure_events(entries)
+      post(configuration.failure_events_uri, { failures: entries })
+    end
+
+    # Same delivery contract again; see Configuration#spans_uri and SpanQueue. payload is one
+    # whole captured trace (a trace_id plus every span belonging to it), matching what
+    # Api::V1::SpansController expects - unlike every other deliver_* method above, this is never
+    # a batch of several distinct traces at once; SpanQueue pushes (and this posts) one trace per
+    # call, the moment it's ready, rather than accumulating several over a flush interval.
+    def deliver_spans(trace_id:, spans:)
+      post(configuration.spans_uri, { trace_id: trace_id, spans: spans })
+    end
+
     private
       attr_reader :configuration
 

data/lib/forge_ops_tracker/configuration.rb

@@ -11,8 +11,11 @@ module ForgeOpsTracker
     attr_accessor :track_sessions, :session_flush_interval
     attr_accessor :track_performance, :performance_flush_interval
     attr_accessor :track_current_user
+    attr_accessor :track_breadcrumbs, :max_breadcrumbs
+    attr_accessor :track_failures, :failure_event_flush_interval
     attr_accessor :metric_flush_interval, :infrastructure_metric_flush_interval
     attr_accessor :gauge_poll_interval
+    attr_accessor :track_tracing, :trace_capture_threshold_ms
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -58,6 +61,27 @@ module ForgeOpsTracker
       # ForgeOpsTracker::Middleware::UserContext for how, and ForgeOpsTracker.set_user for the
       # manual override/fallback when there's no Warden at all or its guess isn't right.
       @track_current_user = true
+      # Auto-instruments SQL queries, controller actions, and outgoing Net::HTTP requests as
+      # breadcrumbs the same "on unless you turn it off" default every other automatic
+      # instrumentation flag above already has. See ForgeOpsTracker::Middleware::BreadcrumbContext
+      # for the per-request buffer this gates, and ForgeOpsTracker.add_breadcrumb for adding one by
+      # hand regardless of this flag (the manual API isn't gated by it: an app that wants only its
+      # own hand-added breadcrumbs, with none of the automatic ones, turns this off and still gets
+      # add_breadcrumb).
+      @track_breadcrumbs = true
+      # Oldest entry dropped once this many have accumulated in a single request: the same
+      # "bounded ring buffer, not an unbounded log" reasoning sdks/typescript's own
+      # Configuration#maxBreadcrumbs already documents, so a request that runs a very large number
+      # of queries doesn't grow the trail (and the payload it rides in) without bound.
+      @max_breadcrumbs = 30
+      # Recorded from the same two instrumentation points that already exist for other reasons
+      # (the perform.active_job subscription, Net::HTTP's own Timing#request): a job whose perform
+      # raised, or an outbound call that raised or came back 5xx. On by default, independently of
+      # track_performance/track_breadcrumbs, the same "several genuinely independent mechanisms
+      # recorded from the same instrumentation point" pattern breadcrumbs already established at
+      # both of those exact call sites.
+      @track_failures = true
+      @failure_event_flush_interval = 60
       # No track_metrics/track_infrastructure boolean the way track_sessions/track_performance
       # each have one: those gate automatic instrumentation that's on unless you turn it off;
       # capture_metric/capture_infrastructure_metric are explicit calls the customer's own code
@@ -71,6 +95,25 @@ module ForgeOpsTracker
       # instrument this app's own operational data," reused rather than adding a separate boolean
       # per new automatic instrumentation source this gem grows.
       @gauge_poll_interval = 60
+      # Auto-instruments controller actions, SQL queries, outgoing Net::HTTP requests, and Redis
+      # calls into a nested per-request span tree, the same "on unless you turn it off" default
+      # every other automatic instrumentation flag above already has. See
+      # ForgeOpsTracker::Middleware::SpanTracing for the per-request buffer this gates, and
+      # ForgeOpsTracker.span for wrapping a customer's own service-level code by hand so it shows
+      # up as a real nested layer (e.g. "PaymentService#charge") rather than its database/HTTP
+      # calls appearing to hang directly off the controller root. Whether this ever actually
+      # reaches the server is a completely separate question from whether it's on: distributed
+      # tracing is a plan-gated feature (see Api::V1::SpansController), enforced server-side the
+      # same way every other ingestion endpoint already is, not by this client-side flag.
+      @track_tracing = true
+      # Only a request whose own total duration is at least this slow ever gets sent at all: see
+      # SpanBuffer#slow?/Middleware::SpanTracing's own comment for why that decision happens
+      # entirely client-side, before a single byte goes over the wire, rather than sending every
+      # trace and letting the server decide. 1 second is a deliberately conservative default (most
+      # web apps consider anything near that already a bad user experience), not tied to any
+      # PerformanceSample-derived threshold elsewhere in this gem: those describe a trend across
+      # many requests, this describes a single one being outright slow.
+      @trace_capture_threshold_ms = 1_000
     end
 
     def api_key
@@ -128,6 +171,24 @@ module ForgeOpsTracker
       uri
     end
 
+    def failure_events_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/failure_events")
+      uri
+    end
+
+    def spans_uri
+      uri = ingestion_uri
+      return nil unless uri
+
+      uri = uri.dup
+      uri.path = uri.path.sub(%r{/events\z}, "/spans")
+      uri
+    end
+
     def enabled?
       !blank?(dsn) && !blank?(api_key) && enabled_environments.map(&:to_s).include?(environment.to_s)
     end

data/lib/forge_ops_tracker/error_subscriber.rb

@@ -5,6 +5,16 @@ module ForgeOpsTracker
   # wrapped to guarantee this never propagates an exception back into the
   # host app's error-handling cycle.
   class ErrorSubscriber
+    # Sidekiq raises its own control-flow exception once it has already scheduled a failed job's
+    # retry (Sidekiq::JobRetry::Handled, and Sidekiq::JobRetry::Skip, a subclass, for a job that
+    # opted out of retries), and Rails' executor, which Sidekiq's Rails integration wraps every
+    # job in, reports whatever escapes that wrap. Left alone, every failing Sidekiq job, whatever
+    # actually went wrong inside it, reaches ForgeOps as the same opaque "Sidekiq::JobRetry::
+    # Handled" issue with the real error nowhere in it. The real error is that exception's #cause
+    # (Sidekiq raises it from inside its own rescue), so it's unwrapped here, by class name
+    # rather than constant since Sidekiq is an optional dependency this gem never requires.
+    RETRY_CONTROL_FLOW_CLASS = "Sidekiq::JobRetry::Handled".freeze
+
     def initialize(configuration, delivery_queue: DeliveryQueue.new(configuration), event_builder: EventBuilder.new(configuration))
       @configuration = configuration
       @delivery_queue = delivery_queue
@@ -18,8 +28,11 @@ module ForgeOpsTracker
       # hands this class the Rack env, so ForgeOpsTracker::Middleware::UserContext (Warden
       # auto-detection) and ForgeOpsTracker.set_user (the manual override/fallback) both
       # communicate with this exact same thread-local instead, the only channel available here.
+      # ForgeOpsTracker::Middleware::BreadcrumbContext's own per-request buffer works the same way.
+      error = unwrap_retry_control_flow(error)
       user = Thread.current[:forge_ops_tracker_current_user]
-      delivery_queue.push(event_builder.build(error, context: context, user: user))
+      breadcrumbs = Thread.current[:forge_ops_tracker_breadcrumbs]&.all || []
+      delivery_queue.push(event_builder.build(error, context: context, user: user, breadcrumbs: breadcrumbs))
       nil
     rescue StandardError => e
       configuration.logger&.debug { "[ForgeOpsTracker] report failed: #{e.class}: #{e.message}" }
@@ -28,5 +41,14 @@ module ForgeOpsTracker
 
     private
       attr_reader :configuration, :delivery_queue, :event_builder
+
+      # Only ever unwraps when there is a real #cause to unwrap to; a Handled with no cause (not
+      # something Sidekiq itself does) is reported as-is rather than dropped.
+      def unwrap_retry_control_flow(error)
+        while error.cause && error.class.ancestors.any? { |ancestor| ancestor.name == RETRY_CONTROL_FLOW_CLASS }
+          error = error.cause
+        end
+        error
+      end
   end
 end

data/lib/forge_ops_tracker/event_builder.rb

@@ -27,7 +27,7 @@ module ForgeOpsTracker
       @configuration = configuration
     end
 
-    def build(error, context: {}, user: nil)
+    def build(error, context: {}, user: nil, breadcrumbs: [])
       payload = {
         exception_class: error.class.name,
         message: error.message.to_s,
@@ -41,6 +41,7 @@ module ForgeOpsTracker
         sdk_name: SDK_NAME
       }
       payload[:user] = user if user && !user.empty?
+      payload[:breadcrumbs] = breadcrumbs if breadcrumbs && !breadcrumbs.empty?
       scrub(payload)
     end
 
@@ -51,16 +52,22 @@ module ForgeOpsTracker
       # structured fields this gem or the host app sets deliberately, not free text an exception
       # or its context could accidentally spill sensitive data into. user specifically is a
       # deliberate exemption, not an oversight: the server's own PiiScrubber-equivalent would
-      # otherwise redact the exact email address this field exists to carry.
+      # otherwise redact the exact email address this field exists to carry. breadcrumbs is *not*
+      # exempt, unlike user: console-style/query/request trail entries are exactly the kind of free
+      # text (a query's bind params showing up in a message, a URL with a token in it) the scrubber
+      # exists to catch, matching the server's own api/v1/events_controller.rb treatment of this
+      # field.
       def scrub(payload)
         return payload unless configuration.scrub_pii
 
-        payload.merge(
+        scrubbed = payload.merge(
           message: PiiScrubber.scrub(payload[:message]),
           backtrace: PiiScrubber.scrub(payload[:backtrace]),
           context: PiiScrubber.scrub(payload[:context]),
           tags: PiiScrubber.scrub(payload[:tags])
         )
+        scrubbed[:breadcrumbs] = PiiScrubber.scrub(payload[:breadcrumbs]) if payload.key?(:breadcrumbs)
+        scrubbed
       end
 
       def backtrace_frames(error)

data/lib/forge_ops_tracker/failure_event_buffer.rb

@@ -0,0 +1,73 @@
+require "thread"
+
+module ForgeOpsTracker
+  # Collects individual job/dependency failures in-process (see Railtie's own perform.active_job
+  # subscription and Integrations::NetHTTP::Timing#request, the two places that actually call
+  # #record) and periodically flushes them as one batch, the same shape MetricBuffer already uses
+  # for the identical reason: a real failure's own detail (which job, what error) is exactly the
+  # kind of thing worth an accurate individual record of, not something to pre-aggregate away the
+  # way PerformanceFlusher's own count/sum/max buckets duration.
+  class FailureEventBuffer
+    def initialize(configuration, client: Client.new(configuration))
+      @configuration = configuration
+      @client = client
+      @mutex = Mutex.new
+      @entries = []
+      @thread = nil
+
+      # Flushes whatever's already buffered on a normal process exit, so the last partial window
+      # isn't silently dropped; same reasoning every other buffer/flusher in this gem documents.
+      at_exit { flush }
+    end
+
+    def record(kind:, transaction_name:, error_class: nil, error_message: nil)
+      ensure_worker_started
+
+      @mutex.synchronize do
+        @entries << {
+          kind: kind, transaction_name: transaction_name, error_class: error_class, error_message: error_message,
+          environment: configuration.environment.to_s, release: configuration.release,
+          occurred_at: Time.now.utc.iso8601
+        }
+      end
+    end
+
+    # Snapshots and resets the buffered entries, then delivers them as one batch. Same "a failed
+    # delivery keeps every entry for the next attempt" contract as MetricBuffer#flush.
+    def flush
+      snapshot = nil
+
+      @mutex.synchronize do
+        return if @entries.empty?
+        snapshot = @entries
+      end
+
+      return unless client.deliver_failure_events(snapshot)
+
+      @mutex.synchronize { @entries = [] }
+    end
+
+    private
+      attr_reader :configuration, :client
+
+      def ensure_worker_started
+        return if @thread&.alive?
+
+        @mutex.synchronize do
+          return if @thread&.alive?
+
+          @thread = Thread.new { run }
+          @thread.abort_on_exception = false
+        end
+      end
+
+      def run
+        loop do
+          sleep configuration.failure_event_flush_interval
+          flush
+        rescue StandardError => e
+          configuration.logger&.debug { "[ForgeOpsTracker] failure event flush thread error: #{e.class}: #{e.message}" }
+        end
+      end
+  end
+end

data/lib/forge_ops_tracker/histogram_bucketer.rb

@@ -0,0 +1,32 @@
+module ForgeOpsTracker
+  # Buckets a single duration into one of a fixed set of latency-range labels, the building block
+  # PerformanceFlusher uses to accumulate an approximate distribution (not just count/sum/max)
+  # alongside every [transaction_name, kind] bucket it already tallies. The server merges these
+  # counts across matching samples at read time and walks cumulative counts to approximate a
+  # percentile, the same "accurate to the bucket width, not exact" trade-off Prometheus's own
+  # histogram_quantile makes: this SDK does not store, or need, the raw duration list a true
+  # percentile would require.
+  #
+  # BOUNDARIES_MS is duplicated (not shared via a gem dependency) on the server side, in
+  # app/services/histogram_percentile.rb: the gem and the Rails app it reports to are separate
+  # deployables, the same "duplicated, not imported, so this package's own dependency graph stays
+  # independent" reasoning already applied elsewhere in this repo (e.g. the React Native SDK's own
+  # copy of the TypeScript SDK's PII scrubber). Change one, change the other, or a released gem
+  # version and the server version it talks to would silently disagree about what each bucket label
+  # means.
+  module HistogramBucketer
+    BOUNDARIES_MS = [ 50, 100, 250, 500, 1000, 2500, 5000, 10_000 ].freeze
+
+    # Returns the label (a String) of the smallest boundary duration_ms fits under, or "inf" for
+    # anything larger than the largest boundary. String, not Integer/Float: this travels as a JSON
+    # object key once flushed (see PerformanceFlusher#flush), and JSON object keys are always
+    # strings regardless of what Ruby type builds the Hash locally, so returning a String here
+    # keeps the in-process Hash's own keys identical to what actually goes over the wire, rather
+    # than looking different locally and then silently getting stringified only at serialization
+    # time.
+    def self.bucket_for(duration_ms)
+      boundary = BOUNDARIES_MS.find { |b| duration_ms <= b }
+      boundary ? boundary.to_s : "inf"
+    end
+  end
+end

data/lib/forge_ops_tracker/infrastructure_metric_buffer.rb

@@ -10,6 +10,12 @@ module ForgeOpsTracker
   # background-thread timer still runs for a long-lived process that chooses to call this
   # continuously instead, but rarely gets the chance to fire in the cron case.
   class InfrastructureMetricBuffer
+    # Once this many entries are buffered, further ones are dropped until a flush succeeds: a plan
+    # without the feature answers 403 on every flush, and an uncapped buffer would then grow for as
+    # long as the process lives. Dropping the newest rather than the oldest keeps the entries a flush
+    # is delivering at the front of the array, which is what makes removing exactly them exact.
+    MAX_ENTRIES = 1000
+
     def initialize(configuration, client: Client.new(configuration))
       @configuration = configuration
       @client = client
@@ -21,9 +27,13 @@ module ForgeOpsTracker
     end
 
     def record(metric_name:, value:, hostname:)
+      return false unless keepable?(value)
+
       ensure_worker_started
 
       @mutex.synchronize do
+        return false if @entries.size >= MAX_ENTRIES
+
         @entries << { metric_name: metric_name, value: value, hostname: hostname, recorded_at: Time.now.utc.iso8601 }
       end
     end
@@ -35,17 +45,24 @@ module ForgeOpsTracker
 
       @mutex.synchronize do
         return if @entries.empty?
-        snapshot = @entries
+        snapshot = @entries.dup
       end
 
       return unless client.deliver_infrastructure_metrics(snapshot)
 
-      @mutex.synchronize { @entries = [] }
+      # Exactly the entries just delivered: anything recorded while the request was in flight sits
+      # after them and stays for the next flush (resetting the whole array here lost it).
+      @mutex.synchronize { @entries.shift(snapshot.size) }
     end
 
     private
       attr_reader :configuration, :client
 
+      # A NaN or infinite value is not valid JSON and would make the server reject the whole batch.
+      def keepable?(value)
+        value.is_a?(Numeric) && value.to_f.finite?
+      end
+
       def ensure_worker_started
         return if @thread&.alive?