forge_ops_tracker 0.15.1 → 0.17.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 81c08199bfeae97d20295b7c04a46dc3b9dd9be519d4790404b88e2ca514fbe1
-  data.tar.gz: dad1efa10164cebcbce9452c1f903812157143e871162076f09298ebf41a6ea7
+  metadata.gz: 304c9a15155ecdf53fe8e181dd7f43b56a08d7397e7d63555e428185ece06560
+  data.tar.gz: d4d2cb88ffec650e579c7279c590e4eaee8f288bc6ef7041d2c2f5785767fa2f
 SHA512:
-  metadata.gz: 2a32b34300ee005cd09be7f188275091ec9ba8cf4ef2e65044176a7e6d1427feef170cbc44dbd9f7a25dcf0073b2afb6186db3db2069a5737bfdcb274e62cc7f
-  data.tar.gz: 4431641c5e95f757dd82e7af0015de824804610ee4ebf252cbe3a4db13ca5f5aa1cac5fda274a94833be652a9d6c5e25ab4406570d4790b165cf242f8490ff66
+  metadata.gz: dfd3a51d60e0decd113e2a8bd50db101ebd4ec57ebb468a522413b4f9ffa0c5c15d5440939c7a6f772561ce6ffe5c7078322d2903f0d3ec87006f1dd368c1718
+  data.tar.gz: 1646c0b506c357122c513604d031974318df0086a0c2a962c99e80143624be459755fab5dfff3df7718f24bb6caebe699ba5325fafb79b35a92f76ffa0783b63

data/CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## 0.17.0 (2026-10-13)
+
+- Fixed: a batch the server refused was re-sent at every flush for as long as the process lived,
+  with everything recorded since added to it. Performance samples, custom and infrastructure
+  metrics, session counts and failure events are now dropped when the server answers with a 4xx
+  other than 408 or 429 (a 400, 401, 403, 413 or 422, for example), since sending the same batch
+  again can only fail the same way. Once a metric buffer had filled up behind such a batch, every
+  new metric was dropped too; that no longer happens.
+- A batch that could not be delivered for now (a network error, a timeout, a 5xx, a 408 or a 429)
+  is still kept for the next flush, but only for 5 flushes in a row. After the fifth it is dropped
+  and the buffer starts clean, so a long outage no longer holds the same data in memory forever.
+- The failure event buffer now holds at most 1000 entries; further ones are dropped until a flush
+  lets go of them, the same way the metric buffers already behave.
+- With a `config.logger`, each drop is logged once at debug with what was dropped and why, and a
+  non-2xx answer is logged with its status code.
+- Internal: `ForgeOpsTracker::Client#deliver` and its sibling `deliver_*` methods now return
+  `:delivered`, `:rejected` or `:failed` instead of `true`/`false`. All three are truthy, so code
+  calling `Client` directly should compare against `:delivered`. Nothing in the documented API
+  changed.
+
+## 0.16.0 (2026-10-02)
+
+- Changed: outside Rails, the environment now defaults to `production` when nothing sets it (it
+  was `development`). It comes from `FORGE_OPS_ENVIRONMENT`, then `RAILS_ENV`, then `RACK_ENV`,
+  then `production`. Only `production` and `staging` send by default, so before this a plain Ruby
+  or Rack process that set just a DSN sent nothing at all. To keep the old behavior there, set
+  `FORGE_OPS_ENVIRONMENT=development` (or `RACK_ENV=development`, or `config.environment`).
+- Changed: under Rails, `FORGE_OPS_ENVIRONMENT` now takes precedence over `Rails.env` when it's
+  set. Without it nothing changes: development and test still don't send.
+- When a DSN is set but the environment isn't in `enabled_environments`, one warning is now logged
+  per process, naming the environment and how to send from it: at boot under Rails (to
+  `Rails.logger`), otherwise from `configure`, the Rack middleware, or `capture_exception` (to
+  `config.logger`, or stderr when there isn't one). Nothing is logged when no DSN is set.
+- New `ForgeOpsTracker.capture_exception(error, context: {})` reports an exception you've rescued,
+  through the same path `Rails.error` reports into, so it works without Rails too.
+- Plain Rack and Sinatra apps are now covered: `use ForgeOpsTracker::Middleware::RequestContext`
+  reports an exception escaping the app, and an error Sinatra itself turned into a 5xx response.
+  Under Rails it still only adds request context, since `Rails.error` already reports those.
+
 ## 0.15.1 (2026-10-01)
 
 - Errors reported just before a process exits are now sent before it ends. Delivery runs on a

data/README.md

@@ -1,6 +1,7 @@
 # ForgeOpsTracker
 
-Rails exception reporting client for [ForgeOps](https://getforgeops.net).
+Exception reporting client for [ForgeOps](https://getforgeops.net), for Rails, plain Rack and
+Sinatra apps, and scripts.
 
 ## Installation
 
@@ -25,6 +26,11 @@ ForgeOpsTracker.configure do |config|
 end
 ```
 
+The environment is `FORGE_OPS_ENVIRONMENT` when set, otherwise `Rails.env` under Rails, and
+outside Rails `RAILS_ENV`, then `RACK_ENV`, then `production`. Only `production` and `staging`
+send by default, so a Rails app in development or test doesn't report. When a DSN is set but the
+environment isn't enabled, one warning is logged per process (at boot under Rails) saying so.
+
 Delivery happens on a small background thread with a bounded queue and short HTTP timeouts. Every
 failure mode (network errors, timeouts, a full queue, a malformed DSN) is caught and dropped
 rather than raised, so a broken or unreachable tracker can never take down the host app.
@@ -71,6 +77,34 @@ Bottom line: if an exception would otherwise crash something, you're already cov
 code already catches and handles it, route that specific `rescue` through `Rails.error.handle`/
 `.record` instead of a bare one wherever you want ForgeOps to know about it.
 
+### Outside Rails: Rack, Sinatra, and scripts
+
+Without Rails there's no `Rails.error`, so add the Rack middleware, which reports any exception
+escaping your app (and, in Sinatra, an error it turned into a 5xx response itself), then re-raises
+it unchanged:
+
+```ruby
+# config.ru, or inside a Sinatra::Base subclass
+require "forge_ops_tracker"
+
+ForgeOpsTracker.configure do |config|
+  config.dsn = ENV["FORGE_OPS_DSN"]
+end
+
+use ForgeOpsTracker::Middleware::RequestContext
+```
+
+Report an exception you rescued yourself, anywhere (Rails included), with
+`ForgeOpsTracker.capture_exception`:
+
+```ruby
+begin
+  charge_card(order)
+rescue Stripe::CardError => e
+  ForgeOpsTracker.capture_exception(e, context: { order_id: order.id })
+end
+```
+
 ### Sidekiq job failures and renamed job classes
 
 A failing Sidekiq job is reported through `Rails.error` like any other error. Since 0.10.1 the issue
@@ -254,7 +288,7 @@ end
 ```
 
 Requires a ForgeOps plan that includes release health; on a plan that doesn't, the periodic
-flushes are simply rejected server-side and dropped, exactly like any other delivery failure.
+flushes are accepted but not recorded, and the response says why.
 
 ## Performance monitoring
 
@@ -279,8 +313,7 @@ end
 ```
 
 Requires a ForgeOps plan that includes performance monitoring; on a plan that doesn't, the
-periodic flushes are simply rejected server-side and dropped, exactly like any other delivery
-failure.
+periodic flushes are accepted but not recorded, and the response says why.
 
 ### Database queries, background jobs, and outbound HTTP calls
 
@@ -353,8 +386,7 @@ unaffected: confirmed directly that Net::HTTP connects lazily, inside `#request`
 time it's called on a not-yet-started instance.
 
 Requires a ForgeOps plan that includes performance monitoring, the same plan feature performance
-monitoring itself already requires; on a plan that doesn't, the periodic flushes are simply
-rejected server-side and dropped, exactly like any other delivery failure.
+monitoring itself already requires; on a plan that doesn't, the periodic flushes are accepted but not recorded, and the response says why.
 
 ## Tracing
 
@@ -408,7 +440,7 @@ end
 ```
 
 Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn't, traces are
-simply rejected server-side and dropped, exactly like any other delivery failure. The `trace_id`
+accepted but not recorded, and the response says why. The `trace_id`
 on error events is sent either way.
 
 ### The SQL behind each database span
@@ -476,7 +508,7 @@ What it doesn't do:
 It does use one pooled connection, briefly, for each plan. With the limits above that's at most 10
 short checkouts a minute, but on a pool that's already at capacity, leave it off or raise
 `explain_threshold_ms`. Plans are part of performance monitoring; on a plan without it, they're
-rejected server-side and dropped.
+accepted but not recorded, and the response says why.
 
 ## What changed
 
@@ -521,8 +553,8 @@ their values), so an added or removed variable shows up as a change. Names that
 host, like `HOSTNAME`, `PATH`, `PORT`, `LC_*`, and Kubernetes service variables, are left out, as
 are the gem's own `FORGE_OPS_*` settings.
 
-Requires a ForgeOps plan that includes change tracking; on a plan that doesn't, both are rejected
-server-side and dropped, exactly like any other delivery failure.
+Requires a ForgeOps plan that includes change tracking; on a plan that doesn't, both are accepted but not recorded,
+and the response says why.
 
 ## Custom metrics
 
@@ -543,7 +575,7 @@ end
 ```
 
 Requires a ForgeOps plan that includes custom metrics; on a plan that doesn't, the periodic
-flushes are simply rejected server-side and dropped, exactly like any other delivery failure.
+flushes are accepted but not recorded, and the response says why.
 
 ## Infrastructure monitoring
 
@@ -572,5 +604,4 @@ ForgeOpsTracker.capture_infrastructure_metric("disk_used_percent", value: disk_u
 
 The example above is Linux-specific (`/proc/loadavg`, `/proc/meminfo`); on another OS, read that
 platform's own equivalents instead. Requires a ForgeOps plan that includes infrastructure
-monitoring; on a plan that doesn't, the periodic flushes are simply rejected server-side and
-dropped, exactly like any other delivery failure.
+monitoring; on a plan that doesn't, the periodic flushes are accepted but not recorded, and the response says why.

data/lib/forge_ops_tracker/change_snapshot.rb

@@ -66,7 +66,7 @@ module ForgeOpsTracker
     end
 
     def deliver
-      client.deliver_change_snapshot(payload)
+      DeliveryStatus.from(client.deliver_change_snapshot(payload)) == DeliveryStatus::DELIVERED
     rescue StandardError => e
       configuration.logger&.debug { "[ForgeOpsTracker] change snapshot failed: #{e.class}: #{e.message}" }
       false

data/lib/forge_ops_tracker/client.rb

@@ -5,8 +5,10 @@ require "uri"
 module ForgeOpsTracker
   # Delivers one payload over HTTP. Every failure mode; DNS, connection,
   # timeout, TLS, a non-2xx response; is caught here and turned into a
-  # `false` return rather than a raised exception, since a broken or
-  # unreachable tracker must never be able to break the host app.
+  # DeliveryStatus rather than a raised exception, since a broken or
+  # unreachable tracker must never be able to break the host app. Every
+  # deliver_* method answers one of DeliveryStatus::DELIVERED, REJECTED or
+  # FAILED (all three are truthy, so compare, never test for truth).
   class Client
     def initialize(configuration)
       @configuration = configuration
@@ -16,8 +18,8 @@ module ForgeOpsTracker
       post(configuration.ingestion_uri, payload)
     end
 
-    # Same delivery contract as deliver above (never raises, returns a bare
-    # true/false), just a different endpoint under the same DSN's host and
+    # Same delivery contract as deliver above (never raises, returns a
+    # DeliveryStatus), just a different endpoint under the same DSN's host and
     # api_key; see Configuration#session_checkins_uri.
     def deliver_session_checkin(payload)
       post(configuration.session_checkins_uri, payload)
@@ -61,7 +63,7 @@ module ForgeOpsTracker
     end
 
     # Same delivery contract again; see Configuration#changes_uri and ForgeOpsTracker.record_change.
-    # One change per call. A plan without change tracking answers 403, which is just a false here.
+    # One change per call. A plan without change tracking answers 403, which is a REJECTED here.
     def deliver_change(payload)
       post(configuration.changes_uri, payload)
     end
@@ -72,7 +74,7 @@ module ForgeOpsTracker
     end
 
     # Same again; see Configuration#query_plans_uri and QueryExplainer. One plan per call. A plan
-    # without performance monitoring answers 403, which is just a false here.
+    # without performance monitoring answers 403, which is a REJECTED here.
     def deliver_query_plan(payload)
       post(configuration.query_plans_uri, payload)
     end
@@ -81,13 +83,15 @@ module ForgeOpsTracker
       attr_reader :configuration
 
       def post(uri, payload)
-        return false unless uri
+        return DeliveryStatus::FAILED unless uri
 
         response = http_for(uri).request(build_request(uri, payload))
-        response.is_a?(Net::HTTPSuccess)
+        status = DeliveryStatus.for_code(response.code)
+        log { "delivery to #{uri.path} answered HTTP #{response.code}" } unless status == DeliveryStatus::DELIVERED
+        status
       rescue StandardError => e
         log { "delivery failed: #{e.class}: #{e.message}" }
-        false
+        DeliveryStatus::FAILED
       end
 
       def http_for(uri)

data/lib/forge_ops_tracker/configuration.rb

@@ -23,7 +23,7 @@ module ForgeOpsTracker
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
-      @environment = ENV["RAILS_ENV"] || ENV["RACK_ENV"] || "development"
+      @environment = self.class.resolve_environment
       @release = ENV["FORGE_OPS_RELEASE"]
       @server_name = safe_hostname
       @enabled_environments = %w[production staging]
@@ -262,10 +262,39 @@ module ForgeOpsTracker
     end
 
     def enabled?
-      !blank?(dsn) && !blank?(api_key) && enabled_environments.map(&:to_s).include?(environment.to_s)
+      configured? && environment_enabled?
+    end
+
+    # Whether a usable DSN is set, regardless of environment.
+    def configured?
+      !blank?(dsn) && !blank?(api_key)
+    end
+
+    # The one-line warning ForgeOpsTracker.warn_if_environment_disabled logs when a DSN is set but
+    # this environment isn't one that sends, or nil when there's nothing to warn about (no DSN, or
+    # the environment is enabled).
+    def disabled_environment_warning
+      return nil if !configured? || environment_enabled?
+
+      enabled = enabled_environments.map(&:to_s)
+      enabled = enabled.empty? ? "none" : enabled.join(", ")
+      "[ForgeOps] Not sending: this environment is \"#{environment}\", and only #{enabled} are enabled. " \
+        "Set FORGE_OPS_ENVIRONMENT=production (or add \"#{environment}\" to enabled_environments) to send from here."
+    end
+
+    # FORGE_OPS_ENVIRONMENT, then RAILS_ENV, then RACK_ENV, then "production". Defaulting to
+    # "production" means a plain Ruby or Rack process that only sets a DSN actually sends; a Rails
+    # app gets Rails.env from the Railtie instead (unless FORGE_OPS_ENVIRONMENT is set), so
+    # development and test stay quiet there, with one warning at boot when a DSN is set.
+    def self.resolve_environment(env = ENV)
+      [ env["FORGE_OPS_ENVIRONMENT"], env["RAILS_ENV"], env["RACK_ENV"] ].find { |value| value && !value.strip.empty? } || "production"
     end
 
     private
+      def environment_enabled?
+        enabled_environments.map(&:to_s).include?(environment.to_s)
+      end
+
       # Same substitution every *_uri method above spells out by hand: a sibling path under the
       # DSN's own /api/v1/events.
       def sibling_uri(path)

data/lib/forge_ops_tracker/delivery_queue.rb

@@ -39,7 +39,7 @@ module ForgeOpsTracker
         rescue ThreadError
           break
         end
-        client.public_send(@deliver_with, payload)
+        deliver(payload)
       end
       sleep 0.01 while @in_flight.positive? && monotonic_now < deadline
     rescue StandardError => e
@@ -78,7 +78,7 @@ module ForgeOpsTracker
           payload = @queue.pop
           @in_flight += 1
           begin
-            client.public_send(@deliver_with, payload)
+            deliver(payload)
           ensure
             @in_flight -= 1
           end
@@ -87,6 +87,17 @@ module ForgeOpsTracker
         end
       end
 
+      # One attempt per payload, as it has always been: a queued event is never held back for a
+      # retry, whatever the outcome, so all that is left to do with a refusal is to say so once.
+      def deliver(payload)
+        case DeliveryStatus.from(client.public_send(@deliver_with, payload))
+        when DeliveryStatus::REJECTED
+          log { "dropping event: the server rejected it and will not accept a retry" }
+        when DeliveryStatus::FAILED
+          log { "dropping event: delivery failed" }
+        end
+      end
+
       # Registered on the first push, not at load, so a process that never reports anything never
       # waits on exit at all.
       def register_drain

data/lib/forge_ops_tracker/delivery_status.rb

@@ -0,0 +1,74 @@
+module ForgeOpsTracker
+  # What became of one delivery attempt, as every Client#deliver_* method reports it. A bare
+  # true/false could not tell "the server will never take this payload" apart from "try again
+  # later", so a batch a plan or a validation rule refused was re-sent at every flush for as long
+  # as the process lived.
+  module DeliveryStatus
+    # Any 2xx.
+    DELIVERED = :delivered
+    # A 4xx other than 408 and 429: sending the same payload again can only fail the same way.
+    REJECTED = :rejected
+    # A network error, a timeout, a 5xx, a 408 or a 429: worth another attempt later.
+    FAILED = :failed
+
+    RETRYABLE_CLIENT_ERRORS = [ 408, 429 ].freeze
+
+    def self.for_code(code)
+      code = code.to_i
+      return DELIVERED if code.between?(200, 299)
+      return REJECTED if code.between?(400, 499) && !RETRYABLE_CLIENT_ERRORS.include?(code)
+
+      FAILED
+    end
+
+    # Reads whatever a client handed back. A status passes through; anything else is read the way
+    # the old boolean contract was (truthy delivered, falsy failed for now), so a client injected
+    # through a `client:` keyword that still answers true/false keeps working.
+    def self.from(result)
+      return result if [ DELIVERED, REJECTED, FAILED ].include?(result)
+
+      result ? DELIVERED : FAILED
+    end
+  end
+
+  # Decides, after each flush of one buffer/flusher, whether the batch it just sent is done with.
+  # Delivered and rejected batches are let go of at once; one that failed for now is kept, but only
+  # for MAX_FAILED_FLUSHES flushes in a row, so an outage (or a server answering 500 to one
+  # particular batch) can never pin the same data in memory for the life of the process.
+  class FlushOutcome
+    MAX_FAILED_FLUSHES = 5
+
+    # what names the batch in the debug log ("metrics", "performance samples").
+    def initialize(configuration, what)
+      @configuration = configuration
+      @what = what
+      @failed_flushes = 0
+    end
+
+    # True when the caller should remove what it just sent (delivered, or dropped), false when it
+    # should keep it for the next flush.
+    def release?(result)
+      case DeliveryStatus.from(result)
+      when DeliveryStatus::DELIVERED
+        @failed_flushes = 0
+        true
+      when DeliveryStatus::REJECTED
+        @failed_flushes = 0
+        log { "dropping buffered #{@what}: the server rejected them and will not accept a retry" }
+        true
+      else
+        @failed_flushes += 1
+        return false if @failed_flushes < MAX_FAILED_FLUSHES
+
+        @failed_flushes = 0
+        log { "dropping buffered #{@what}: #{MAX_FAILED_FLUSHES} flushes in a row failed" }
+        true
+      end
+    end
+
+    private
+      def log
+        @configuration.logger&.debug { "[ForgeOpsTracker] #{yield}" }
+      end
+  end
+end

data/lib/forge_ops_tracker/failure_event_buffer.rb

@@ -8,12 +8,17 @@ module ForgeOpsTracker
   # kind of thing worth an accurate individual record of, not something to pre-aggregate away the
   # way PerformanceFlusher's own count/sum/max buckets duration.
   class FailureEventBuffer
+    # Same cap, and the same "drop the newest, not the oldest" reasoning, as MetricBuffer's own
+    # MAX_ENTRIES: a burst of failures during an outage must not grow this without bound.
+    MAX_ENTRIES = 1000
+
     def initialize(configuration, client: Client.new(configuration))
       @configuration = configuration
       @client = client
       @mutex = Mutex.new
       @entries = []
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "failure events")
 
       # Flushes whatever's already buffered on a normal process exit, so the last partial window
       # isn't silently dropped; same reasoning every other buffer/flusher in this gem documents.
@@ -24,6 +29,8 @@ module ForgeOpsTracker
       ensure_worker_started
 
       @mutex.synchronize do
+        return false if @entries.size >= MAX_ENTRIES
+
         @entries << {
           kind: kind, transaction_name: transaction_name, error_class: error_class, error_message: error_message,
           environment: configuration.environment.to_s, release: configuration.release,
@@ -33,18 +40,22 @@ module ForgeOpsTracker
     end
 
     # Snapshots and resets the buffered entries, then delivers them as one batch. Same "a failed
-    # delivery keeps every entry for the next attempt" contract as MetricBuffer#flush.
+    # delivery keeps every entry for the next attempt" contract as MetricBuffer#flush, with the
+    # same two exceptions: a rejected batch, and one that failed FlushOutcome::MAX_FAILED_FLUSHES
+    # flushes in a row, are dropped.
     def flush
       snapshot = nil
 
       @mutex.synchronize do
         return if @entries.empty?
-        snapshot = @entries
+        snapshot = @entries.dup
       end
 
-      return unless client.deliver_failure_events(snapshot)
+      return unless @outcome.release?(client.deliver_failure_events(snapshot))
 
-      @mutex.synchronize { @entries = [] }
+      # Exactly the entries just sent: anything recorded while the request was in flight sits after
+      # them and stays for the next flush.
+      @mutex.synchronize { @entries.shift(snapshot.size) }
     end
 
     private

data/lib/forge_ops_tracker/infrastructure_metric_buffer.rb

@@ -3,16 +3,16 @@ require "thread"
 module ForgeOpsTracker
   # Collects individual ForgeOpsTracker.capture_infrastructure_metric readings in-process and
   # periodically flushes them as one batch, same shape as MetricBuffer (a sibling class, not a
-  # shared base -- this gem keeps SessionFlusher/PerformanceFlusher as separate, independently-
+  # shared base; this gem keeps SessionFlusher/PerformanceFlusher as separate, independently-
   # readable classes despite their own similar shape too, and these two follow that precedent).
   # Its main real caller is a short-lived cron/script process that exits right after a handful of
   # capture calls, so the at_exit flush below is what actually matters in practice; the periodic
   # background-thread timer still runs for a long-lived process that chooses to call this
   # continuously instead, but rarely gets the chance to fire in the cron case.
   class InfrastructureMetricBuffer
-    # Once this many entries are buffered, further ones are dropped until a flush succeeds: a plan
-    # without the feature answers 403 on every flush, and an uncapped buffer would then grow for as
-    # long as the process lives. Dropping the newest rather than the oldest keeps the entries a flush
+    # Once this many entries are buffered, further ones are dropped until a flush lets go of them:
+    # an unreachable server fails every flush, and an uncapped buffer would keep growing until
+    # FlushOutcome gives up on the batch. Dropping the newest rather than the oldest keeps the entries a flush
     # is delivering at the front of the array, which is what makes removing exactly them exact.
     MAX_ENTRIES = 1000
 
@@ -22,6 +22,7 @@ module ForgeOpsTracker
       @mutex = Mutex.new
       @entries = []
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "infrastructure metrics")
 
       at_exit { flush }
     end
@@ -38,7 +39,7 @@ module ForgeOpsTracker
       end
     end
 
-    # Same reset-on-success/keep-on-failure contract as MetricBuffer#flush; see that class's own
+    # Same reset-on-success/keep-on-failure/drop-on-rejection contract as MetricBuffer#flush; see that class's own
     # comment for the accepted race this shares with it.
     def flush
       snapshot = nil
@@ -48,9 +49,9 @@ module ForgeOpsTracker
         snapshot = @entries.dup
       end
 
-      return unless client.deliver_infrastructure_metrics(snapshot)
+      return unless @outcome.release?(client.deliver_infrastructure_metrics(snapshot))
 
-      # Exactly the entries just delivered: anything recorded while the request was in flight sits
+      # Exactly the entries just sent: anything recorded while the request was in flight sits
       # after them and stays for the next flush (resetting the whole array here lost it).
       @mutex.synchronize { @entries.shift(snapshot.size) }
     end

data/lib/forge_ops_tracker/metric_buffer.rb

@@ -10,9 +10,9 @@ module ForgeOpsTracker
   # they'll want a genuinely accurate count/sum of later, not something to pre-aggregate away
   # client-side, so CustomMetricsController stores one row per entry as-is.
   class MetricBuffer
-    # Once this many entries are buffered, further ones are dropped until a flush succeeds: a plan
-    # without the feature answers 403 on every flush, and an uncapped buffer would then grow for as
-    # long as the process lives. Dropping the newest rather than the oldest keeps the entries a flush
+    # Once this many entries are buffered, further ones are dropped until a flush lets go of them:
+    # an unreachable server fails every flush, and an uncapped buffer would keep growing until
+    # FlushOutcome gives up on the batch. Dropping the newest rather than the oldest keeps the entries a flush
     # is delivering at the front of the array, which is what makes removing exactly them exact.
     MAX_ENTRIES = 1000
 
@@ -22,6 +22,7 @@ module ForgeOpsTracker
       @mutex = Mutex.new
       @entries = []
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "metrics")
 
       # Flushes whatever's already buffered on a normal process exit, so the last partial window
       # isn't silently dropped; same reasoning every other buffer/flusher in this gem documents.
@@ -44,10 +45,11 @@ module ForgeOpsTracker
       end
     end
 
-    # Snapshots and resets the buffered entries, then delivers them as one batch. A failed
-    # delivery keeps every entry where it is rather than resetting, so the next flush's batch
-    # just grows instead of losing what was already buffered; same reasoning SessionFlusher#flush
-    # already documents, including the same small, accepted race (an entry recorded during the
+    # Snapshots and resets the buffered entries, then delivers them as one batch. A delivery that
+    # failed for now keeps every entry where it is rather than resetting, so the next flush's batch
+    # just grows instead of losing what was already buffered (a rejected batch, or one that failed
+    # FlushOutcome::MAX_FAILED_FLUSHES flushes in a row, is dropped instead); same reasoning
+    # SessionFlusher#flush already documents, including the same small, accepted race (an entry recorded during the
     # in-flight HTTP request gets folded into this snapshot's own reset rather than kept for the
     # next one) that class already lives with, for the same reason: simple and consistent beats a
     # cleverer dedup that would only trade one rare edge case for a subtler one.
@@ -59,9 +61,9 @@ module ForgeOpsTracker
         snapshot = @entries.dup
       end
 
-      return unless client.deliver_metrics(snapshot)
+      return unless @outcome.release?(client.deliver_metrics(snapshot))
 
-      # Exactly the entries just delivered: anything recorded while the request was in flight sits
+      # Exactly the entries just sent: anything recorded while the request was in flight sits
       # after them and stays for the next flush (resetting the whole array here lost it).
       @mutex.synchronize { @entries.shift(snapshot.size) }
     end

data/lib/forge_ops_tracker/middleware/request_context.rb

@@ -19,6 +19,14 @@ module ForgeOpsTracker
     # Rescues Exception, not StandardError, and always re-raises: this only ever observes an
     # exception on its way out, it never handles one, so there's no reason to let anything slip
     # past without its context attached.
+    #
+    # Outside Rails it is also what reports an unhandled request error, since there's no
+    # Rails.error to do it: add it to a plain Rack or Sinatra app with
+    # `use ForgeOpsTracker::Middleware::RequestContext`. It reports a StandardError escaping the
+    # app, and a Sinatra error the app turned into a 5xx response itself (Sinatra keeps that in
+    # env["sinatra.error"] rather than raising it when raise_errors is off, its production
+    # default). report_exceptions defaults to true only when Rails isn't loaded, and the Railtie
+    # passes false, so nothing is reported twice under Rails.
     class RequestContext
       THREAD_KEY = :forge_ops_tracker_request_context
 
@@ -28,9 +36,13 @@ module ForgeOpsTracker
       # app raised them.
       SNAPSHOT_IVAR = :@__forge_ops_tracker_request_snapshot
 
-      def initialize(app, configuration: ForgeOpsTracker.configuration)
+      FRAMEWORK_ERROR_KEY = "sinatra.error".freeze
+
+      def initialize(app, configuration: ForgeOpsTracker.configuration, report_exceptions: !defined?(::Rails::Railtie))
         @app = app
         @configuration = configuration
+        @report_exceptions = report_exceptions
+        ForgeOpsTracker.warn_if_environment_disabled if report_exceptions && configuration.equal?(ForgeOpsTracker.configuration)
       end
 
       def call(env)
@@ -38,11 +50,14 @@ module ForgeOpsTracker
 
         state = RequestState.for(env)
         Thread.current[THREAD_KEY] = state
-        app.call(env)
+        response = app.call(env)
+        report_framework_error(env, response) if report_exceptions
+        response
       rescue Exception => e
         if state
           state.mark_errored!
           self.class.attach_snapshot(e, state)
+          report(e, env) if report_exceptions && e.is_a?(StandardError)
         end
         raise
       ensure
@@ -84,7 +99,23 @@ module ForgeOpsTracker
       end
 
       private
-        attr_reader :app, :configuration
+        attr_reader :app, :configuration, :report_exceptions
+
+        # Called while this request's RequestState and the other middlewares' thread-locals are
+        # all still set, so the report carries the trace id, user and breadcrumbs.
+        def report(error, env)
+          ForgeOpsTracker.capture_exception(error, context: { "path" => env["PATH_INFO"], "method" => env["REQUEST_METHOD"] }.compact)
+        rescue StandardError
+          nil
+        end
+
+        def report_framework_error(env, response)
+          error = env[FRAMEWORK_ERROR_KEY]
+          return unless error.is_a?(StandardError) && response.is_a?(Array) && response.first.to_i >= 500
+
+          RequestContext.current&.mark_errored!
+          report(error, env)
+        end
     end
   end
 end

data/lib/forge_ops_tracker/performance_flusher.rb

@@ -17,6 +17,7 @@ module ForgeOpsTracker
       @buckets = {}
       @period_started_at = Time.now.utc
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "performance samples")
 
       # Flushes whatever's already tallied on a normal process exit, so the last partial window
       # (anything shorter than a full performance_flush_interval) isn't silently dropped; same
@@ -48,6 +49,9 @@ module ForgeOpsTracker
     # just grows instead of losing what was already tallied; same reasoning SessionFlusher#flush
     # already documents, and the same reason PerformanceSamplesController accepts a batch rather
     # than a single row: a failed flush shouldn't have to re-deliver by transaction one at a time.
+    # That only holds for a delivery that failed for now, and only FlushOutcome::MAX_FAILED_FLUSHES
+    # times in a row: a batch the server rejected, or one that keeps failing, is removed exactly
+    # the way a delivered one is, since re-sending it could never succeed.
     #
     # Only exactly what this snapshot delivered is removed afterward (subtracted from whatever is in
     # each bucket by then), never the whole hash reset: #record can run on another thread while the
@@ -84,7 +88,7 @@ module ForgeOpsTracker
         end
       end
 
-      return unless snapshot && client.deliver_performance_samples(snapshot)
+      return unless snapshot && @outcome.release?(client.deliver_performance_samples(snapshot))
 
       @mutex.synchronize do
         sent.each do |key, delivered|

data/lib/forge_ops_tracker/query_explainer.rb

@@ -135,10 +135,11 @@ module ForgeOpsTracker
         return false
       end
 
-      client.deliver_query_plan(
+      result = client.deliver_query_plan(
         statement: job.statement, db_system: "postgresql", plan: masked_plan,
         duration_ms: job.duration_ms, environment: configuration.environment, captured_at: job.captured_at
       )
+      DeliveryStatus.from(result) == DeliveryStatus::DELIVERED
     rescue StandardError => e
       log { "EXPLAIN failed: #{e.class}: #{e.message}" }
       false

data/lib/forge_ops_tracker/railtie.rb

@@ -38,10 +38,19 @@ module ForgeOpsTracker
     initializer "forge_ops_tracker.subscribe_error_reporter" do |app|
       configuration = ForgeOpsTracker.configuration
       configuration.app_root ||= app.root.to_s
-      configuration.environment = Rails.env.to_s
+      # FORGE_OPS_ENVIRONMENT wins when set, so a staging box running RAILS_ENV=production can
+      # still report as staging; otherwise Rails.env, so development and test stay quiet.
+      forge_ops_environment = ENV["FORGE_OPS_ENVIRONMENT"].to_s.strip
+      configuration.environment = forge_ops_environment.empty? ? Rails.env.to_s : forge_ops_environment
       configuration.logger ||= Rails.logger
 
-      Rails.error.subscribe(ForgeOpsTracker::ErrorSubscriber.new(configuration))
+      Rails.error.subscribe(ForgeOpsTracker.error_subscriber)
+    end
+
+    # After every config/initializers file has run, so the DSN and environment are final: one
+    # warning at boot when a DSN is set but this environment doesn't send (development, say).
+    config.after_initialize do
+      ForgeOpsTracker.warn_if_environment_disabled
     end
 
     initializer "forge_ops_tracker.track_sessions" do |app|
@@ -94,7 +103,8 @@ module ForgeOpsTracker
     # PerformanceInstrumentation.controller_endpoint for what each is used for).
     initializer "forge_ops_tracker.request_context" do |app|
       configuration = ForgeOpsTracker.configuration
-      app.middleware.use ForgeOpsTracker::Middleware::RequestContext, configuration: configuration
+      # report_exceptions: false because Rails.error already reports an unhandled request error.
+      app.middleware.use ForgeOpsTracker::Middleware::RequestContext, configuration: configuration, report_exceptions: false
 
       ActiveSupport::Notifications.subscribe("start_processing.action_controller") do |*args|
         next unless configuration.enabled?

data/lib/forge_ops_tracker/session_flusher.rb

@@ -16,6 +16,7 @@ module ForgeOpsTracker
       @crashed_sessions_count = 0
       @period_started_at = Time.now.utc
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "session counts")
 
       # Flushes whatever's already tallied on a normal process exit, so the last partial window
       # (anything shorter than a full session_flush_interval) isn't silently dropped.
@@ -34,6 +35,8 @@ module ForgeOpsTracker
     # Snapshots and resets the in-process counters, then delivers them. A failed delivery keeps
     # the counts where they are rather than resetting, so the next flush's window just grows
     # instead of losing what was already tallied; there's no other copy of this data anywhere.
+    # Counts the server rejected, or that failed FlushOutcome::MAX_FAILED_FLUSHES flushes in a
+    # row, are reset all the same: sending them again could never succeed.
     def flush
       snapshot = nil
 
@@ -50,7 +53,7 @@ module ForgeOpsTracker
         }
       end
 
-      return unless snapshot && client.deliver_session_checkin(snapshot)
+      return unless snapshot && @outcome.release?(client.deliver_session_checkin(snapshot))
 
       @mutex.synchronize do
         @sessions_count = 0

data/lib/forge_ops_tracker/version.rb

@@ -1,3 +1,3 @@
 module ForgeOpsTracker
-  VERSION = "0.15.1"
+  VERSION = "0.17.0"
 end

data/lib/forge_ops_tracker.rb

@@ -4,6 +4,7 @@ require "forge_ops_tracker/configuration"
 require "forge_ops_tracker/pii_scrubber"
 require "forge_ops_tracker/sql_statement"
 require "forge_ops_tracker/event_builder"
+require "forge_ops_tracker/delivery_status"
 require "forge_ops_tracker/client"
 require "forge_ops_tracker/delivery_queue"
 require "forge_ops_tracker/change"
@@ -30,6 +31,8 @@ require "forge_ops_tracker/middleware/span_tracing"
 require "forge_ops_tracker/middleware/request_context"
 
 module ForgeOpsTracker
+  ENVIRONMENT_WARNING_LOCK = Mutex.new
+
   class << self
     def configuration
       @configuration ||= Configuration.new
@@ -37,10 +40,54 @@ module ForgeOpsTracker
 
     # Outside Rails this is also where the startup ChangeSnapshot goes out, since there's no
     # Railtie to send it once the app has booted; under Rails the Railtie sends it instead, after
-    # initialization, when Active Record can report the migration version.
+    # initialization, when Active Record can report the migration version. The same goes for the
+    # one-time "Not sending" warning (see warn_if_environment_disabled).
     def configure
       yield configuration
-      ChangeSnapshot.start(configuration) unless defined?(::Rails::Railtie)
+      return if defined?(::Rails::Railtie)
+
+      warn_if_environment_disabled
+      ChangeSnapshot.start(configuration)
+    end
+
+    # Reports an exception you've already rescued, e.g. from a Sinatra error handler, a plain Rack
+    # app, a script, or a background worker that isn't wired through Rails.error. Goes through the
+    # same ErrorSubscriber Rails.error reports into, so it carries the current request's user,
+    # breadcrumbs and trace the same way, and is a no-op when the gem isn't enabled. Under Rails,
+    # Rails.error.report/handle/record do the same thing. Never raises.
+    def capture_exception(error, context: {})
+      warn_if_environment_disabled unless configuration.enabled?
+      error_subscriber.report(error, handled: true, context: context || {}, source: "application")
+    end
+
+    # The one ErrorSubscriber this process reports through: the Railtie subscribes it to
+    # Rails.error, and capture_exception calls it directly, so both share one delivery queue.
+    def error_subscriber
+      @error_subscriber ||= ErrorSubscriber.new(configuration)
+    end
+
+    # Logs one warning per process when a DSN is set but the environment isn't in
+    # enabled_environments, so a process that would send nothing says so instead of staying
+    # silent. Goes to configuration.logger (Rails.logger under Rails), or $stderr when there isn't
+    # one. Never logs without a DSN. Returns whether it logged.
+    def warn_if_environment_disabled
+      message = configuration.disabled_environment_warning
+      return false unless message
+
+      ENVIRONMENT_WARNING_LOCK.synchronize do
+        return false if @environment_warning_logged
+
+        @environment_warning_logged = true
+      end
+
+      if configuration.logger
+        configuration.logger.warn(message)
+      else
+        Kernel.warn(message)
+      end
+      true
+    rescue StandardError
+      false
     end
 
     # Records one thing that changed in a running system, e.g.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forge_ops_tracker
 version: !ruby/object:Gem::Version
-  version: 0.15.1
+  version: 0.17.0
 platform: ruby
 authors:
 - ForgeOps
@@ -79,6 +79,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.30'
+- !ruby/object:Gem::Dependency
+  name: rack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.2'
 description: Hooks Rails' error reporter and reports exceptions to ForgeOps over HTTP,
   without ever raising back into the host application.
 executables: []
@@ -95,6 +109,7 @@ files:
 - lib/forge_ops_tracker/client.rb
 - lib/forge_ops_tracker/configuration.rb
 - lib/forge_ops_tracker/delivery_queue.rb
+- lib/forge_ops_tracker/delivery_status.rb
 - lib/forge_ops_tracker/error_subscriber.rb
 - lib/forge_ops_tracker/event_builder.rb
 - lib/forge_ops_tracker/failure_event_buffer.rb