forge_ops_tracker 0.16.0 → 0.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a535fb7cb193fd614ee40259a9b20e62832149c7a2dbeb7d1d40253c0630f65a
-  data.tar.gz: 559a19cd52a56addacc87c5533bc0954c7ca2de6eac81a6afb18769f280c06c5
+  metadata.gz: e9434056bed332dad65c4a8dc051bace54e7acae535043138833f60ba2586cb0
+  data.tar.gz: 270cec5a96267fc7d1a41be7c02ab1bfdb9271fb2d7a5e8ae91581a0181848d0
 SHA512:
-  metadata.gz: 158342795c1dbb5e7575da43ddcd239c25b52111c268ede3fdfdbb9b5621cf1a10cd120e602f75fe2e57bd46faed9483aaa1a61f8c9f91770474210414400e8f
-  data.tar.gz: 5b4f11f1758833c58f8925f1354f8a7da49b80d04344cc2c340a2662a186fcfc0a8451709faef4c78a60d618c9f0ea936c460c97b3af8bad2687fbc8f6fbfdae
+  metadata.gz: 34730c4a7e8af87adb9bdaf490d72d923a0bb96b958a9131c28671d3198a43638dc5f13a2331c92b3d1dd506d24b08b6118d6d0559e4c64d5b12b231a0a8c97c
+  data.tar.gz: 0a9704a43c0c74397e082ab191deab1895de0d17210469b5906c123c33cfd7fe9130992ee10932106a093c4d6c329dadca6ca59da16f46173a063395e0eb7b9c

data/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 0.18.0 (2026-10-20)
+
+- New `config.strict_privacy` (off by default, or `FORGE_OPS_STRICT_PRIVACY=true`) for apps that
+  can't let personal or health data leave their servers. With it on, an error event carries no
+  message (a `message_fingerprint` computed in your process is sent instead, so errors still
+  group), no breadcrumbs, no `endpoint`, no SQL statement and no source context, and the user only
+  by `id`. Failed jobs and outbound calls keep their error class but not their message, and
+  outbound HTTP spans are named by method and host only, with no span data but `db.statement` and
+  `db.system`. A `transaction_name` that is a raw path, not a route or `Controller#action`, has its
+  ids taken out in error events, performance samples and failure events.
+- New `config.privacy_allowed_fields` (or `FORGE_OPS_PRIVACY_ALLOWED_FIELDS=error_code,feature`):
+  the context and tag keys strict privacy still sends, when their value is a string, number or
+  boolean. Empty by default.
+
+## 0.17.0 (2026-10-13)
+
+- Fixed: a batch the server refused was re-sent at every flush for as long as the process lived,
+  with everything recorded since added to it. Performance samples, custom and infrastructure
+  metrics, session counts and failure events are now dropped when the server answers with a 4xx
+  other than 408 or 429 (a 400, 401, 403, 413 or 422, for example), since sending the same batch
+  again can only fail the same way. Once a metric buffer had filled up behind such a batch, every
+  new metric was dropped too; that no longer happens.
+- A batch that could not be delivered for now (a network error, a timeout, a 5xx, a 408 or a 429)
+  is still kept for the next flush, but only for 5 flushes in a row. After the fifth it is dropped
+  and the buffer starts clean, so a long outage no longer holds the same data in memory forever.
+- The failure event buffer now holds at most 1000 entries; further ones are dropped until a flush
+  lets go of them, the same way the metric buffers already behave.
+- With a `config.logger`, each drop is logged once at debug with what was dropped and why, and a
+  non-2xx answer is logged with its status code.
+- Internal: `ForgeOpsTracker::Client#deliver` and its sibling `deliver_*` methods now return
+  `:delivered`, `:rejected` or `:failed` instead of `true`/`false`. All three are truthy, so code
+  calling `Client` directly should compare against `:delivered`. Nothing in the documented API
+  changed.
+
 ## 0.16.0 (2026-10-02)
 
 - Changed: outside Rails, the environment now defaults to `production` when nothing sets it (it

data/LICENSE.txt

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2026 ForgeOps
+Copyright (c) 2026 Luke Popwell
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -226,6 +226,42 @@ ForgeOpsTracker.configure do |config|
 end
 ```
 
+## Strict privacy
+
+For apps that can't let personal or health data (PII, PHI) leave their own servers. With strict
+privacy on, the gem sends only structured facts about an error (its class, its stack frames'
+file, line and method, the release, environment, server, the request's `transaction_name` and
+trace id, and the database tables or procedures involved), never free text that could hold a
+name, an email or a diagnosis:
+
+- No error message. A `message_fingerprint` is sent instead: a one-way hash of the message with
+  its numbers, hex ids and UUIDs taken out, computed in your process, so errors still group
+  without the message ever leaving it.
+- No breadcrumbs (they aren't recorded at all), no `endpoint`, no SQL statement even with
+  `capture_sql_statement` on, and no source context (the source file is never read).
+- The user is sent by `id` only, never `email` or `username`.
+- Context and tags only keep the top-level keys you list in `privacy_allowed_fields`, and only
+  when the value is a string, number or boolean. Nothing else from them is sent.
+- Failed jobs and outbound calls keep their error class, never their error message.
+- A `transaction_name` that is a raw path rather than a route or `Controller#action` has its ids
+  taken out (`GET /patients/:id`), in error events, performance samples and failure events alike.
+- Outbound HTTP spans are named by method and host only (`GET api.example.com`), and span data
+  keeps only the masked `db.statement` and `db.system`.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.strict_privacy = true                           # default false
+  config.privacy_allowed_fields = %w[error_code feature] # default []
+end
+```
+
+Both can also come from the environment: `FORGE_OPS_STRICT_PRIVACY=true` and
+`FORGE_OPS_PRIVACY_ALLOWED_FIELDS=error_code,feature` (comma separated). A value set in
+`configure` wins over the environment.
+
+ForgeOps applies the same rules again on arrival when the project's own Strict privacy setting is
+on, so an app with an older gem, or one with this option off, still can't get that text stored.
+
 ## Source context
 
 By default, each in-app backtrace frame (never a vendored gem) is captured along with the 5 lines

data/lib/forge_ops_tracker/breadcrumb_buffer.rb

@@ -23,7 +23,8 @@ module ForgeOpsTracker
       # ForgeOpsTracker.configure takes effect on whatever's added next, not just a buffer created
       # afterward.
       max_size = [ configuration.max_breadcrumbs, 0 ].max
-      return if max_size.zero?
+      # Strict privacy never sends breadcrumbs, so they're never recorded either.
+      return if max_size.zero? || configuration.strict_privacy
 
       entries << {
         category: category.to_s,

data/lib/forge_ops_tracker/change_snapshot.rb

@@ -66,7 +66,7 @@ module ForgeOpsTracker
     end
 
     def deliver
-      client.deliver_change_snapshot(payload)
+      DeliveryStatus.from(client.deliver_change_snapshot(payload)) == DeliveryStatus::DELIVERED
     rescue StandardError => e
       configuration.logger&.debug { "[ForgeOpsTracker] change snapshot failed: #{e.class}: #{e.message}" }
       false

data/lib/forge_ops_tracker/client.rb

@@ -5,8 +5,10 @@ require "uri"
 module ForgeOpsTracker
   # Delivers one payload over HTTP. Every failure mode; DNS, connection,
   # timeout, TLS, a non-2xx response; is caught here and turned into a
-  # `false` return rather than a raised exception, since a broken or
-  # unreachable tracker must never be able to break the host app.
+  # DeliveryStatus rather than a raised exception, since a broken or
+  # unreachable tracker must never be able to break the host app. Every
+  # deliver_* method answers one of DeliveryStatus::DELIVERED, REJECTED or
+  # FAILED (all three are truthy, so compare, never test for truth).
   class Client
     def initialize(configuration)
       @configuration = configuration
@@ -16,8 +18,8 @@ module ForgeOpsTracker
       post(configuration.ingestion_uri, payload)
     end
 
-    # Same delivery contract as deliver above (never raises, returns a bare
-    # true/false), just a different endpoint under the same DSN's host and
+    # Same delivery contract as deliver above (never raises, returns a
+    # DeliveryStatus), just a different endpoint under the same DSN's host and
     # api_key; see Configuration#session_checkins_uri.
     def deliver_session_checkin(payload)
       post(configuration.session_checkins_uri, payload)
@@ -61,7 +63,7 @@ module ForgeOpsTracker
     end
 
     # Same delivery contract again; see Configuration#changes_uri and ForgeOpsTracker.record_change.
-    # One change per call. A plan without change tracking answers 403, which is just a false here.
+    # One change per call. A plan without change tracking answers 403, which is a REJECTED here.
     def deliver_change(payload)
       post(configuration.changes_uri, payload)
     end
@@ -72,7 +74,7 @@ module ForgeOpsTracker
     end
 
     # Same again; see Configuration#query_plans_uri and QueryExplainer. One plan per call. A plan
-    # without performance monitoring answers 403, which is just a false here.
+    # without performance monitoring answers 403, which is a REJECTED here.
     def deliver_query_plan(payload)
       post(configuration.query_plans_uri, payload)
     end
@@ -81,13 +83,15 @@ module ForgeOpsTracker
       attr_reader :configuration
 
       def post(uri, payload)
-        return false unless uri
+        return DeliveryStatus::FAILED unless uri
 
         response = http_for(uri).request(build_request(uri, payload))
-        response.is_a?(Net::HTTPSuccess)
+        status = DeliveryStatus.for_code(response.code)
+        log { "delivery to #{uri.path} answered HTTP #{response.code}" } unless status == DeliveryStatus::DELIVERED
+        status
       rescue StandardError => e
         log { "delivery failed: #{e.class}: #{e.message}" }
-        false
+        DeliveryStatus::FAILED
       end
 
       def http_for(uri)

data/lib/forge_ops_tracker/configuration.rb

@@ -20,6 +20,8 @@ module ForgeOpsTracker
     attr_accessor :explain_slow_queries, :explain_threshold_ms
     attr_accessor :propagate_traces, :trace_propagation_targets
     attr_accessor :detect_changes, :track_env_var_names
+    attr_accessor :strict_privacy
+    attr_reader :privacy_allowed_fields
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -156,6 +158,24 @@ module ForgeOpsTracker
       # but even names can describe more about an app than some teams want to share. See
       # ChangeSnapshot::ENV_VAR_DENYLIST for the host-specific names that are always left out.
       @track_env_var_names = false
+      # Off by default. On, nothing this gem sends carries free text that could hold personal or
+      # health data: no error message (a message_fingerprint is sent instead, so errors still
+      # group), no breadcrumbs, no request path, no SQL statement, no source context, no user email
+      # or username, no job failure message, and outbound HTTP spans named by method and host only.
+      # See StrictPrivacy for every rule. FORGE_OPS_STRICT_PRIVACY=true turns it on from the
+      # environment; setting it here wins over that.
+      @strict_privacy = self.class.env_flag(ENV["FORGE_OPS_STRICT_PRIVACY"])
+      # Context and tag keys strict privacy still sends (only plain string, number or boolean
+      # values under exactly these top-level keys), e.g. %w[error_code feature]. Empty by default,
+      # so nothing from context or tags goes out. FORGE_OPS_PRIVACY_ALLOWED_FIELDS=error_code,feature
+      # sets it from the environment.
+      self.privacy_allowed_fields = ENV["FORGE_OPS_PRIVACY_ALLOWED_FIELDS"].to_s.split(",")
+    end
+
+    # Accepts an Array of names or one comma-separated String; always kept as stripped Strings.
+    def privacy_allowed_fields=(fields)
+      fields = fields.split(",") if fields.is_a?(String)
+      @privacy_allowed_fields = Array(fields).map { |field| field.to_s.strip }.reject(&:empty?)
     end
 
     # Whether an outbound request to host should carry a traceparent header; see
@@ -290,6 +310,11 @@ module ForgeOpsTracker
       [ env["FORGE_OPS_ENVIRONMENT"], env["RAILS_ENV"], env["RACK_ENV"] ].find { |value| value && !value.strip.empty? } || "production"
     end
 
+    # "true", "1", "yes" or "on" (any case) is on; anything else, or unset, is off.
+    def self.env_flag(value)
+      %w[true 1 yes on].include?(value.to_s.strip.downcase)
+    end
+
     private
       def environment_enabled?
         enabled_environments.map(&:to_s).include?(environment.to_s)

data/lib/forge_ops_tracker/delivery_queue.rb

@@ -39,7 +39,7 @@ module ForgeOpsTracker
         rescue ThreadError
           break
         end
-        client.public_send(@deliver_with, payload)
+        deliver(payload)
       end
       sleep 0.01 while @in_flight.positive? && monotonic_now < deadline
     rescue StandardError => e
@@ -78,7 +78,7 @@ module ForgeOpsTracker
           payload = @queue.pop
           @in_flight += 1
           begin
-            client.public_send(@deliver_with, payload)
+            deliver(payload)
           ensure
             @in_flight -= 1
           end
@@ -87,6 +87,17 @@ module ForgeOpsTracker
         end
       end
 
+      # One attempt per payload, as it has always been: a queued event is never held back for a
+      # retry, whatever the outcome, so all that is left to do with a refusal is to say so once.
+      def deliver(payload)
+        case DeliveryStatus.from(client.public_send(@deliver_with, payload))
+        when DeliveryStatus::REJECTED
+          log { "dropping event: the server rejected it and will not accept a retry" }
+        when DeliveryStatus::FAILED
+          log { "dropping event: delivery failed" }
+        end
+      end
+
       # Registered on the first push, not at load, so a process that never reports anything never
       # waits on exit at all.
       def register_drain

data/lib/forge_ops_tracker/delivery_status.rb

@@ -0,0 +1,74 @@
+module ForgeOpsTracker
+  # What became of one delivery attempt, as every Client#deliver_* method reports it. A bare
+  # true/false could not tell "the server will never take this payload" apart from "try again
+  # later", so a batch a plan or a validation rule refused was re-sent at every flush for as long
+  # as the process lived.
+  module DeliveryStatus
+    # Any 2xx.
+    DELIVERED = :delivered
+    # A 4xx other than 408 and 429: sending the same payload again can only fail the same way.
+    REJECTED = :rejected
+    # A network error, a timeout, a 5xx, a 408 or a 429: worth another attempt later.
+    FAILED = :failed
+
+    RETRYABLE_CLIENT_ERRORS = [ 408, 429 ].freeze
+
+    def self.for_code(code)
+      code = code.to_i
+      return DELIVERED if code.between?(200, 299)
+      return REJECTED if code.between?(400, 499) && !RETRYABLE_CLIENT_ERRORS.include?(code)
+
+      FAILED
+    end
+
+    # Reads whatever a client handed back. A status passes through; anything else is read the way
+    # the old boolean contract was (truthy delivered, falsy failed for now), so a client injected
+    # through a `client:` keyword that still answers true/false keeps working.
+    def self.from(result)
+      return result if [ DELIVERED, REJECTED, FAILED ].include?(result)
+
+      result ? DELIVERED : FAILED
+    end
+  end
+
+  # Decides, after each flush of one buffer/flusher, whether the batch it just sent is done with.
+  # Delivered and rejected batches are let go of at once; one that failed for now is kept, but only
+  # for MAX_FAILED_FLUSHES flushes in a row, so an outage (or a server answering 500 to one
+  # particular batch) can never pin the same data in memory for the life of the process.
+  class FlushOutcome
+    MAX_FAILED_FLUSHES = 5
+
+    # what names the batch in the debug log ("metrics", "performance samples").
+    def initialize(configuration, what)
+      @configuration = configuration
+      @what = what
+      @failed_flushes = 0
+    end
+
+    # True when the caller should remove what it just sent (delivered, or dropped), false when it
+    # should keep it for the next flush.
+    def release?(result)
+      case DeliveryStatus.from(result)
+      when DeliveryStatus::DELIVERED
+        @failed_flushes = 0
+        true
+      when DeliveryStatus::REJECTED
+        @failed_flushes = 0
+        log { "dropping buffered #{@what}: the server rejected them and will not accept a retry" }
+        true
+      else
+        @failed_flushes += 1
+        return false if @failed_flushes < MAX_FAILED_FLUSHES
+
+        @failed_flushes = 0
+        log { "dropping buffered #{@what}: #{MAX_FAILED_FLUSHES} flushes in a row failed" }
+        true
+      end
+    end
+
+    private
+      def log
+        @configuration.logger&.debug { "[ForgeOpsTracker] #{yield}" }
+      end
+  end
+end

data/lib/forge_ops_tracker/event_builder.rb

@@ -35,6 +35,12 @@ module ForgeOpsTracker
     # request (a background job, a console session) and each key is left out of the payload
     # entirely when nil, the same "omit rather than send an empty value" treatment user and
     # breadcrumbs already get.
+    #
+    # With configuration.strict_privacy on, the finished payload goes through StrictPrivacy.event:
+    # the message is replaced by its message_fingerprint (computed from the message as it would
+    # otherwise have been sent, so after scrub_pii, which is also what the server fingerprints from
+    # a gem without strict privacy, and keeps a message holding an email grouping together), and breadcrumbs, endpoint, the SQL statement, source context, the user's
+    # email/username and every context/tag key not in privacy_allowed_fields are left out.
     def build(error, context: {}, user: nil, breadcrumbs: [], transaction_name: nil, endpoint: nil, trace_id: nil)
       payload = {
         exception_class: error.class.name,
@@ -54,7 +60,10 @@ module ForgeOpsTracker
       payload[:endpoint] = endpoint if endpoint
       payload[:trace_id] = trace_id if trace_id
       attach_sql(payload, error)
-      scrub(payload)
+      payload = scrub(payload)
+      return payload unless configuration.strict_privacy
+
+      StrictPrivacy.event(payload, allowed_fields: configuration.privacy_allowed_fields)
     end
 
     private
@@ -96,7 +105,7 @@ module ForgeOpsTracker
 
         objects = SqlStatement.objects(masked)
         payload[:sql_objects] = objects if objects && configuration.capture_sql_objects
-        payload[:sql_statement] = masked if configuration.capture_sql_statement
+        payload[:sql_statement] = masked if configuration.capture_sql_statement && !configuration.strict_privacy
       end
 
       def backtrace_frames(error)
@@ -135,9 +144,10 @@ module ForgeOpsTracker
       # to turn it off). Best-effort: any file that can't be read (deleted,
       # permission denied, a path that only ever existed inside a build
       # step and isn't present in this deployment) just means this one
-      # frame gets no source context, not a raised error of its own.
+      # frame gets no source context, not a raised error of its own. Never read at all under
+      # strict privacy, which never sends it.
       def attach_source_context(frame)
-        return frame unless configuration.capture_source_context && frame[:in_app]
+        return frame unless configuration.capture_source_context && frame[:in_app] && !configuration.strict_privacy
 
         lines = File.readlines(frame[:file])
         index = frame[:line] - 1

data/lib/forge_ops_tracker/failure_event_buffer.rb

@@ -8,12 +8,17 @@ module ForgeOpsTracker
   # kind of thing worth an accurate individual record of, not something to pre-aggregate away the
   # way PerformanceFlusher's own count/sum/max buckets duration.
   class FailureEventBuffer
+    # Same cap, and the same "drop the newest, not the oldest" reasoning, as MetricBuffer's own
+    # MAX_ENTRIES: a burst of failures during an outage must not grow this without bound.
+    MAX_ENTRIES = 1000
+
     def initialize(configuration, client: Client.new(configuration))
       @configuration = configuration
       @client = client
       @mutex = Mutex.new
       @entries = []
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "failure events")
 
       # Flushes whatever's already buffered on a normal process exit, so the last partial window
       # isn't silently dropped; same reasoning every other buffer/flusher in this gem documents.
@@ -24,27 +29,35 @@ module ForgeOpsTracker
       ensure_worker_started
 
       @mutex.synchronize do
-        @entries << {
+        return false if @entries.size >= MAX_ENTRIES
+
+        entry = {
           kind: kind, transaction_name: transaction_name, error_class: error_class, error_message: error_message,
           environment: configuration.environment.to_s, release: configuration.release,
           occurred_at: Time.now.utc.iso8601
         }
+        # Strict privacy keeps the error's class, never its message.
+        @entries << (configuration.strict_privacy ? StrictPrivacy.failure_event(entry) : entry)
       end
     end
 
     # Snapshots and resets the buffered entries, then delivers them as one batch. Same "a failed
-    # delivery keeps every entry for the next attempt" contract as MetricBuffer#flush.
+    # delivery keeps every entry for the next attempt" contract as MetricBuffer#flush, with the
+    # same two exceptions: a rejected batch, and one that failed FlushOutcome::MAX_FAILED_FLUSHES
+    # flushes in a row, are dropped.
     def flush
       snapshot = nil
 
       @mutex.synchronize do
         return if @entries.empty?
-        snapshot = @entries
+        snapshot = @entries.dup
       end
 
-      return unless client.deliver_failure_events(snapshot)
+      return unless @outcome.release?(client.deliver_failure_events(snapshot))
 
-      @mutex.synchronize { @entries = [] }
+      # Exactly the entries just sent: anything recorded while the request was in flight sits after
+      # them and stays for the next flush.
+      @mutex.synchronize { @entries.shift(snapshot.size) }
     end
 
     private

data/lib/forge_ops_tracker/infrastructure_metric_buffer.rb

@@ -3,16 +3,16 @@ require "thread"
 module ForgeOpsTracker
   # Collects individual ForgeOpsTracker.capture_infrastructure_metric readings in-process and
   # periodically flushes them as one batch, same shape as MetricBuffer (a sibling class, not a
-  # shared base -- this gem keeps SessionFlusher/PerformanceFlusher as separate, independently-
+  # shared base; this gem keeps SessionFlusher/PerformanceFlusher as separate, independently-
   # readable classes despite their own similar shape too, and these two follow that precedent).
   # Its main real caller is a short-lived cron/script process that exits right after a handful of
   # capture calls, so the at_exit flush below is what actually matters in practice; the periodic
   # background-thread timer still runs for a long-lived process that chooses to call this
   # continuously instead, but rarely gets the chance to fire in the cron case.
   class InfrastructureMetricBuffer
-    # Once this many entries are buffered, further ones are dropped until a flush succeeds: a plan
-    # without the feature answers 403 on every flush, and an uncapped buffer would then grow for as
-    # long as the process lives. Dropping the newest rather than the oldest keeps the entries a flush
+    # Once this many entries are buffered, further ones are dropped until a flush lets go of them:
+    # an unreachable server fails every flush, and an uncapped buffer would keep growing until
+    # FlushOutcome gives up on the batch. Dropping the newest rather than the oldest keeps the entries a flush
     # is delivering at the front of the array, which is what makes removing exactly them exact.
     MAX_ENTRIES = 1000
 
@@ -22,6 +22,7 @@ module ForgeOpsTracker
       @mutex = Mutex.new
       @entries = []
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "infrastructure metrics")
 
       at_exit { flush }
     end
@@ -38,7 +39,7 @@ module ForgeOpsTracker
       end
     end
 
-    # Same reset-on-success/keep-on-failure contract as MetricBuffer#flush; see that class's own
+    # Same reset-on-success/keep-on-failure/drop-on-rejection contract as MetricBuffer#flush; see that class's own
     # comment for the accepted race this shares with it.
     def flush
       snapshot = nil
@@ -48,9 +49,9 @@ module ForgeOpsTracker
         snapshot = @entries.dup
       end
 
-      return unless client.deliver_infrastructure_metrics(snapshot)
+      return unless @outcome.release?(client.deliver_infrastructure_metrics(snapshot))
 
-      # Exactly the entries just delivered: anything recorded while the request was in flight sits
+      # Exactly the entries just sent: anything recorded while the request was in flight sits
       # after them and stays for the next flush (resetting the whole array here lost it).
       @mutex.synchronize { @entries.shift(snapshot.size) }
     end

data/lib/forge_ops_tracker/metric_buffer.rb

@@ -10,9 +10,9 @@ module ForgeOpsTracker
   # they'll want a genuinely accurate count/sum of later, not something to pre-aggregate away
   # client-side, so CustomMetricsController stores one row per entry as-is.
   class MetricBuffer
-    # Once this many entries are buffered, further ones are dropped until a flush succeeds: a plan
-    # without the feature answers 403 on every flush, and an uncapped buffer would then grow for as
-    # long as the process lives. Dropping the newest rather than the oldest keeps the entries a flush
+    # Once this many entries are buffered, further ones are dropped until a flush lets go of them:
+    # an unreachable server fails every flush, and an uncapped buffer would keep growing until
+    # FlushOutcome gives up on the batch. Dropping the newest rather than the oldest keeps the entries a flush
     # is delivering at the front of the array, which is what makes removing exactly them exact.
     MAX_ENTRIES = 1000
 
@@ -22,6 +22,7 @@ module ForgeOpsTracker
       @mutex = Mutex.new
       @entries = []
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "metrics")
 
       # Flushes whatever's already buffered on a normal process exit, so the last partial window
       # isn't silently dropped; same reasoning every other buffer/flusher in this gem documents.
@@ -44,10 +45,11 @@ module ForgeOpsTracker
       end
     end
 
-    # Snapshots and resets the buffered entries, then delivers them as one batch. A failed
-    # delivery keeps every entry where it is rather than resetting, so the next flush's batch
-    # just grows instead of losing what was already buffered; same reasoning SessionFlusher#flush
-    # already documents, including the same small, accepted race (an entry recorded during the
+    # Snapshots and resets the buffered entries, then delivers them as one batch. A delivery that
+    # failed for now keeps every entry where it is rather than resetting, so the next flush's batch
+    # just grows instead of losing what was already buffered (a rejected batch, or one that failed
+    # FlushOutcome::MAX_FAILED_FLUSHES flushes in a row, is dropped instead); same reasoning
+    # SessionFlusher#flush already documents, including the same small, accepted race (an entry recorded during the
     # in-flight HTTP request gets folded into this snapshot's own reset rather than kept for the
     # next one) that class already lives with, for the same reason: simple and consistent beats a
     # cleverer dedup that would only trade one rare edge case for a subtler one.
@@ -59,9 +61,9 @@ module ForgeOpsTracker
         snapshot = @entries.dup
       end
 
-      return unless client.deliver_metrics(snapshot)
+      return unless @outcome.release?(client.deliver_metrics(snapshot))
 
-      # Exactly the entries just delivered: anything recorded while the request was in flight sits
+      # Exactly the entries just sent: anything recorded while the request was in flight sits
       # after them and stays for the next flush (resetting the whole array here lost it).
       @mutex.synchronize { @entries.shift(snapshot.size) }
     end

data/lib/forge_ops_tracker/middleware/span_tracing.rb

@@ -41,7 +41,7 @@ module ForgeOpsTracker
         app.call(env)
       ensure
         if buffer && send_trace?(buffer, state)
-          Railtie.span_queue.push(trace_id: buffer.trace_id, spans: buffer.spans)
+          Railtie.span_queue.push(trace_id: buffer.trace_id, spans: outgoing_spans(buffer))
         end
         Thread.current[:forge_ops_tracker_spans] = nil
       end
@@ -54,6 +54,14 @@ module ForgeOpsTracker
 
           state.errored? && buffer.root_recorded?
         end
+
+        # Under strict privacy, every span goes through StrictPrivacy.span: an outbound call named
+        # by its method and host only, and no span data but the masked statement and db.system.
+        def outgoing_spans(buffer)
+          return buffer.spans unless configuration.strict_privacy
+
+          buffer.spans.map { |span| StrictPrivacy.span(span) }
+        end
     end
   end
 end

data/lib/forge_ops_tracker/performance_flusher.rb

@@ -17,6 +17,7 @@ module ForgeOpsTracker
       @buckets = {}
       @period_started_at = Time.now.utc
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "performance samples")
 
       # Flushes whatever's already tallied on a normal process exit, so the last partial window
       # (anything shorter than a full performance_flush_interval) isn't silently dropped; same
@@ -30,6 +31,8 @@ module ForgeOpsTracker
     # without colliding into the same bucket.
     def record(transaction_name:, duration_ms:, kind: "controller")
       ensure_worker_started
+      # Under strict privacy a raw-path name loses its ids; a route or Controller#action is kept.
+      transaction_name = StrictPrivacy.transaction_name(transaction_name) if configuration.strict_privacy
 
       @mutex.synchronize do
         bucket = (@buckets[[ transaction_name, kind ]] ||= { count: 0, duration_sum_ms: 0.0, max_duration_ms: 0.0, histogram: Hash.new(0) })
@@ -48,6 +51,9 @@ module ForgeOpsTracker
     # just grows instead of losing what was already tallied; same reasoning SessionFlusher#flush
     # already documents, and the same reason PerformanceSamplesController accepts a batch rather
     # than a single row: a failed flush shouldn't have to re-deliver by transaction one at a time.
+    # That only holds for a delivery that failed for now, and only FlushOutcome::MAX_FAILED_FLUSHES
+    # times in a row: a batch the server rejected, or one that keeps failing, is removed exactly
+    # the way a delivered one is, since re-sending it could never succeed.
     #
     # Only exactly what this snapshot delivered is removed afterward (subtracted from whatever is in
     # each bucket by then), never the whole hash reset: #record can run on another thread while the
@@ -84,7 +90,7 @@ module ForgeOpsTracker
         end
       end
 
-      return unless snapshot && client.deliver_performance_samples(snapshot)
+      return unless snapshot && @outcome.release?(client.deliver_performance_samples(snapshot))
 
       @mutex.synchronize do
         sent.each do |key, delivered|

data/lib/forge_ops_tracker/query_explainer.rb

@@ -135,10 +135,11 @@ module ForgeOpsTracker
         return false
       end
 
-      client.deliver_query_plan(
+      result = client.deliver_query_plan(
         statement: job.statement, db_system: "postgresql", plan: masked_plan,
         duration_ms: job.duration_ms, environment: configuration.environment, captured_at: job.captured_at
       )
+      DeliveryStatus.from(result) == DeliveryStatus::DELIVERED
     rescue StandardError => e
       log { "EXPLAIN failed: #{e.class}: #{e.message}" }
       false

data/lib/forge_ops_tracker/session_flusher.rb

@@ -16,6 +16,7 @@ module ForgeOpsTracker
       @crashed_sessions_count = 0
       @period_started_at = Time.now.utc
       @thread = nil
+      @outcome = FlushOutcome.new(configuration, "session counts")
 
       # Flushes whatever's already tallied on a normal process exit, so the last partial window
       # (anything shorter than a full session_flush_interval) isn't silently dropped.
@@ -34,6 +35,8 @@ module ForgeOpsTracker
     # Snapshots and resets the in-process counters, then delivers them. A failed delivery keeps
     # the counts where they are rather than resetting, so the next flush's window just grows
     # instead of losing what was already tallied; there's no other copy of this data anywhere.
+    # Counts the server rejected, or that failed FlushOutcome::MAX_FAILED_FLUSHES flushes in a
+    # row, are reset all the same: sending them again could never succeed.
     def flush
       snapshot = nil
 
@@ -50,7 +53,7 @@ module ForgeOpsTracker
         }
       end
 
-      return unless snapshot && client.deliver_session_checkin(snapshot)
+      return unless snapshot && @outcome.release?(client.deliver_session_checkin(snapshot))
 
       @mutex.synchronize do
         @sessions_count = 0

data/lib/forge_ops_tracker/strict_privacy.rb

@@ -0,0 +1,155 @@
+require "digest"
+
+module ForgeOpsTracker
+  # What Configuration#strict_privacy changes in everything this gem sends, for apps that can't let
+  # personal or health data leave their own servers. With it on, only structured facts go out (the
+  # error's class, its stack frames, release, environment, server, which request it happened in,
+  # timings and counts); free text, which is where a name, an email or a diagnosis hides, never
+  # leaves this process: no error message, no breadcrumbs, no raw request path, no SQL statement,
+  # no source context, no user email or username, and only the context/tag keys listed in
+  # Configuration#privacy_allowed_fields.
+  #
+  # The same rules as the server's own StrictPrivacy (app/services/strict_privacy.rb), which
+  # applies them again on arrival when the project's Strict privacy setting is on;
+  # sdks/strict_privacy_vectors.json pins the results both have to agree on, byte for byte.
+  module StrictPrivacy
+    FINGERPRINT_LENGTH = 32
+
+    # ASCII "word" characters; boundaries are defined against these, the same in every SDK, so
+    # Ruby's own idea of a word never changes the result.
+    WORD = "A-Za-z0-9_".freeze
+    HEX_OBJECT_ID = /0x[0-9a-fA-F]{4,}/
+    UUID = /(?<![#{WORD}])[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}(?![#{WORD}])/
+    NUMBER = /(?<![#{WORD}])[0-9]+(?![#{WORD}])/
+
+    # Span data that's structure, not content: the masked statement and which database it ran on.
+    SPAN_DATA_KEYS = %w[db.statement db.system].freeze
+    HTTP_METHOD = /\A(GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)\s+(\S+)/i
+    SCHEME = %r{\A[a-z][a-z0-9+.-]*://}i
+
+    # Error event fields that are never sent, and the per-frame source context.
+    EVENT_FIELDS_DROPPED = %i[message breadcrumbs endpoint sql_statement].freeze
+    SOURCE_CONTEXT_FIELDS = %i[context_line pre_context post_context].freeze
+
+    module_function
+
+    # The message with hex object ids, UUIDs and standalone numbers replaced, in that order.
+    def normalize_message(message)
+      message.to_s.gsub(HEX_OBJECT_ID, "0x").gsub(UUID, "<uuid>").gsub(NUMBER, "<n>")
+    end
+
+    # Lets ForgeOps group errors without ever seeing their message: the first 32 hex characters of
+    # the SHA-256 of the normalized message's UTF-8 bytes, so two messages that differ only in ids
+    # hash the same, and nothing about the text can be read back from it.
+    #
+    # A message labelled binary (some C extensions raise with one) is read as the UTF-8 it almost
+    # always is, rather than converted byte by byte, which would turn every non-ASCII character into
+    # replacement characters and hash differently from the same text arriving as UTF-8.
+    def message_fingerprint(message)
+      normalized = normalize_message(message)
+      normalized = normalized.dup.force_encoding(Encoding::UTF_8) if normalized.encoding == Encoding::BINARY
+      normalized = normalized.encode(Encoding::UTF_8, invalid: :replace, undef: :replace) unless normalized.encoding == Encoding::UTF_8
+      Digest::SHA256.hexdigest(normalized.scrub)[0, FINGERPRINT_LENGTH]
+    end
+
+    # An error event payload (EventBuilder#build's shape, symbol keys) reduced to what strict
+    # privacy sends. message is sent as "" alongside message_fingerprint, exactly what the server
+    # stores for a strict project.
+    def event(payload, allowed_fields: [])
+      payload.reject { |key, _value| EVENT_FIELDS_DROPPED.include?(key) }.merge(
+        message: "",
+        message_fingerprint: message_fingerprint(payload[:message]),
+        transaction_name: transaction_name(payload[:transaction_name]),
+        context: allowed(payload[:context], allowed_fields),
+        tags: allowed(payload[:tags], allowed_fields),
+        user: user(payload[:user]),
+        backtrace: Array(payload[:backtrace]).map { |frame| frame.reject { |key, _value| SOURCE_CONTEXT_FIELDS.include?(key) } }
+      ).compact
+    end
+
+    # A background job or dependency failure: its class is kept, its message never is, and its
+    # transaction_name goes through transaction_name above.
+    def failure_event(entry)
+      entry.reject { |key, _value| key == :error_message }.merge(transaction_name: transaction_name(entry[:transaction_name]))
+    end
+
+    # A trace's span: an outbound call's name keeps its method and host, never its path or query,
+    # and its data keeps only the masked statement and database system. data is left out entirely
+    # when nothing is left of it, the same as the server does.
+    def span(span)
+      data = (span[:data] || {}).select { |key, _value| SPAN_DATA_KEYS.include?(key.to_s) }
+      span.merge(name: span_name(span[:name]), data: (data unless data.empty?)).compact
+    end
+
+    # "GET https://api.example.com/patients/42?x=1" or "GET api.example.com/patients/42" becomes
+    # "GET api.example.com". A bare path ("/patients/42"), or a method and a path
+    # ("PUT /orders/17/items"), has its ids taken out, like a page path. A name that's neither
+    # ("OrdersController#create", "patients#show") is kept.
+    def span_name(name)
+      return path_without_ids(name) if name.to_s.start_with?("/")
+
+      match = name.to_s.match(HTTP_METHOD)
+      return name unless match
+
+      method = match[1].upcase
+      target = match[2]
+      return "#{method} #{path_without_ids(target)}" if target.start_with?("/")
+
+      "#{method} #{host(target) || path_without_ids(target)}"
+    end
+
+    # Which request something happened in: a route or a controller action is kept, but a raw path
+    # (what a request no route matched falls back to) has its ids stripped, by the span name rule.
+    def transaction_name(name)
+      name.nil? || name.to_s.empty? ? name : span_name(name)
+    end
+
+    # An outbound call's host, read by hand rather than with URI.parse (an address a parser rejects
+    # must never be sent whole): what comes before the first "/", "?" or "#", without a scheme,
+    # credentials (up to the last "@", since a password can hold one) or port. Only a host when the
+    # address had a scheme or a port, or the name has a dot or is "localhost"; nil otherwise (a
+    # relative path).
+    def host(target)
+      scheme = target.match?(SCHEME)
+      authority = target.sub(SCHEME, "").split(%r{[/?#]}, 2).first.to_s.sub(/\A.*@/m, "")
+      port = authority.match?(/:\d+\z/)
+      host = authority.sub(/:\d+\z/, "")
+      return nil if host.empty?
+
+      host if scheme || port || host.include?(".") || host.casecmp?("localhost")
+    end
+
+    # A path with every segment that holds a digit, or is long enough to be a token or a name
+    # slug, replaced by ":id"; no query or fragment. "/patients/42/notes?q=x" is "/patients/:id/notes".
+    def path_without_ids(path)
+      segments = path.to_s.split(/[?#]/).first.to_s.split("/").map do |segment|
+        segment.match?(/\d/) || segment.length > 40 ? ":id" : segment
+      end
+      joined = segments.join("/")
+      joined.empty? ? "/" : joined
+    end
+
+    # Only top-level keys named in allowed_fields, and only plain values under them; nil (left out
+    # of the payload) when nothing is left.
+    def allowed(fields, allowed_fields)
+      allowed_fields = Array(allowed_fields).map(&:to_s)
+      return nil if allowed_fields.empty? || !fields.is_a?(Hash)
+
+      kept = fields.each_with_object({}) do |(key, value), result|
+        next unless allowed_fields.include?(key.to_s)
+        next unless value.is_a?(String) || value.is_a?(Numeric) || value == true || value == false
+
+        result[key.to_s] = value
+      end
+      kept unless kept.empty?
+    end
+
+    # The user's id, as a string, and nothing else; nil when there's no id.
+    def user(user)
+      return nil unless user.is_a?(Hash)
+
+      id = user[:id] || user["id"]
+      { id: id.to_s } unless id.nil? || id.to_s.strip.empty?
+    end
+  end
+end

data/lib/forge_ops_tracker/version.rb

@@ -1,3 +1,3 @@
 module ForgeOpsTracker
-  VERSION = "0.16.0"
+  VERSION = "0.18.0"
 end

data/lib/forge_ops_tracker.rb

@@ -3,7 +3,9 @@ require "forge_ops_tracker/version"
 require "forge_ops_tracker/configuration"
 require "forge_ops_tracker/pii_scrubber"
 require "forge_ops_tracker/sql_statement"
+require "forge_ops_tracker/strict_privacy"
 require "forge_ops_tracker/event_builder"
+require "forge_ops_tracker/delivery_status"
 require "forge_ops_tracker/client"
 require "forge_ops_tracker/delivery_queue"
 require "forge_ops_tracker/change"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forge_ops_tracker
 version: !ruby/object:Gem::Version
-  version: 0.16.0
+  version: 0.18.0
 platform: ruby
 authors:
 - ForgeOps
@@ -109,6 +109,7 @@ files:
 - lib/forge_ops_tracker/client.rb
 - lib/forge_ops_tracker/configuration.rb
 - lib/forge_ops_tracker/delivery_queue.rb
+- lib/forge_ops_tracker/delivery_status.rb
 - lib/forge_ops_tracker/error_subscriber.rb
 - lib/forge_ops_tracker/event_builder.rb
 - lib/forge_ops_tracker/failure_event_buffer.rb
@@ -137,6 +138,7 @@ files:
 - lib/forge_ops_tracker/span_buffer.rb
 - lib/forge_ops_tracker/span_queue.rb
 - lib/forge_ops_tracker/sql_statement.rb
+- lib/forge_ops_tracker/strict_privacy.rb
 - lib/forge_ops_tracker/trace_parent.rb
 - lib/forge_ops_tracker/version.rb
 homepage: https://getforgeops.net