forge_ops_tracker 0.12.0 → 0.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70707edca2368561ce9b785b91dc9e8a26280071ec3db9ed34fa71db6c402fcf
-  data.tar.gz: a6fc1013df0404e46e0683749f2cedd688ac30273bda4ac113a7dd6334a19430
+  metadata.gz: 5b7f1809933ffeccac536f117de7c234002826fe29414fc3fa92f12c22bb1a18
+  data.tar.gz: 84a9edadbaeb72dbe62164a84f26c7f23b6a600f176126ca82370709dc72b5c3
 SHA512:
-  metadata.gz: 420ff144de5e84636348c245fef836cf240c94fe6bccb49217f8d51455a3c08ce81c18dadcf7260ccc46f35dd7b2470d809b1eecd92d9bea70e17ecf149f1a78
-  data.tar.gz: 74fc60b3d7451827618e0d63deef8cd6563d29eab144c8556bc912e91bbb4a7c8467110ae25940a7e4a7457db8718cad8b763ff752311014ed325b22ce5c13b0
+  metadata.gz: ca419d26d17f08e4544bb213c2949dedf7001402f163a15b0def3c0942ffcc534637911d1bc7e8890f6a3dc556f174a3ecea703351fe291d492c1c151838e3f9
+  data.tar.gz: 6890f1adbbf73ae399699ee3482a005bbfee521909c6af6acedb8a805a2aa69c3d46f75bf33a3a4722be9f0e6e84a0263850729e42380d1d1b26e23e50f285cb

data/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## 0.14.0 (2026-09-25)
+
+- Every Active Record database span now carries the query's SQL (`db.statement`), masked so every
+  string and number is `?` and bind values are never sent, and the database it ran on
+  (`db.system`: `postgresql`, `mysql`, `sqlite`, `mssql`, `oracle`, or the adapter's own name).
+- `ForgeOpsTracker.span` takes optional `statement:` and `db_system:` on a `kind: "database"`
+  span, masked the same way.
+- New opt-in `config.explain_slow_queries` (default false) and `config.explain_threshold_ms`
+  (default 500). On PostgreSQL, a single plain `SELECT` at least that slow gets a plain
+  `EXPLAIN (FORMAT JSON)` (never `EXPLAIN ANALYZE`), run on a background thread with its own
+  pooled connection, in a read-only transaction with a 2 second statement timeout that is always
+  rolled back. The plan is sent with every string in it masked. Limited to once per 10 minutes per
+  distinct query and 10 per minute per process; plans over 64 KB are dropped; any failure is
+  logged at debug and never raised.
+
+## 0.13.0 (2026-09-25)
+
+- New `ForgeOpsTracker.record_change(kind:, title:, details: {}, environment: nil, service: nil,
+  actor: nil, url: nil, id: nil, occurred_at: nil)` records something that changed in your system
+  (a feature flag, a config value, a hand-run migration) so ForgeOps can show it next to the errors
+  that followed. `kind` is one of `feature_flag`, `config`, `migration`, `dependency`,
+  `infrastructure`, or `other`; anything else is sent as `other`. Sent from a background thread,
+  never raises, and a no-op wherever reporting is off.
+- Changes between deploys are now detected automatically. Once per process, after Rails finishes
+  booting (or on `ForgeOpsTracker.configure` outside Rails), a background thread sends a snapshot of
+  the Ruby version, the gem versions in `Gemfile.lock`, and the current Active Record migration
+  version, and ForgeOps records whatever changed since the previous boot. New
+  `config.detect_changes` (default true) turns this off.
+- New `config.track_env_var_names` (default false) adds environment variable names, never values,
+  to that snapshot. Host-specific names (`HOSTNAME`, `PATH`, `PORT`, `LC_*`, Kubernetes service
+  variables, and others) and the gem's own `FORGE_OPS_*` settings are always left out.
+
 ## 0.12.0
 
 - Every error event reported during a request now says where it happened: `transaction_name` (the

data/README.md

@@ -411,6 +411,116 @@ Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn
 simply rejected server-side and dropped, exactly like any other delivery failure. The `trace_id`
 on error events is sent either way.
 
+### The SQL behind each database span
+
+Every Active Record query span carries the query's SQL and the database it ran on, so a slow trace
+shows exactly which query was slow. The SQL is masked before it leaves your process: every string
+and number is replaced by `?`, so `WHERE email = 'a@b.co' AND id = 42` is sent as
+`WHERE email = ? AND id = ?`, and bind values are never sent at all. ForgeOps masks it again on
+arrival. The database comes from the connection's adapter (`postgresql`, `mysql`, `sqlite`,
+`mssql`, `oracle`, or the adapter's own name otherwise).
+
+Spans you create yourself can carry SQL too. Pass `statement:` (and optionally `db_system:`) on a
+`kind: "database"` span; it's masked the same way:
+
+```ruby
+sql = ActiveRecord::Base.sanitize_sql_array(
+  [ "SELECT * FROM orders WHERE customer_id = ? AND status = ?", customer.id, "open" ]
+)
+
+ForgeOpsTracker.span("Load open orders", kind: "database", statement: sql, db_system: "postgresql") do
+  ReportingDatabase.connection.select_all(sql)
+end
+# Sent as "SELECT * FROM orders WHERE customer_id = ? AND status = ?"
+```
+
+### Query plans for slow queries (PostgreSQL)
+
+Opt in, and a slow `SELECT` also sends Postgres's plan for it, so ForgeOps can show why it was
+slow (a sequential scan over a big table, a missing index, a bad join order):
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.explain_slow_queries = true # default false
+  config.explain_threshold_ms = 500  # default 500; only queries at least this slow
+end
+```
+
+What it does:
+
+- For a query that took at least `explain_threshold_ms`, runs `EXPLAIN (FORMAT JSON)` with the
+  same SQL and bind values, and sends the plan with the query's masked SQL. Every string inside the
+  plan (a filter like `(email = 'a@b.co'::text)`) is masked the same way before it's sent. A plan
+  over 64 KB is dropped rather than sent.
+- Runs on the gem's own background thread, on a connection it checks out of your app's pool itself,
+  inside its own `READ ONLY` transaction with a 2 second `statement_timeout`, always rolled back.
+  Never on the connection your request is using, and never inside your request's transaction.
+- At most once per 10 minutes for each distinct (masked) query, and at most 10 per minute per
+  process overall.
+
+What it doesn't do:
+
+- It never runs `EXPLAIN ANALYZE`, so your query is planned again but never executed again.
+- It only explains a single plain `SELECT`: nothing that starts with anything else (including
+  `WITH`), nothing with a second statement, no `FOR UPDATE`/`FOR SHARE` (or `FOR NO KEY UPDATE`/
+  `FOR KEY SHARE`), and nothing with `INSERT`, `UPDATE`, `DELETE` or `MERGE` anywhere in it.
+- It only runs on PostgreSQL. Other databases are left alone.
+- It never raises into your app. A timeout, a permission error, an exhausted connection pool or
+  anything else is logged at debug level and skipped.
+- The original SQL and its bind values are only held in memory until the `EXPLAIN` runs. Only the
+  masked SQL and the masked plan are sent.
+
+It does use one pooled connection, briefly, for each plan. With the limits above that's at most 10
+short checkouts a minute, but on a pool that's already at capacity, leave it off or raise
+`explain_threshold_ms`. Plans are part of performance monitoring; on a plan without it, they're
+rejected server-side and dropped.
+
+## What changed
+
+ForgeOps can show what changed in your system next to the errors and slowdowns that followed it.
+Two ways in:
+
+**Record a change yourself** when something changes that no deploy captures, like a feature flag
+flipped, a config value edited, or a migration run by hand:
+
+```ruby
+ForgeOpsTracker.record_change(
+  kind: "feature_flag", # feature_flag, config, migration, dependency, infrastructure, or other
+  title: "Enabled new_checkout for 10% of users",
+  details: { flag: "new_checkout", rollout_percent: 10 },
+  actor: current_user.email,
+  url: "https://flags.example.com/new_checkout"
+)
+```
+
+`kind` and `title` are required; `details`, `environment` (defaults to `config.environment`),
+`service`, `actor`, `url`, `id` (an idempotency key, so sending the same change twice records it
+once), and `occurred_at` (defaults to now) are optional. An unknown `kind` is sent as `other`. It's
+queued and sent from a background thread, so it never slows down the caller, never raises, and is
+a no-op wherever reporting is off.
+
+**Changes between deploys are detected for you.** Once per process, right after Rails finishes
+booting, a background thread sends ForgeOps a snapshot of what the app is running: the Ruby
+version, every gem version in your `Gemfile.lock`, and the current Active Record migration version
+(left out if the database can't be reached). ForgeOps compares it with the previous boot's and
+records whatever changed, such as a gem upgrade or a new migration. Outside Rails the snapshot goes
+out when you call `ForgeOpsTracker.configure`, without the migration version.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.detect_changes = true       # default; false sends no startup snapshot
+  config.track_env_var_names = false # default; true also sends environment variable names
+end
+```
+
+With `track_env_var_names` on, the snapshot lists the names of your environment variables (never
+their values), so an added or removed variable shows up as a change. Names that differ from host to
+host, like `HOSTNAME`, `PATH`, `PORT`, `LC_*`, and Kubernetes service variables, are left out, as
+are the gem's own `FORGE_OPS_*` settings.
+
+Requires a ForgeOps plan that includes change tracking; on a plan that doesn't, both are rejected
+server-side and dropped, exactly like any other delivery failure.
+
 ## Custom metrics
 
 Unlike session/performance tracking above, there's no automatic instrumentation here at all: a

data/lib/forge_ops_tracker/change.rb

@@ -0,0 +1,47 @@
+require "time"
+
+module ForgeOpsTracker
+  # Builds the body for one ForgeOpsTracker.record_change call (POST /api/v1/changes): something
+  # that changed in a running system (a feature flag flipped, a config value edited, a migration
+  # run by hand), so ForgeOps can line it up against the errors and slowdowns around it.
+  module Change
+    # The server rejects any other kind outright (422), so an unknown one is sent as "other"
+    # rather than dropped: the change still gets recorded, just less specifically categorized.
+    KINDS = %w[feature_flag config migration dependency infrastructure other].freeze
+    MAX_TITLE_LENGTH = 200
+
+    module_function
+
+    # Returns nil when there's nothing sendable (a blank title): the server requires one, so
+    # posting it anyway would only ever come back 422.
+    def build(configuration, kind:, title:, details: {}, environment: nil, service: nil, actor: nil, url: nil, id: nil, occurred_at: nil)
+      title = title.to_s.strip
+      return nil if title.empty?
+
+      {
+        kind: normalize_kind(kind),
+        title: title[0, MAX_TITLE_LENGTH],
+        details: details.is_a?(Hash) ? details : {},
+        environment: (environment || configuration.environment)&.to_s,
+        service: service&.to_s,
+        actor: actor&.to_s,
+        url: url&.to_s,
+        id: id&.to_s,
+        occurred_at: iso8601(occurred_at || Time.now)
+      }.compact
+    end
+
+    def normalize_kind(kind)
+      kind = kind.to_s
+      KINDS.include?(kind) ? kind : "other"
+    end
+
+    def iso8601(value)
+      case value
+      when Time then value.utc.iso8601(3)
+      when String then value
+      else value.respond_to?(:to_time) ? value.to_time.utc.iso8601(3) : Time.now.utc.iso8601(3)
+      end
+    end
+  end
+end

data/lib/forge_ops_tracker/change_snapshot.rb

@@ -0,0 +1,133 @@
+require "thread"
+
+module ForgeOpsTracker
+  # Tells ForgeOps what this process sees at startup (the Ruby version, the locked gem versions, the
+  # current Active Record migration version, and optionally the names of its environment variables),
+  # once per process, from a background thread. The server diffs it against the last snapshot for
+  # the same project and environment and records whatever changed as a change, so this gem stays
+  # stateless: it never remembers what it sent last time.
+  #
+  # Any key that can't be determined reliably is left out rather than guessed at, since the server
+  # reads a missing key as "unknown", never as "everything was removed".
+  class ChangeSnapshot
+    MAX_DEPENDENCIES = 3000
+    MAX_NAME_LENGTH = 200
+    MAX_VERSION_LENGTH = 100
+    MAX_ENV_VAR_NAMES = 3000
+
+    # Host-specific variables that differ from one box to the next (or one boot to the next) without
+    # anything about the deploy having changed, so a fleet doesn't look like it's changing on every
+    # restart. The gem's own FORGE_OPS_* settings are dropped too.
+    ENV_VAR_DENYLIST = %w[
+      HOSTNAME HOST HOME PATH PWD OLDPWD SHLVL _ TERM USER LOGNAME SHELL LANG TMPDIR TZ PORT DYNO
+      INVOCATION_ID JOURNAL_STREAM
+    ].freeze
+    ENV_VAR_DENYLIST_PATTERNS = [
+      /\ALC_/, /\ASYSTEMD_/, /\AMEMORY_PRESSURE_/, /\AKUBERNETES_/, /\AFORGE_OPS_/,
+      /_SERVICE_HOST\z/, /_SERVICE_PORT/, /_PORT_.*_TCP/
+    ].freeze
+
+    MUTEX = Mutex.new
+
+    class << self
+      # Starts the one snapshot this process sends, on its own thread so it never delays boot. A
+      # second call is a no-op, so the Railtie and a plain ForgeOpsTracker.configure can't both
+      # send one. Does nothing (and doesn't use up the once) when the gem isn't enabled or
+      # detect_changes is off. Never raises.
+      def start(configuration)
+        return false unless configuration.enabled? && configuration.detect_changes
+
+        MUTEX.synchronize do
+          return false if @started
+
+          @started = true
+        end
+
+        thread = Thread.new { new(configuration).deliver }
+        thread.report_on_exception = false
+        true
+      rescue StandardError
+        false
+      end
+
+      # Names only, never values, sorted so the same set always serializes the same way.
+      def env_var_names(env = ENV)
+        env.keys.reject { |name| denied_env_var?(name) }.sort.first(MAX_ENV_VAR_NAMES)
+      end
+
+      def denied_env_var?(name)
+        ENV_VAR_DENYLIST.include?(name) || ENV_VAR_DENYLIST_PATTERNS.any? { |pattern| pattern.match?(name) }
+      end
+    end
+
+    def initialize(configuration, client: Client.new(configuration))
+      @configuration = configuration
+      @client = client
+    end
+
+    def deliver
+      client.deliver_change_snapshot(payload)
+    rescue StandardError => e
+      configuration.logger&.debug { "[ForgeOpsTracker] change snapshot failed: #{e.class}: #{e.message}" }
+      false
+    end
+
+    def payload
+      {
+        environment: configuration.environment.to_s,
+        state: {
+          schema_version: schema_version,
+          runtime: runtime,
+          dependencies: dependencies,
+          env_var_names: configuration.track_env_var_names ? self.class.env_var_names : nil
+        }.compact
+      }
+    end
+
+    def runtime
+      "#{RUBY_ENGINE} #{RUBY_ENGINE_VERSION}"
+    end
+
+    # The app's Gemfile.lock (through Bundler, so it's the exact resolved versions), falling back to
+    # whatever gems are actually activated when there's no bundle at all. nil, and so left out,
+    # when neither is available.
+    def dependencies
+      specs = bundler_specs || Gem.loaded_specs.values
+      return nil if specs.empty?
+
+      specs.each_with_object({}) do |spec, deps|
+        deps[spec.name.to_s[0, MAX_NAME_LENGTH]] = spec.version.to_s[0, MAX_VERSION_LENGTH]
+      end.sort.first(MAX_DEPENDENCIES).to_h
+    rescue StandardError
+      nil
+    end
+
+    # The newest migration Active Record has run against this app's database, as a string. Left
+    # out when Active Record isn't loaded, the database can't be reached, or nothing has been
+    # migrated yet: a snapshot must never fail (or fail boot) because the database is down.
+    def schema_version
+      return nil unless defined?(::ActiveRecord::Base)
+
+      version = ::ActiveRecord::Base.connection_pool.with_connection do |connection|
+        pool = ::ActiveRecord::Base.connection_pool
+        context = pool.respond_to?(:migration_context) ? pool.migration_context : connection.migration_context
+        context.current_version
+      end
+      version.to_i.positive? ? version.to_s : nil
+    rescue StandardError, ScriptError
+      nil
+    end
+
+    private
+      attr_reader :configuration, :client
+
+      def bundler_specs
+        return nil unless defined?(::Bundler)
+
+        specs = ::Bundler.locked_gems&.specs
+        specs.nil? || specs.empty? ? nil : specs
+      rescue StandardError
+        nil
+      end
+  end
+end

data/lib/forge_ops_tracker/client.rb

@@ -60,6 +60,23 @@ module ForgeOpsTracker
       post(configuration.spans_uri, { trace_id: trace_id, spans: spans })
     end
 
+    # Same delivery contract again; see Configuration#changes_uri and ForgeOpsTracker.record_change.
+    # One change per call. A plan without change tracking answers 403, which is just a false here.
+    def deliver_change(payload)
+      post(configuration.changes_uri, payload)
+    end
+
+    # Same again; see Configuration#change_snapshots_uri and ChangeSnapshot.
+    def deliver_change_snapshot(payload)
+      post(configuration.change_snapshots_uri, payload)
+    end
+
+    # Same again; see Configuration#query_plans_uri and QueryExplainer. One plan per call. A plan
+    # without performance monitoring answers 403, which is just a false here.
+    def deliver_query_plan(payload)
+      post(configuration.query_plans_uri, payload)
+    end
+
     private
       attr_reader :configuration
 

data/lib/forge_ops_tracker/configuration.rb

@@ -17,7 +17,9 @@ module ForgeOpsTracker
     attr_accessor :metric_flush_interval, :infrastructure_metric_flush_interval
     attr_accessor :gauge_poll_interval
     attr_accessor :track_tracing, :trace_capture_threshold_ms
+    attr_accessor :explain_slow_queries, :explain_threshold_ms
     attr_accessor :propagate_traces, :trace_propagation_targets
+    attr_accessor :detect_changes, :track_env_var_names
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -125,6 +127,13 @@ module ForgeOpsTracker
       # PerformanceSample-derived threshold elsewhere in this gem: those describe a trend across
       # many requests, this describes a single one being outright slow.
       @trace_capture_threshold_ms = 1_000
+      # Opt-in: for a SELECT on PostgreSQL that took at least explain_threshold_ms, ask Postgres for
+      # its plan (plain EXPLAIN, never EXPLAIN ANALYZE, so the query is never run again) and send
+      # it, masked, so ForgeOps can show why the query was slow. Off by default because it runs one
+      # extra statement against the app's own database. See QueryExplainer for exactly what it
+      # does and doesn't do, and its rate limits.
+      @explain_slow_queries = false
+      @explain_threshold_ms = 500
       # Adds a W3C `traceparent` header (https://www.w3.org/TR/trace-context/) to every outbound
       # Net::HTTP request made during a request, so the service on the other end can continue this
       # request's trace instead of starting its own (see Integrations::NetHTTP). On by default, the
@@ -138,6 +147,15 @@ module ForgeOpsTracker
       # Regexp is matched against the host. Worth narrowing for an app that calls third-party APIs
       # which reject unknown headers, or that shouldn't learn this app's trace ids at all.
       @trace_propagation_targets = nil
+      # Sends one ChangeSnapshot per process at startup (Ruby version, locked gem versions, the
+      # current migration version), which ForgeOps diffs against the previous boot's to record what
+      # changed between deploys. On by default, like every other automatic behavior here.
+      @detect_changes = true
+      # Whether that snapshot also lists the names of this process's environment variables, so an
+      # added or removed variable shows up as a change. Off by default: names only, never values,
+      # but even names can describe more about an app than some teams want to share. See
+      # ChangeSnapshot::ENV_VAR_DENYLIST for the host-specific names that are always left out.
+      @track_env_var_names = false
     end
 
     # Whether an outbound request to host should carry a traceparent header; see
@@ -231,11 +249,34 @@ module ForgeOpsTracker
       uri
     end
 
+    def query_plans_uri
+      sibling_uri("/query_plans")
+    end
+
+    def changes_uri
+      sibling_uri("/changes")
+    end
+
+    def change_snapshots_uri
+      sibling_uri("/change_snapshots")
+    end
+
     def enabled?
       !blank?(dsn) && !blank?(api_key) && enabled_environments.map(&:to_s).include?(environment.to_s)
     end
 
     private
+      # Same substitution every *_uri method above spells out by hand: a sibling path under the
+      # DSN's own /api/v1/events.
+      def sibling_uri(path)
+        uri = ingestion_uri
+        return nil unless uri
+
+        uri = uri.dup
+        uri.path = uri.path.sub(%r{/events\z}, path)
+        uri
+      end
+
       def parsed_dsn
         return nil if blank?(dsn)
 

data/lib/forge_ops_tracker/delivery_queue.rb

@@ -8,9 +8,12 @@ module ForgeOpsTracker
   # in each forked Puma/Passenger worker process rather than being carried
   # across a fork, which would leave it dead in the child.
   class DeliveryQueue
-    def initialize(configuration, client: Client.new(configuration))
+    # deliver_with names the Client method each payload goes through: :deliver (error events) by
+    # default, :deliver_change for ForgeOpsTracker.record_change's own queue.
+    def initialize(configuration, client: Client.new(configuration), deliver_with: :deliver)
       @configuration = configuration
       @client = client
+      @deliver_with = deliver_with
       @queue = SizedQueue.new(configuration.queue_size)
       @mutex = Mutex.new
       @thread = nil
@@ -44,7 +47,7 @@ module ForgeOpsTracker
 
       def run
         loop do
-          client.deliver(@queue.pop)
+          client.public_send(@deliver_with, @queue.pop)
         rescue StandardError => e
           log { "delivery thread error: #{e.class}: #{e.message}" }
         end

data/lib/forge_ops_tracker/performance_instrumentation.rb

@@ -70,6 +70,38 @@ module ForgeOpsTracker
       name.nil? || name.empty? ? "SQL" : name
     end
 
+    # Active Record's adapter names, as ForgeOps' "db.system" values. Anything not listed here is
+    # sent as its own adapter name, lowercased.
+    DB_SYSTEMS = {
+      "postgresql" => "postgresql", "postgis" => "postgresql",
+      "mysql" => "mysql", "mysql2" => "mysql", "trilogy" => "mysql",
+      "sqlite" => "sqlite", "sqlite3" => "sqlite",
+      "sqlserver" => "mssql",
+      "oracle" => "oracle", "oracleenhanced" => "oracle"
+    }.freeze
+
+    # The "data" a database span carries: the query's SQL, masked by SqlStatement (every string
+    # and number literal becomes "?", so no value the app queried with is ever sent), and which
+    # database it ran on, from the connection's adapter. Either is left out when it isn't known.
+    def database_span_data(payload)
+      {
+        "db.statement" => SqlStatement.mask(payload[:sql].to_s),
+        "db.system" => db_system(payload[:connection])
+      }.compact
+    end
+
+    # payload[:connection] is the adapter instance the query ran on (Rails 6.0 and later).
+    def db_system(connection)
+      return nil unless connection.respond_to?(:adapter_name)
+
+      name = connection.adapter_name.to_s.downcase
+      return nil if name.empty?
+
+      DB_SYSTEMS.fetch(name, name)
+    rescue StandardError
+      nil
+    end
+
     def job_transaction_name(payload)
       payload[:job].class.name
     end

data/lib/forge_ops_tracker/query_explainer.rb

@@ -0,0 +1,249 @@
+require "json"
+require "thread"
+require "time"
+
+module ForgeOpsTracker
+  # Opt-in (configuration.explain_slow_queries, default false): asks PostgreSQL for the plan of a
+  # slow SELECT and sends it, masked, to ForgeOps so a slow query shows why it was slow.
+  #
+  # What it never does, by design:
+  # - EXPLAIN ANALYZE. Plain EXPLAIN only plans the statement, it never executes it.
+  # - Touch the app's own connection or transaction. #consider only reads the notification payload
+  #   and remembers the connection's pool; the EXPLAIN runs later, on this class's own background
+  #   thread, on a connection it checks out of that pool itself, inside its own READ ONLY
+  #   transaction with a 2 second statement_timeout, always rolled back.
+  # - Explain anything but a single plain SELECT (see .explainable?).
+  # - Raise. Every failure (a timeout, a permission error, an exhausted pool, a statement Postgres
+  #   can't parse) is logged at debug and dropped.
+  # - Send the original statement or its bind values anywhere. Both are held in memory only until
+  #   the EXPLAIN runs; what goes out is the masked statement (exactly as sent on the span) and the
+  #   plan with every string in it masked the same way, since plan conditions quote literals
+  #   ("Filter": "(email = 'a@b.co'::text)").
+  #
+  # Rate limited per process: each distinct masked statement at most once per 10 minutes, and at
+  # most 10 EXPLAINs per minute overall. Both are reserved when a query is accepted, not when its
+  # EXPLAIN succeeds, so a statement whose EXPLAIN keeps failing still can't be retried more often.
+  class QueryExplainer
+    STATEMENT_INTERVAL = 600.0
+    GLOBAL_WINDOW = 60.0
+    GLOBAL_LIMIT = 10
+    MAX_PLAN_BYTES = 64 * 1024
+    STATEMENT_TIMEOUT = "2s".freeze
+    CHECKOUT_TIMEOUT = 1
+    QUEUE_SIZE = GLOBAL_LIMIT
+    # Past this many remembered statements, the ones whose 10 minutes are up are forgotten, so an
+    # app with endless distinct statements can't grow this without bound.
+    MAX_REMEMBERED_STATEMENTS = 1_000
+
+    # One pass, left to right, so a quote inside a comment or comment syntax inside a string can't
+    # confuse the other: string literals (single-quoted and dollar-quoted) become "?", comments
+    # become a space.
+    LITERALS_AND_COMMENTS = %r{
+      '(?:[^']|'')*(?:'|\z)
+      | (?<tag>\$[A-Za-z_]*\$).*?(?:\k<tag>|\z)
+      | /\*.*?(?:\*/|\z)
+      | --[^\n]*
+    }mx
+    LOCKING_CLAUSE = /\bFOR\s+(?:NO\s+KEY\s+UPDATE|KEY\s+SHARE|UPDATE|SHARE)\b/i
+    DATA_MODIFYING = /\b(?:INSERT|UPDATE|DELETE|MERGE)\b/i
+
+    Job = Struct.new(:sql, :binds, :pool, :statement, :duration_ms, :captured_at, keyword_init: true)
+
+    class << self
+      # True on this class's own worker thread while it's running an EXPLAIN. Railtie's
+      # sql.active_record subscriber skips those queries entirely, so the EXPLAIN (and its BEGIN,
+      # SET LOCAL and ROLLBACK) never shows up as the app's own query timing, breadcrumbs or spans.
+      def explaining?
+        Thread.current[:forge_ops_tracker_explaining] == true
+      end
+
+      # Whether statement is a single plain SELECT: after dropping comments and string literals,
+      # it starts with SELECT, has no ";" other than one trailing one, no locking clause (FOR
+      # UPDATE, FOR SHARE, FOR NO KEY UPDATE, FOR KEY SHARE), and no INSERT, UPDATE, DELETE or
+      # MERGE anywhere (so no data-modifying CTE either).
+      def explainable?(statement)
+        return false unless statement.is_a?(String)
+
+        sql = statement.gsub(LITERALS_AND_COMMENTS) { |match| match.start_with?("'", "$") ? "?" : " " }.strip
+        return false unless sql.match?(/\ASELECT\b/i)
+        return false if sql.delete_suffix(";").include?(";")
+
+        !sql.match?(LOCKING_CLAUSE) && !sql.match?(DATA_MODIFYING)
+      end
+
+      # Masks every String inside a parsed EXPLAIN (FORMAT JSON) result with SqlStatement, leaving
+      # keys, numbers, booleans, arrays and the structure itself exactly as they were.
+      def mask_plan(value)
+        case value
+        when Hash then value.transform_values { |inner| mask_plan(inner) }
+        when Array then value.map { |inner| mask_plan(inner) }
+        when String then SqlStatement.mask(value) || value
+        else value
+        end
+      end
+    end
+
+    def initialize(configuration, client: Client.new(configuration), clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
+      @configuration = configuration
+      @client = client
+      @clock = clock
+      @queue = SizedQueue.new(QUEUE_SIZE)
+      @mutex = Mutex.new
+      @thread = nil
+      @last_explained = {}
+      @recent = []
+    end
+
+    # Called from the sql.active_record subscriber with that event's payload and duration. Cheap
+    # checks first, so a fast query costs one comparison. Returns true only when an EXPLAIN was
+    # queued; never raises.
+    def consider(payload, duration_ms:)
+      return false unless configuration.explain_slow_queries
+      return false unless duration_ms && duration_ms >= configuration.explain_threshold_ms.to_f
+
+      connection = payload[:connection]
+      return false unless PerformanceInstrumentation.db_system(connection) == "postgresql"
+
+      sql = payload[:sql]
+      return false unless self.class.explainable?(sql)
+
+      pool = connection.respond_to?(:pool) ? connection.pool : nil
+      return false unless pool
+
+      statement = SqlStatement.mask(sql)
+      return false unless reserve(statement)
+
+      enqueue(Job.new(
+        sql: sql.dup, binds: bind_values(payload), pool: pool, statement: statement,
+        duration_ms: duration_ms.to_f.round(1), captured_at: Time.now.utc.iso8601(3)
+      ))
+    rescue StandardError => e
+      log { "EXPLAIN skipped: #{e.class}: #{e.message}" }
+      false
+    end
+
+    # Runs one queued EXPLAIN and delivers its plan. Normally called on the worker thread; public so
+    # a test can run a job synchronously. Never raises.
+    def perform(job)
+      plan = explain(job)
+      return false unless plan
+
+      masked_plan = self.class.mask_plan(plan)
+      if JSON.generate(masked_plan).bytesize > MAX_PLAN_BYTES
+        log { "EXPLAIN plan over #{MAX_PLAN_BYTES} bytes, dropped" }
+        return false
+      end
+
+      client.deliver_query_plan(
+        statement: job.statement, db_system: "postgresql", plan: masked_plan,
+        duration_ms: job.duration_ms, environment: configuration.environment, captured_at: job.captured_at
+      )
+    rescue StandardError => e
+      log { "EXPLAIN failed: #{e.class}: #{e.message}" }
+      false
+    end
+
+    private
+      attr_reader :configuration, :client
+
+      # Both rate limits at once, under one lock, so two threads finishing the same slow query
+      # together still only explain it once.
+      def reserve(statement)
+        return false if statement.nil?
+
+        @mutex.synchronize do
+          now = @clock.call
+          @recent.reject! { |at| now - at >= GLOBAL_WINDOW }
+          return false if @recent.size >= GLOBAL_LIMIT
+
+          last = @last_explained[statement]
+          return false if last && now - last < STATEMENT_INTERVAL
+
+          if @last_explained.size >= MAX_REMEMBERED_STATEMENTS
+            @last_explained.reject! { |_, at| now - at >= STATEMENT_INTERVAL }
+          end
+          @last_explained[statement] = now
+          @recent << now
+          true
+        end
+      end
+
+      # The values the app's own query was run with, already type cast for the database. Rails
+      # passes these either as an Array or as a lambda that builds one.
+      def bind_values(payload)
+        binds = payload[:type_casted_binds]
+        binds = binds.call if binds.respond_to?(:call)
+        Array(binds).dup
+      end
+
+      def enqueue(job)
+        ensure_worker_started
+        @queue.push(job, true)
+        true
+      rescue ThreadError
+        log { "EXPLAIN queue full, dropping" }
+        false
+      end
+
+      def ensure_worker_started
+        return if @thread&.alive?
+
+        @mutex.synchronize do
+          return if @thread&.alive?
+
+          @thread = Thread.new { run }
+          @thread.abort_on_exception = false
+        end
+      end
+
+      def run
+        Thread.current[:forge_ops_tracker_explaining] = true
+        loop do
+          perform(@queue.pop)
+        rescue StandardError => e
+          log { "EXPLAIN thread error: #{e.class}: #{e.message}" }
+        end
+      end
+
+      # The parsed plan (the JSON array Postgres returns), or nil. The connection is checked out
+      # explicitly rather than borrowed with with_connection, so it's never one this thread (or
+      # any other) already holds. Checked back in only once its transaction is rolled back; if
+      # even the ROLLBACK fails, the connection is removed from the pool and closed instead, so a
+      # connection in an unknown state is never handed to the app.
+      def explain(job)
+        connection = job.pool.checkout(CHECKOUT_TIMEOUT)
+        begin
+          connection.execute("BEGIN READ ONLY")
+          connection.execute("SET LOCAL statement_timeout = '#{STATEMENT_TIMEOUT}'")
+          result = connection.exec_query("EXPLAIN (FORMAT JSON) #{job.sql}", "EXPLAIN", job.binds)
+          value = result.rows.first&.first
+          value.is_a?(String) ? JSON.parse(value) : value
+        ensure
+          release(job.pool, connection, rollback(connection))
+        end
+      end
+
+      def rollback(connection)
+        connection.execute("ROLLBACK")
+        true
+      rescue StandardError => e
+        log { "EXPLAIN rollback failed: #{e.class}: #{e.message}" }
+        false
+      end
+
+      def release(pool, connection, healthy)
+        if healthy
+          pool.checkin(connection)
+        else
+          pool.remove(connection)
+          connection.disconnect!
+        end
+      rescue StandardError => e
+        log { "EXPLAIN connection release failed: #{e.class}: #{e.message}" }
+      end
+
+      def log
+        configuration.logger&.debug { "[ForgeOpsTracker] #{yield}" }
+      end
+  end
+end

data/lib/forge_ops_tracker/railtie.rb

@@ -28,6 +28,11 @@ module ForgeOpsTracker
       def span_queue
         @span_queue ||= ForgeOpsTracker::SpanQueue.new(ForgeOpsTracker.configuration)
       end
+
+      # One per process, so its rate limits (see QueryExplainer) are per process too.
+      def query_explainer
+        @query_explainer ||= ForgeOpsTracker::QueryExplainer.new(ForgeOpsTracker.configuration)
+      end
     end
 
     initializer "forge_ops_tracker.subscribe_error_reporter" do |app|
@@ -167,6 +172,9 @@ module ForgeOpsTracker
       end
 
       ActiveSupport::Notifications.subscribe("sql.active_record") do |*args|
+        # QueryExplainer's own EXPLAIN (and its BEGIN/SET LOCAL/ROLLBACK), on its own thread: not
+        # the app's query, so never timed, traced or explained itself.
+        next if QueryExplainer.explaining?
         next unless configuration.enabled?
 
         event = ActiveSupport::Notifications::Event.new(*args)
@@ -184,13 +192,19 @@ module ForgeOpsTracker
         # A leaf span, parented onto whatever's currently open (the controller root, or a
         # manually-wrapped ForgeOpsTracker.span block around it): a query has no children of its
         # own to nest anything under, so unlike the controller root above, this never touches the
-        # stack itself, just records against whatever's already on top of it.
+        # stack itself, just records against whatever's already on top of it. Carries the masked
+        # SQL and the database system (see PerformanceInstrumentation.database_span_data).
         if configuration.track_tracing && (buffer = Thread.current[:forge_ops_tracker_spans])
           buffer.record(
             span_id: SecureRandom.hex(8), name: transaction_name, kind: "database",
-            started_at: Time.at(event.time), duration_ms: event.duration
+            started_at: Time.at(event.time), duration_ms: event.duration,
+            data: PerformanceInstrumentation.database_span_data(event.payload)
           )
         end
+
+        # Opt-in, PostgreSQL only; a no-op unless config.explain_slow_queries is on. Only reads the
+        # payload here; the EXPLAIN itself runs later on QueryExplainer's own thread and connection.
+        Railtie.query_explainer.consider(event.payload, duration_ms: event.duration) if configuration.explain_slow_queries
       end
 
       ActiveSupport::Notifications.subscribe("perform.active_job") do |*args|
@@ -305,6 +319,14 @@ module ForgeOpsTracker
     # sidekiq, ActiveJob's redis queue adapter, ...), which this gem has no control over; deferring
     # the defined? check to after every railtie (including this app's own dependencies) has
     # finished initializing avoids a load-order-dependent false negative.
+    # After initialization, not in a plain initializer, so the app's own
+    # config/initializers/forge_ops_tracker.rb has already run (detect_changes/track_env_var_names
+    # and the DSN are final) and Active Record is configured enough to report the migration
+    # version. ChangeSnapshot does the actual work on its own thread, so boot never waits on it.
+    config.after_initialize do
+      ForgeOpsTracker::ChangeSnapshot.start(ForgeOpsTracker.configuration)
+    end
+
     config.after_initialize do
       next unless defined?(::RedisClient)
 

data/lib/forge_ops_tracker/version.rb

@@ -1,3 +1,3 @@
 module ForgeOpsTracker
-  VERSION = "0.12.0"
+  VERSION = "0.14.0"
 end

data/lib/forge_ops_tracker.rb

@@ -6,6 +6,8 @@ require "forge_ops_tracker/sql_statement"
 require "forge_ops_tracker/event_builder"
 require "forge_ops_tracker/client"
 require "forge_ops_tracker/delivery_queue"
+require "forge_ops_tracker/change"
+require "forge_ops_tracker/change_snapshot"
 require "forge_ops_tracker/error_subscriber"
 require "forge_ops_tracker/session_flusher"
 require "forge_ops_tracker/histogram_bucketer"
@@ -18,6 +20,7 @@ require "forge_ops_tracker/periodic_poller"
 require "forge_ops_tracker/breadcrumb_buffer"
 require "forge_ops_tracker/span_buffer"
 require "forge_ops_tracker/span_queue"
+require "forge_ops_tracker/query_explainer"
 require "forge_ops_tracker/trace_parent"
 require "forge_ops_tracker/request_state"
 require "forge_ops_tracker/middleware/session_tracking"
@@ -32,8 +35,33 @@ module ForgeOpsTracker
       @configuration ||= Configuration.new
     end
 
+    # Outside Rails this is also where the startup ChangeSnapshot goes out, since there's no
+    # Railtie to send it once the app has booted; under Rails the Railtie sends it instead, after
+    # initialization, when Active Record can report the migration version.
     def configure
       yield configuration
+      ChangeSnapshot.start(configuration) unless defined?(::Rails::Railtie)
+    end
+
+    # Records one thing that changed in a running system, e.g.
+    # `ForgeOpsTracker.record_change(kind: "feature_flag", title: "Enabled new_checkout for 10%")`,
+    # so ForgeOps can show it next to the errors and slowdowns that followed. kind is one of
+    # Change::KINDS (anything else is sent as "other"); environment defaults to the configured one.
+    # Delivered from a background thread like error events, so it never blocks the caller, and a
+    # no-op when the gem isn't enabled. Never raises: returns false when nothing was queued.
+    def record_change(kind:, title:, details: {}, environment: nil, service: nil, actor: nil, url: nil, id: nil, occurred_at: nil)
+      return false unless configuration.enabled?
+
+      payload = Change.build(
+        configuration, kind: kind, title: title, details: details, environment: environment,
+        service: service, actor: actor, url: url, id: id, occurred_at: occurred_at
+      )
+      return false unless payload
+
+      change_queue.push(payload)
+    rescue StandardError => e
+      configuration.logger&.debug { "[ForgeOpsTracker] record_change failed: #{e.class}: #{e.message}" }
+      false
     end
 
     # Records a named business metric (a signup, a payment, anything a customer wants to name),
@@ -101,12 +129,18 @@ module ForgeOpsTracker
     # a standalone buffer for itself: a lone span with no request-level trace around it, and no
     # middleware left running to ever flush it, would just accumulate in this thread's memory
     # forever with nothing to ever send it, which is worse than never recording it at all.
-    def span(name, kind: "service", data: {})
+    #
+    # A span with kind: "database" can also carry the SQL it ran and which database it ran on, e.g.
+    # `ForgeOpsTracker.span("Load orders", kind: "database", statement: sql, db_system: "postgresql")`.
+    # The statement is masked by SqlStatement before it's kept (every string and number becomes
+    # "?"), so no value it was run with is ever sent. Both are ignored on any other kind of span.
+    def span(name, kind: "service", data: {}, statement: nil, db_system: nil)
       return yield unless configuration.enabled? && configuration.track_tracing
 
       buffer = Thread.current[:forge_ops_tracker_spans]
       return yield unless buffer
 
+      data = database_span_data(data, statement, db_system) if kind.to_s == "database"
       span_id = SecureRandom.hex(8)
       started_at = Time.now.utc
       buffer.push(span_id)
@@ -120,6 +154,18 @@ module ForgeOpsTracker
     end
 
     private
+      def database_span_data(data, statement, db_system)
+        extra = {
+          "db.statement" => (SqlStatement.mask(statement.to_s) unless statement.nil?),
+          "db.system" => (db_system.to_s.strip.downcase unless db_system.nil? || db_system.to_s.strip.empty?)
+        }.compact
+        extra.empty? ? data : data.merge(extra)
+      end
+
+      def change_queue
+        @change_queue ||= DeliveryQueue.new(configuration, deliver_with: :deliver_change)
+      end
+
       def metric_buffer
         @metric_buffer ||= MetricBuffer.new(configuration)
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forge_ops_tracker
 version: !ruby/object:Gem::Version
-  version: 0.12.0
+  version: 0.14.0
 platform: ruby
 authors:
 - ForgeOps
@@ -90,6 +90,8 @@ files:
 - README.md
 - lib/forge_ops_tracker.rb
 - lib/forge_ops_tracker/breadcrumb_buffer.rb
+- lib/forge_ops_tracker/change.rb
+- lib/forge_ops_tracker/change_snapshot.rb
 - lib/forge_ops_tracker/client.rb
 - lib/forge_ops_tracker/configuration.rb
 - lib/forge_ops_tracker/delivery_queue.rb
@@ -114,6 +116,7 @@ files:
 - lib/forge_ops_tracker/performance_instrumentation.rb
 - lib/forge_ops_tracker/periodic_poller.rb
 - lib/forge_ops_tracker/pii_scrubber.rb
+- lib/forge_ops_tracker/query_explainer.rb
 - lib/forge_ops_tracker/railtie.rb
 - lib/forge_ops_tracker/request_state.rb
 - lib/forge_ops_tracker/session_flusher.rb