forge_ops_tracker 0.12.0 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70707edca2368561ce9b785b91dc9e8a26280071ec3db9ed34fa71db6c402fcf
-  data.tar.gz: a6fc1013df0404e46e0683749f2cedd688ac30273bda4ac113a7dd6334a19430
+  metadata.gz: ccbeed25a3172994d20455c19fa79a8336f3c73324ed667fd2f31d2cc05a19f5
+  data.tar.gz: 3cb7e1562daf2ccb8f09226bc915b7f00f6e8486ebbf361f8adf0a16c9851afe
 SHA512:
-  metadata.gz: 420ff144de5e84636348c245fef836cf240c94fe6bccb49217f8d51455a3c08ce81c18dadcf7260ccc46f35dd7b2470d809b1eecd92d9bea70e17ecf149f1a78
-  data.tar.gz: 74fc60b3d7451827618e0d63deef8cd6563d29eab144c8556bc912e91bbb4a7c8467110ae25940a7e4a7457db8718cad8b763ff752311014ed325b22ce5c13b0
+  metadata.gz: 6287470f8bc768e95d1d1afe55f2c7a0e640a4d66ded8a0e3c518e0248031a7f5f76621adf0436d2de3bc7da029733b9debab07e31d2f1b37c7867cfd3aee7e5
+  data.tar.gz: b7c996fbf89c2c0a90f6c39c848542cde734242a3fa4efaf450a36b1e7362815322417ceafc89d05a71a453a9965d7b4ab0c4deb2eedad0fdafd6b21808ad790

data/CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## 0.13.0 (2026-09-25)
+
+- New `ForgeOpsTracker.record_change(kind:, title:, details: {}, environment: nil, service: nil,
+  actor: nil, url: nil, id: nil, occurred_at: nil)` records something that changed in your system
+  (a feature flag, a config value, a hand-run migration) so ForgeOps can show it next to the errors
+  that followed. `kind` is one of `feature_flag`, `config`, `migration`, `dependency`,
+  `infrastructure`, or `other`; anything else is sent as `other`. Sent from a background thread,
+  never raises, and a no-op wherever reporting is off.
+- Changes between deploys are now detected automatically. Once per process, after Rails finishes
+  booting (or on `ForgeOpsTracker.configure` outside Rails), a background thread sends a snapshot of
+  the Ruby version, the gem versions in `Gemfile.lock`, and the current Active Record migration
+  version, and ForgeOps records whatever changed since the previous boot. New
+  `config.detect_changes` (default true) turns this off.
+- New `config.track_env_var_names` (default false) adds environment variable names, never values,
+  to that snapshot. Host-specific names (`HOSTNAME`, `PATH`, `PORT`, `LC_*`, Kubernetes service
+  variables, and others) and the gem's own `FORGE_OPS_*` settings are always left out.
+
 ## 0.12.0
 
 - Every error event reported during a request now says where it happened: `transaction_name` (the

data/README.md

@@ -411,6 +411,52 @@ Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn
 simply rejected server-side and dropped, exactly like any other delivery failure. The `trace_id`
 on error events is sent either way.
 
+## What changed
+
+ForgeOps can show what changed in your system next to the errors and slowdowns that followed it.
+Two ways in:
+
+**Record a change yourself** when something changes that no deploy captures, like a feature flag
+flipped, a config value edited, or a migration run by hand:
+
+```ruby
+ForgeOpsTracker.record_change(
+  kind: "feature_flag", # feature_flag, config, migration, dependency, infrastructure, or other
+  title: "Enabled new_checkout for 10% of users",
+  details: { flag: "new_checkout", rollout_percent: 10 },
+  actor: current_user.email,
+  url: "https://flags.example.com/new_checkout"
+)
+```
+
+`kind` and `title` are required; `details`, `environment` (defaults to `config.environment`),
+`service`, `actor`, `url`, `id` (an idempotency key, so sending the same change twice records it
+once), and `occurred_at` (defaults to now) are optional. An unknown `kind` is sent as `other`. It's
+queued and sent from a background thread, so it never slows down the caller, never raises, and is
+a no-op wherever reporting is off.
+
+**Changes between deploys are detected for you.** Once per process, right after Rails finishes
+booting, a background thread sends ForgeOps a snapshot of what the app is running: the Ruby
+version, every gem version in your `Gemfile.lock`, and the current Active Record migration version
+(left out if the database can't be reached). ForgeOps compares it with the previous boot's and
+records whatever changed, such as a gem upgrade or a new migration. Outside Rails the snapshot goes
+out when you call `ForgeOpsTracker.configure`, without the migration version.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.detect_changes = true       # default; false sends no startup snapshot
+  config.track_env_var_names = false # default; true also sends environment variable names
+end
+```
+
+With `track_env_var_names` on, the snapshot lists the names of your environment variables (never
+their values), so an added or removed variable shows up as a change. Names that differ from host to
+host, like `HOSTNAME`, `PATH`, `PORT`, `LC_*`, and Kubernetes service variables, are left out, as
+are the gem's own `FORGE_OPS_*` settings.
+
+Requires a ForgeOps plan that includes change tracking; on a plan that doesn't, both are rejected
+server-side and dropped, exactly like any other delivery failure.
+
 ## Custom metrics
 
 Unlike session/performance tracking above, there's no automatic instrumentation here at all: a

data/lib/forge_ops_tracker/change.rb

@@ -0,0 +1,47 @@
+require "time"
+
+module ForgeOpsTracker
+  # Builds the body for one ForgeOpsTracker.record_change call (POST /api/v1/changes): something
+  # that changed in a running system (a feature flag flipped, a config value edited, a migration
+  # run by hand), so ForgeOps can line it up against the errors and slowdowns around it.
+  module Change
+    # The server rejects any other kind outright (422), so an unknown one is sent as "other"
+    # rather than dropped: the change still gets recorded, just less specifically categorized.
+    KINDS = %w[feature_flag config migration dependency infrastructure other].freeze
+    MAX_TITLE_LENGTH = 200
+
+    module_function
+
+    # Returns nil when there's nothing sendable (a blank title): the server requires one, so
+    # posting it anyway would only ever come back 422.
+    def build(configuration, kind:, title:, details: {}, environment: nil, service: nil, actor: nil, url: nil, id: nil, occurred_at: nil)
+      title = title.to_s.strip
+      return nil if title.empty?
+
+      {
+        kind: normalize_kind(kind),
+        title: title[0, MAX_TITLE_LENGTH],
+        details: details.is_a?(Hash) ? details : {},
+        environment: (environment || configuration.environment)&.to_s,
+        service: service&.to_s,
+        actor: actor&.to_s,
+        url: url&.to_s,
+        id: id&.to_s,
+        occurred_at: iso8601(occurred_at || Time.now)
+      }.compact
+    end
+
+    def normalize_kind(kind)
+      kind = kind.to_s
+      KINDS.include?(kind) ? kind : "other"
+    end
+
+    def iso8601(value)
+      case value
+      when Time then value.utc.iso8601(3)
+      when String then value
+      else value.respond_to?(:to_time) ? value.to_time.utc.iso8601(3) : Time.now.utc.iso8601(3)
+      end
+    end
+  end
+end

data/lib/forge_ops_tracker/change_snapshot.rb

@@ -0,0 +1,133 @@
+require "thread"
+
+module ForgeOpsTracker
+  # Tells ForgeOps what this process sees at startup (the Ruby version, the locked gem versions, the
+  # current Active Record migration version, and optionally the names of its environment variables),
+  # once per process, from a background thread. The server diffs it against the last snapshot for
+  # the same project and environment and records whatever changed as a change, so this gem stays
+  # stateless: it never remembers what it sent last time.
+  #
+  # Any key that can't be determined reliably is left out rather than guessed at, since the server
+  # reads a missing key as "unknown", never as "everything was removed".
+  class ChangeSnapshot
+    MAX_DEPENDENCIES = 3000
+    MAX_NAME_LENGTH = 200
+    MAX_VERSION_LENGTH = 100
+    MAX_ENV_VAR_NAMES = 3000
+
+    # Host-specific variables that differ from one box to the next (or one boot to the next) without
+    # anything about the deploy having changed, so a fleet doesn't look like it's changing on every
+    # restart. The gem's own FORGE_OPS_* settings are dropped too.
+    ENV_VAR_DENYLIST = %w[
+      HOSTNAME HOST HOME PATH PWD OLDPWD SHLVL _ TERM USER LOGNAME SHELL LANG TMPDIR TZ PORT DYNO
+      INVOCATION_ID JOURNAL_STREAM
+    ].freeze
+    ENV_VAR_DENYLIST_PATTERNS = [
+      /\ALC_/, /\ASYSTEMD_/, /\AMEMORY_PRESSURE_/, /\AKUBERNETES_/, /\AFORGE_OPS_/,
+      /_SERVICE_HOST\z/, /_SERVICE_PORT/, /_PORT_.*_TCP/
+    ].freeze
+
+    MUTEX = Mutex.new
+
+    class << self
+      # Starts the one snapshot this process sends, on its own thread so it never delays boot. A
+      # second call is a no-op, so the Railtie and a plain ForgeOpsTracker.configure can't both
+      # send one. Does nothing (and doesn't use up the once) when the gem isn't enabled or
+      # detect_changes is off. Never raises.
+      def start(configuration)
+        return false unless configuration.enabled? && configuration.detect_changes
+
+        MUTEX.synchronize do
+          return false if @started
+
+          @started = true
+        end
+
+        thread = Thread.new { new(configuration).deliver }
+        thread.report_on_exception = false
+        true
+      rescue StandardError
+        false
+      end
+
+      # Names only, never values, sorted so the same set always serializes the same way.
+      def env_var_names(env = ENV)
+        env.keys.reject { |name| denied_env_var?(name) }.sort.first(MAX_ENV_VAR_NAMES)
+      end
+
+      def denied_env_var?(name)
+        ENV_VAR_DENYLIST.include?(name) || ENV_VAR_DENYLIST_PATTERNS.any? { |pattern| pattern.match?(name) }
+      end
+    end
+
+    def initialize(configuration, client: Client.new(configuration))
+      @configuration = configuration
+      @client = client
+    end
+
+    def deliver
+      client.deliver_change_snapshot(payload)
+    rescue StandardError => e
+      configuration.logger&.debug { "[ForgeOpsTracker] change snapshot failed: #{e.class}: #{e.message}" }
+      false
+    end
+
+    def payload
+      {
+        environment: configuration.environment.to_s,
+        state: {
+          schema_version: schema_version,
+          runtime: runtime,
+          dependencies: dependencies,
+          env_var_names: configuration.track_env_var_names ? self.class.env_var_names : nil
+        }.compact
+      }
+    end
+
+    def runtime
+      "#{RUBY_ENGINE} #{RUBY_ENGINE_VERSION}"
+    end
+
+    # The app's Gemfile.lock (through Bundler, so it's the exact resolved versions), falling back to
+    # whatever gems are actually activated when there's no bundle at all. nil, and so left out,
+    # when neither is available.
+    def dependencies
+      specs = bundler_specs || Gem.loaded_specs.values
+      return nil if specs.empty?
+
+      specs.each_with_object({}) do |spec, deps|
+        deps[spec.name.to_s[0, MAX_NAME_LENGTH]] = spec.version.to_s[0, MAX_VERSION_LENGTH]
+      end.sort.first(MAX_DEPENDENCIES).to_h
+    rescue StandardError
+      nil
+    end
+
+    # The newest migration Active Record has run against this app's database, as a string. Left
+    # out when Active Record isn't loaded, the database can't be reached, or nothing has been
+    # migrated yet: a snapshot must never fail (or fail boot) because the database is down.
+    def schema_version
+      return nil unless defined?(::ActiveRecord::Base)
+
+      version = ::ActiveRecord::Base.connection_pool.with_connection do |connection|
+        pool = ::ActiveRecord::Base.connection_pool
+        context = pool.respond_to?(:migration_context) ? pool.migration_context : connection.migration_context
+        context.current_version
+      end
+      version.to_i.positive? ? version.to_s : nil
+    rescue StandardError, ScriptError
+      nil
+    end
+
+    private
+      attr_reader :configuration, :client
+
+      def bundler_specs
+        return nil unless defined?(::Bundler)
+
+        specs = ::Bundler.locked_gems&.specs
+        specs.nil? || specs.empty? ? nil : specs
+      rescue StandardError
+        nil
+      end
+  end
+end

data/lib/forge_ops_tracker/client.rb

@@ -60,6 +60,17 @@ module ForgeOpsTracker
       post(configuration.spans_uri, { trace_id: trace_id, spans: spans })
     end
 
+    # Same delivery contract again; see Configuration#changes_uri and ForgeOpsTracker.record_change.
+    # One change per call. A plan without change tracking answers 403, which is just a false here.
+    def deliver_change(payload)
+      post(configuration.changes_uri, payload)
+    end
+
+    # Same again; see Configuration#change_snapshots_uri and ChangeSnapshot.
+    def deliver_change_snapshot(payload)
+      post(configuration.change_snapshots_uri, payload)
+    end
+
     private
       attr_reader :configuration
 

data/lib/forge_ops_tracker/configuration.rb

@@ -18,6 +18,7 @@ module ForgeOpsTracker
     attr_accessor :gauge_poll_interval
     attr_accessor :track_tracing, :trace_capture_threshold_ms
     attr_accessor :propagate_traces, :trace_propagation_targets
+    attr_accessor :detect_changes, :track_env_var_names
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -138,6 +139,15 @@ module ForgeOpsTracker
       # Regexp is matched against the host. Worth narrowing for an app that calls third-party APIs
       # which reject unknown headers, or that shouldn't learn this app's trace ids at all.
       @trace_propagation_targets = nil
+      # Sends one ChangeSnapshot per process at startup (Ruby version, locked gem versions, the
+      # current migration version), which ForgeOps diffs against the previous boot's to record what
+      # changed between deploys. On by default, like every other automatic behavior here.
+      @detect_changes = true
+      # Whether that snapshot also lists the names of this process's environment variables, so an
+      # added or removed variable shows up as a change. Off by default: names only, never values,
+      # but even names can describe more about an app than some teams want to share. See
+      # ChangeSnapshot::ENV_VAR_DENYLIST for the host-specific names that are always left out.
+      @track_env_var_names = false
     end
 
     # Whether an outbound request to host should carry a traceparent header; see
@@ -231,11 +241,30 @@ module ForgeOpsTracker
       uri
     end
 
+    def changes_uri
+      sibling_uri("/changes")
+    end
+
+    def change_snapshots_uri
+      sibling_uri("/change_snapshots")
+    end
+
     def enabled?
       !blank?(dsn) && !blank?(api_key) && enabled_environments.map(&:to_s).include?(environment.to_s)
     end
 
     private
+      # Same substitution every *_uri method above spells out by hand: a sibling path under the
+      # DSN's own /api/v1/events.
+      def sibling_uri(path)
+        uri = ingestion_uri
+        return nil unless uri
+
+        uri = uri.dup
+        uri.path = uri.path.sub(%r{/events\z}, path)
+        uri
+      end
+
       def parsed_dsn
         return nil if blank?(dsn)
 

data/lib/forge_ops_tracker/delivery_queue.rb

@@ -8,9 +8,12 @@ module ForgeOpsTracker
   # in each forked Puma/Passenger worker process rather than being carried
   # across a fork, which would leave it dead in the child.
   class DeliveryQueue
-    def initialize(configuration, client: Client.new(configuration))
+    # deliver_with names the Client method each payload goes through: :deliver (error events) by
+    # default, :deliver_change for ForgeOpsTracker.record_change's own queue.
+    def initialize(configuration, client: Client.new(configuration), deliver_with: :deliver)
       @configuration = configuration
       @client = client
+      @deliver_with = deliver_with
       @queue = SizedQueue.new(configuration.queue_size)
       @mutex = Mutex.new
       @thread = nil
@@ -44,7 +47,7 @@ module ForgeOpsTracker
 
       def run
         loop do
-          client.deliver(@queue.pop)
+          client.public_send(@deliver_with, @queue.pop)
         rescue StandardError => e
           log { "delivery thread error: #{e.class}: #{e.message}" }
         end

data/lib/forge_ops_tracker/railtie.rb

@@ -305,6 +305,14 @@ module ForgeOpsTracker
     # sidekiq, ActiveJob's redis queue adapter, ...), which this gem has no control over; deferring
     # the defined? check to after every railtie (including this app's own dependencies) has
     # finished initializing avoids a load-order-dependent false negative.
+    # After initialization, not in a plain initializer, so the app's own
+    # config/initializers/forge_ops_tracker.rb has already run (detect_changes/track_env_var_names
+    # and the DSN are final) and Active Record is configured enough to report the migration
+    # version. ChangeSnapshot does the actual work on its own thread, so boot never waits on it.
+    config.after_initialize do
+      ForgeOpsTracker::ChangeSnapshot.start(ForgeOpsTracker.configuration)
+    end
+
     config.after_initialize do
       next unless defined?(::RedisClient)
 

data/lib/forge_ops_tracker/version.rb

@@ -1,3 +1,3 @@
 module ForgeOpsTracker
-  VERSION = "0.12.0"
+  VERSION = "0.13.0"
 end

data/lib/forge_ops_tracker.rb

@@ -6,6 +6,8 @@ require "forge_ops_tracker/sql_statement"
 require "forge_ops_tracker/event_builder"
 require "forge_ops_tracker/client"
 require "forge_ops_tracker/delivery_queue"
+require "forge_ops_tracker/change"
+require "forge_ops_tracker/change_snapshot"
 require "forge_ops_tracker/error_subscriber"
 require "forge_ops_tracker/session_flusher"
 require "forge_ops_tracker/histogram_bucketer"
@@ -32,8 +34,33 @@ module ForgeOpsTracker
       @configuration ||= Configuration.new
     end
 
+    # Outside Rails this is also where the startup ChangeSnapshot goes out, since there's no
+    # Railtie to send it once the app has booted; under Rails the Railtie sends it instead, after
+    # initialization, when Active Record can report the migration version.
     def configure
       yield configuration
+      ChangeSnapshot.start(configuration) unless defined?(::Rails::Railtie)
+    end
+
+    # Records one thing that changed in a running system, e.g.
+    # `ForgeOpsTracker.record_change(kind: "feature_flag", title: "Enabled new_checkout for 10%")`,
+    # so ForgeOps can show it next to the errors and slowdowns that followed. kind is one of
+    # Change::KINDS (anything else is sent as "other"); environment defaults to the configured one.
+    # Delivered from a background thread like error events, so it never blocks the caller, and a
+    # no-op when the gem isn't enabled. Never raises: returns false when nothing was queued.
+    def record_change(kind:, title:, details: {}, environment: nil, service: nil, actor: nil, url: nil, id: nil, occurred_at: nil)
+      return false unless configuration.enabled?
+
+      payload = Change.build(
+        configuration, kind: kind, title: title, details: details, environment: environment,
+        service: service, actor: actor, url: url, id: id, occurred_at: occurred_at
+      )
+      return false unless payload
+
+      change_queue.push(payload)
+    rescue StandardError => e
+      configuration.logger&.debug { "[ForgeOpsTracker] record_change failed: #{e.class}: #{e.message}" }
+      false
     end
 
     # Records a named business metric (a signup, a payment, anything a customer wants to name),
@@ -120,6 +147,10 @@ module ForgeOpsTracker
     end
 
     private
+      def change_queue
+        @change_queue ||= DeliveryQueue.new(configuration, deliver_with: :deliver_change)
+      end
+
       def metric_buffer
         @metric_buffer ||= MetricBuffer.new(configuration)
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forge_ops_tracker
 version: !ruby/object:Gem::Version
-  version: 0.12.0
+  version: 0.13.0
 platform: ruby
 authors:
 - ForgeOps
@@ -90,6 +90,8 @@ files:
 - README.md
 - lib/forge_ops_tracker.rb
 - lib/forge_ops_tracker/breadcrumb_buffer.rb
+- lib/forge_ops_tracker/change.rb
+- lib/forge_ops_tracker/change_snapshot.rb
 - lib/forge_ops_tracker/client.rb
 - lib/forge_ops_tracker/configuration.rb
 - lib/forge_ops_tracker/delivery_queue.rb