necropsy 0.1.0 → 0.3.0

data/docs/necropsy_performance_adr.md

@@ -0,0 +1,43 @@
+# Performance profiling and cache decision
+
+## Decision
+
+Necropsy exposes an opt-in phase profiler through `Necropsy.analyze(profile: true)` and
+records wall time, allocations, RSS, graph counts, resolution-cache hits, and report/index
+size. The benchmark runner remains the release source for pinned corpus budgets.
+
+The existing scan cache now keys every input file by SHA-256 content digest and writes through
+a temporary file followed by rename. A same-size or restored-mtime edit therefore cannot reuse
+stale facts, and an interrupted/corrupt write falls back to a fresh scan.
+
+## PERF-02 no-go for now: per-file facts
+
+The first measurement on the maintained sample fixture showed the analyzer phases dominating
+the run (name resolution, CHA, and RTA) while the scan phase was smaller. That does not justify
+the complexity and invalidation surface of a per-file fact store yet. The current cache keeps a
+single deterministic scan result; a per-file cache should be proposed only after a pinned,
+representative corpus demonstrates parse/scan as a material cost.
+
+## Reconsideration threshold
+
+Revisit per-file facts when the benchmark profile shows parse/scan as the largest phase or at
+least 40% of wall time across two representative corpora, with a warm-run improvement target of
+20% and no fresh-scan parity differences.
+
+## PERF-03 no-go for now: process-parallel parsing
+
+Process-parallel Prism parsing adds worker startup, serialization, deterministic ordinal merge,
+crash recovery, and cache invalidation surfaces while the profiled scan phase is not dominant.
+Thread parallelism is not a substitute because the supported Ruby engines do not provide the same
+execution guarantees. Reconsider only after PERF-02's 40% scan threshold is met and a prototype
+shows at least 25% cold-run improvement on two corpora with byte-identical reports and identical
+health under worker failure.
+
+## PERF-04 no-go for now: template/reference fact cache
+
+Reference files already participate in the source snapshot and scan-cache key, but the conservative
+reference barrier intentionally reruns after candidate generation. Persisting candidate-specific
+matches would couple cache entries to physical identities, thresholds, parser policy, and ignored
+artifact paths. Reconsider when reference/template analysis is at least 20% of p95 wall time on two
+corpora and a prototype demonstrates byte-identical blockers across edits, budget exhaustion, and
+generated-file policy changes.

data/docs/necropsy_scope_decisions.md

@@ -0,0 +1,58 @@
+# Scope decisions from the 148-item adversarial review
+
+This record covers proposals whose desired safety outcome is already met by a smaller mechanism,
+or whose literal implementation would weaken the reviewed removal contract. A no-go is a completed
+decision with a measurable reconsideration gate, not an untracked deferral.
+
+## Definition activation and root-seeded RTA (#44, #52)
+
+Necropsy does not emit `shadowed_definition`. Ruby activation depends on load order, conditional
+reopens, `remove_method`, `undef_method`, and eval. Duplicate physical definitions remain visible
+and blocked unless an evidenced load/activation witness closes those variables. Reconsider the
+classification only when an activation oracle corpus covers those operations with zero
+known-positive loss.
+
+Root-seeded fixed-point RTA is also not enabled. Default RTA is rank-only and cannot remove an edge;
+legacy pruning makes analysis health invalid. A second fixed point would therefore add complexity
+without improving the safety claim. Reconsider it as a precision experiment only if two reviewed
+corpora show at least a 10% blocked-candidate reduction with unchanged known-positive recall.
+
+## Clusters, frontier, and a separate activation graph (#69, #70, #72)
+
+The primary review identity remains a physical definition because baseline, quarantine, SARIF, and
+source edits operate at that granularity. Collapsing cycles into removal units can hide a mixed-risk
+member, while the current `why-not` payload already exposes witnesses, incoming sites, blockers, and
+the next evidence for each definition. Literal load edges are represented as typed evidence and
+unrooted load units are diagnosed without maintaining a second mutable graph.
+
+Reconsider cluster/frontier presentation after a review-time study demonstrates at least a 20%
+reduction in median triage time without missed mixed-risk definitions. Split the activation graph
+only if a new analysis requires state that cannot be represented by typed load evidence.
+
+## Template/reference parsers and generated files (#81, #90, #95)
+
+Haml, Slim, Jbuilder, Builder, YAML, JSON, TOML, GraphQL SDL, SQL, and OpenAPI do not share a sound
+notion of “method reference.” A token adapter branded as a parser would create unjustified negative
+evidence. The current generic barrier only adds blockers; unreadable, oversized, generated, or
+budget-exhausted runtime inputs add a global blocker and degraded health. ERB is the exception
+because executable Ruby regions can be extracted and parsed by Prism.
+
+`trusted_generated: true` is rejected: provenance does not prove that generated output cannot name
+a runtime callback. Reconsider a format adapter only with a versioned grammar, conformance corpus,
+and adversarial false-negative tests. Reconsider a generated bypass only when its producing input
+is scanned and a content-addressed derivation proves reference equivalence.
+
+## Presenter conventions (#85)
+
+ActiveModelSerializers, Blueprinter, and ViewComponent have declared runtime contracts and use the
+shared `ConventionRules` schema. “Presenter” alone has no gem-independent invocation contract, so
+rooting every `*Presenter` public method would erase useful candidates. Projects can declare their
+actual hook ancestry through `implicit_callers`. Add a built-in presenter rule only for a named gem
+with documented dispatch semantics and a reviewed fixture.
+
+## Compatibility producer versions (#115)
+
+All current producers emit a concrete version. Legacy custom analyzers are normalized to the stable
+literal `unversioned`; breaking their construction contract inside schema v2 would not improve the
+claim because capability validation and atomic staging still gate complete resolution. Make profile
+version mandatory only in the next major analyzer contract, with a migration reader for v2.

data/docs/necropsy_type_facts_adr.md

@@ -0,0 +1,22 @@
+# TYPE-01 decision record
+
+Status: no-go for external and interprocedural type providers in the removal decision.
+
+The 0.4 implementation keeps a small `TypeFact` value object and an empty provider profile so that optional type evidence has a stable boundary. No Sorbet or RBS parser is enabled by default, and no type fact can remove a target unless it is explicitly authoritative and complete. Hints and conflicting facts remain explanatory evidence only.
+
+The repository contains syntax fixtures with `.rbs` files, but those fixtures are not a reviewed type corpus: they do not label receiver facts, expected call targets, stale signatures, conflicting reopenings, or generated RBI/RBS behavior. Treating their existence as evidence for a provider would therefore be circular.
+
+The same decision covers demand-driven points-to, interprocedural return and argument propagation,
+constructor instance-variable summaries, and cross-load-unit constant facts. Those analyses can add
+ranking evidence in an experiment, but cannot remove a conservative target until their open-world,
+mutation, aliasing, and load-order assumptions are represented as claims. The existing local finite
+flow remains bounded and fails closed; Proc/lambda values are local only.
+
+## Reconsideration gate
+
+Pin at least two reviewed RBS projects and two reviewed Sorbet projects. Each must label static call
+targets and stale/conflicting type declarations at physical-definition granularity. Compare the
+provider on/off for candidate precision, known-positive recall, blocked reduction, wall time, p95
+RSS, and failure health. A provider may ship as hint-only when it improves a primary metric without
+reducing recall. It may participate in complete resolution only after adversarial stale-signature,
+dynamic-reopen, and load-order mutations preserve every known-positive target.

data/gemfiles/prism_min.gemfile

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+gemspec path: '..'
+
+gem 'prism', '~> 1.0.0'
+gem 'rake', '~> 13.0'
+gem 'rspec', '~> 3.0'

data/gemfiles/prism_min.gemfile.lock

@@ -0,0 +1,50 @@
+PATH
+  remote: ..
+  specs:
+    necropsy (0.3.0)
+      prism (>= 1.0, < 2.0)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    diff-lcs (1.6.2)
+    prism (1.0.0)
+    rake (13.4.2)
+    rspec (3.13.2)
+      rspec-core (~> 3.13.0)
+      rspec-expectations (~> 3.13.0)
+      rspec-mocks (~> 3.13.0)
+    rspec-core (3.13.6)
+      rspec-support (~> 3.13.0)
+    rspec-expectations (3.13.5)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.13.0)
+    rspec-mocks (3.13.8)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.13.0)
+    rspec-support (3.13.7)
+
+PLATFORMS
+  arm64-darwin-24
+  ruby
+
+DEPENDENCIES
+  necropsy!
+  prism (~> 1.0.0)
+  rake (~> 13.0)
+  rspec (~> 3.0)
+
+CHECKSUMS
+  bundler (4.0.13) sha256=19f08be7f27022cf0b89f27da0b044ae075e8270a9ef44ad248a932614e1ca3b
+  diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962
+  necropsy (0.3.0)
+  prism (1.0.0) sha256=cc20cb9b52c2530a6a94f5fb532349ca7a5dca9c32e43bb7cef0cc7b1834db70
+  rake (13.4.2) sha256=cb825b2bd5f1f8e91ca37bddb4b9aaf345551b4731da62949be002fa89283701
+  rspec (3.13.2) sha256=206284a08ad798e61f86d7ca3e376718d52c0bc944626b2349266f239f820587
+  rspec-core (3.13.6) sha256=a8823c6411667b60a8bca135364351dda34cd55e44ff94c4be4633b37d828b2d
+  rspec-expectations (3.13.5) sha256=33a4d3a1d95060aea4c94e9f237030a8f9eae5615e9bd85718fe3a09e4b58836
+  rspec-mocks (3.13.8) sha256=086ad3d3d17533f4237643de0b5c42f04b66348c28bf6b9c2d3f4a3b01af1d47
+  rspec-support (3.13.7) sha256=0640e5570872aafefd79867901deeeeb40b0c9875a36b983d85f54fb7381c47c
+
+BUNDLED WITH
+  4.0.13

data/lib/necropsy/analyzer.rb

@@ -1,6 +1,69 @@
 # frozen_string_literal: true
 
+require 'digest'
+require_relative 'bounded_canonicalizer'
+
 module Necropsy
+  module EvidenceIdentity
+    module_function
+
+    def generate(attributes)
+      "evidence:v1:#{Digest::SHA256.hexdigest(BoundedCanonicalizer.dump(attributes))}"
+    end
+
+    # Analyzer-produced records are assembled from trusted scalar/model data.
+    # Sorting ordinary Hash keys and using the JSON encoder avoids the
+    # allocation-heavy type-tag/hex walk needed for untrusted legacy payloads.
+    # EvidenceStore still canonicalizes the final payload and quarantines any
+    # accidental fast-path collision.
+    def generate_fast(attributes)
+      "evidence:v1:#{Digest::SHA256.hexdigest(JSON.generate(fast_payload(attributes)))}"
+    rescue JSON::GeneratorError, TypeError, SystemStackError
+      generate(attributes)
+    end
+
+    def fast_payload(value)
+      case value
+      when Hash
+        value.keys.sort_by(&:to_s).to_h { |key| [key.to_s, fast_payload(value.fetch(key))] }
+      when Array
+        value.map { |item| fast_payload(item) }
+      when Symbol
+        value.to_s
+      else
+        value.respond_to?(:to_h) ? fast_payload(value.to_h) : value
+      end
+    end
+    private_class_method :fast_payload
+  end
+  private_constant :EvidenceIdentity
+
+  module EvidenceCollection
+    module_function
+
+    def collect(*collections)
+      records = collections.flatten.compact.map { |item| unwrap(item) }
+      records.uniq { |record| identity(record) }
+             .sort_by { |record| identity(record) }
+             .freeze
+    end
+
+    def unwrap(item)
+      item.respond_to?(:evidence) ? item.evidence : item
+    end
+    private_class_method :unwrap
+
+    def identity(record)
+      evidence_id = record.evidence_id if record.respond_to?(:evidence_id)
+      return evidence_id if evidence_id
+
+      payload = record.respond_to?(:to_h) ? record.to_h.except('evidence_id') : record
+      EvidenceIdentity.generate(payload)
+    end
+    private_class_method :identity
+  end
+  private_constant :EvidenceCollection
+
   class Analyzer
     def analyze(_graph, _project)
       raise NotImplementedError, "#{self.class} must implement #analyze"
@@ -10,10 +73,66 @@ module Necropsy
       raise NotImplementedError, "#{self.class} must implement #profile"
     end
 
+    def capabilities
+      [].freeze
+    end
+
     private
 
-    def evidence(kind:, details:, analyzer: profile.name, weight: 1.0, metadata: {})
-      Evidence.new(analyzer: analyzer, kind: kind, weight: weight, details: details, metadata: metadata)
+    def evidence(kind:, details:, analyzer: nil, weight: 1.0, metadata: {}, producer: nil, producer_version: nil,
+                 grade: :heuristic, relation: nil, source: nil, assumptions: nil, scope: nil)
+      analyzer_profile = profile
+      analyzer ||= analyzer_profile.name
+      producer ||= analyzer
+      producer_version ||= analyzer_profile.version
+      relation ||= kind
+      assumptions = analyzer_profile.assumptions if assumptions.nil?
+      producer_version ||= 'unversioned'
+      source ||= { 'type' => 'analyzer', 'producer' => producer.to_s }
+      scope ||= {}
+      record = Evidence.new(
+        analyzer: analyzer, kind: kind, weight: weight, details: details, metadata: metadata,
+        producer: producer, producer_version: producer_version, grade: grade, relation: relation,
+        source: source, assumptions: assumptions, scope: scope
+      )
+      evidence_id = EvidenceIdentity.generate_fast(record.to_h.except('evidence_id'))
+      record.with(evidence_id: evidence_id)
+    end
+
+    def result_evidences(*collections)
+      EvidenceCollection.collect(*collections)
+    end
+
+    def resolution_record(site, targets, evidences, status: nil, rejected_targets: [], unknown_scope: nil)
+      analyzer_profile = profile
+      target_ids = targets.map(&:graph_id).uniq.sort
+      status ||= target_ids.empty? ? :unknown : :partial
+      unknown_scope = status == :complete ? nil : (unknown_scope || residual_scope(site))
+      ResolutionRecord.new(
+        resolution: Resolution.new(
+          call_site_id: site.call_site_id,
+          target_definition_ids: target_ids,
+          status: status,
+          unknown_scope: unknown_scope,
+          rejected_targets: rejected_targets,
+          evidence_ids: result_evidences(evidences).filter_map(&:evidence_id)
+        ),
+        producer: analyzer_profile.name,
+        producer_version: analyzer_profile.version || 'unversioned',
+        assumptions: analyzer_profile.assumptions
+      )
+    end
+
+    def residual_scope(site)
+      UnknownScope.new(scope_kind: :message, scope_value: site.message, match: :exact)
+    end
+
+    def call_site_evidence_source(site)
+      { 'call_site_id' => site.call_site_id, 'file' => site.file, 'line' => site.line }
+    end
+
+    def call_site_evidence_scope(site)
+      { 'call_site_id' => site.call_site_id, 'caller_definition_id' => site.caller_id }
     end
   end
 end

data/lib/necropsy/analyzers/dynamic/coverage_collector.rb

@@ -4,41 +4,45 @@ require 'coverage'
 require 'fileutils'
 require 'time'
 require 'yaml'
+require_relative '../../clock'
+require_relative 'runtime_reference'
+require_relative 'observation_policy'
 
 module Necropsy
   module Analyzers
     module Dynamic
       class CoverageCollector
-        def self.record(root:, output:, &)
-          new(root: root, output: output).record(&)
+        def self.record(root:, output:, clock: nil, &)
+          new(root: root, output: output, clock: clock).record(&)
         end
 
-        def self.install_at_exit(root:, output:, merge: false, run_id: nil)
-          new(root: root, output: output, merge: merge, run_id: run_id).install_at_exit
+        def self.install_at_exit(root:, output:, merge: false, run_id: nil, clock: nil)
+          new(root: root, output: output, merge: merge, run_id: run_id, clock: clock).install_at_exit
         end
 
-        def initialize(root:, output:, merge: false, run_id: nil)
+        def initialize(root:, output:, merge: false, run_id: nil, clock: nil)
           @root = File.expand_path(root)
           @output = output
           @merge = merge
           @run_id = run_id
+          @clock = clock || -> { Clock.new.time }
         end
 
         def record
-          started_at = Time.now.utc
+          started_at = current_time
           Coverage.start(methods: true)
           yield
-          write_payload(result: Coverage.result, started_at: started_at, finished_at: Time.now.utc)
+          write_payload(result: Coverage.result, started_at: started_at, finished_at: current_time)
         ensure
           Coverage.result(stop: true, clear: true) if Coverage.running?
         end
 
         def install_at_exit
-          started_at = Time.now.utc
+          started_at = current_time
           started = start_coverage
 
           at_exit do
-            finished_at = Time.now.utc
+            finished_at = current_time
             result = coverage_result(started: started)
             write_payload(result: result, started_at: started_at, finished_at: finished_at)
           rescue StandardError => e
@@ -48,12 +52,16 @@ module Necropsy
 
         private
 
-        attr_reader :root, :output, :run_id
+        attr_reader :root, :output, :run_id, :clock
 
         def merge?
           @merge
         end
 
+        def current_time
+          clock.call.utc
+        end
+
         def start_coverage
           return false if Coverage.running?
 
@@ -63,19 +71,39 @@ module Necropsy
 
         def coverage_result(started:)
           return Coverage.result(stop: true, clear: true) if started && Coverage.running?
-          return Coverage.peek_result if Coverage.respond_to?(:peek_result) && Coverage.running?
+
+          if Coverage.respond_to?(:peek_result) && Coverage.running?
+            result = Coverage.peek_result
+            return result if method_coverage?(result)
+
+            warn 'Necropsy coverage collector found Coverage already running without methods: true; no methods were recorded.'
+          end
 
           {}
         end
 
+        def method_coverage?(result)
+          result.values.any? { |coverage| coverage.is_a?(Hash) && coverage.key?(:methods) }
+        end
+
         def write_payload(result:, started_at:, finished_at:)
+          references = executed_references(result)
           payload = {
-            'nodes' => executed_nodes(result).sort,
+            'schema_version' => 2,
+            'collector' => { 'name' => 'necropsy-coverage', 'version' => Necropsy::VERSION },
+            'scope' => { 'sample_unit' => 'process', 'sample_rate' => 1.0 },
+            'quality' => { 'dropped_events' => 0, 'overflowed' => false },
+            'nodes' => references.map { |reference| reference.fetch('symbol_id') }.uniq.sort,
+            'node_references' => references,
             'observation' => {
               'started_at' => started_at.iso8601,
               'finished_at' => finished_at.iso8601,
               'days' => [((finished_at - started_at) / 86_400.0).ceil, 1].max,
-              'collector' => 'coverage'
+              'collector' => 'coverage',
+              'collector_overhead' => {
+                'wall_time_seconds' => [finished_at - started_at, 0].max.round(6),
+                'observed_nodes' => references.length
+              }
             }.tap { |observation| observation['run_id'] = run_id if run_id }
           }
           FileUtils.mkdir_p(File.dirname(output))
@@ -99,7 +127,7 @@ module Necropsy
           content = file.read
           return {} if content.empty?
 
-          YAML.load(content) || {}
+          YAML.safe_load(content, aliases: false) || {}
         rescue Psych::Exception
           {}
         end
@@ -114,13 +142,19 @@ module Necropsy
         end
 
         def empty_payload?(payload)
-          Array(payload['nodes']).empty? && payload.fetch('observation', {}).empty?
+          Array(payload['nodes']).empty? && Array(payload['node_references']).empty? &&
+            payload.fetch('observation', {}).empty?
         end
 
         def merge_payload(left, right)
           observation = merge_observation(left.fetch('observation', {}), right.fetch('observation', {}))
           {
+            'schema_version' => [left['schema_version'], right['schema_version'], 2].compact.max,
+            'collector' => right['collector'] || left['collector'],
+            'scope' => right['scope'] || left['scope'],
+            'quality' => right['quality'] || left['quality'],
             'nodes' => (Array(left['nodes']) + Array(right['nodes'])).uniq.sort,
+            'node_references' => merge_references(left['node_references'], right['node_references']),
             'observation' => observation
           }
         end
@@ -128,11 +162,12 @@ module Necropsy
         def merge_observation(left, right)
           started_at = [left['started_at'], right['started_at']].compact.min
           finished_at = [left['finished_at'], right['finished_at']].compact.max
-          observation = left.merge(right)
+          observation = ObservationPolicy.compatible_merge(left, right)
           observation['started_at'] = started_at if started_at
           observation['finished_at'] = finished_at if finished_at
           observation['days'] = merged_days(left, right, started_at, finished_at)
           observation['collector'] = 'coverage'
+          observation['collector_overhead'] = merge_overhead(left, right)
           observation['processes'] = process_count(left) + process_count(right)
           observation
         end
@@ -157,16 +192,25 @@ module Necropsy
           1
         end
 
-        def executed_nodes(result)
+        def merge_overhead(left, right)
+          left_overhead = left.fetch('collector_overhead', {})
+          right_overhead = right.fetch('collector_overhead', {})
+          {
+            'wall_time_seconds' => left_overhead['wall_time_seconds'].to_f + right_overhead['wall_time_seconds'].to_f,
+            'observed_nodes' => left_overhead['observed_nodes'].to_i + right_overhead['observed_nodes'].to_i
+          }
+        end
+
+        def executed_references(result)
           result.flat_map do |path, coverage|
             next [] unless project_path?(path)
 
             coverage.fetch(:methods, {}).filter_map do |method_key, count|
               next unless count.to_i.positive?
 
-              node_id_for(method_key)
+              node_reference_for(path, method_key)
             end
-          end
+          end.uniq { |reference| RuntimeReference.key(reference) }.sort_by { |reference| RuntimeReference.sort_key(reference) }
         end
 
         def project_path?(path)
@@ -182,6 +226,24 @@ module Necropsy
           "#{owner_name}#{separator}#{method_name}"
         end
 
+        def node_reference_for(path, method_key)
+          symbol_id = node_id_for(method_key)
+          return unless symbol_id
+
+          RuntimeReference.build(
+            symbol_id: symbol_id,
+            file: RuntimeReference.relative_file(root, path),
+            line: method_key[2]
+          )
+        end
+
+        def merge_references(left, right)
+          (Array(left) + Array(right))
+            .filter_map { |reference| RuntimeReference.normalize(reference) }
+            .uniq { |reference| RuntimeReference.key(reference) }
+            .sort_by { |reference| RuntimeReference.sort_key(reference) }
+        end
+
         def owner_and_separator(owner)
           return [owner.name, '#'] if owner.respond_to?(:name) && owner.name