audition 0.4.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 52c12c94d193fa6946553c81ab22b8c21272f269ad4f6f758e99fa4a1c05217e
-  data.tar.gz: 858d3ef4e5742ca430337656b9ecea865a89c3f3525ef990073499c8eb438bca
+  metadata.gz: c47dfccb03a9d164583e3ae0a5530c74a41cc176f62865807fa4a4d1ed11d7f4
+  data.tar.gz: 75697dacb090c469d8f2966d5174fb08186e1808b85737872b0db5f6034a688d
 SHA512:
-  metadata.gz: 60d78e8622cfe35764e7dd30be43ba7e1ceda4f0faf10796230dd6ddb85d952736b345bd380b58823cf97c6687209f6c0fe709193f58326464b61950af1c8a70
-  data.tar.gz: 8edc6f7ba79f531317e26f5f1866540ec784d3791db0b6a43ae56d990b22bd173b5234a05618a8b75ba4c5bf1d95bb06a9898c726b493456c36604cdb7c90b66
+  metadata.gz: a631d1f7cd378ff04c47dcffb9608d332bfb048fc37dbda66179b085c487e936206359b3ea61a1e8fb5c70d121b6b7d94275b2e6d436f32772e1ea68a540fbd4
+  data.tar.gz: 974bbcd0a531588edf1d2474e9c8afb7bd5d3869a985b483ebb2c628007837ed20e50d10a580f6f0204db95e42128bb3de061237bbdaeab28f3d7008d7838a37

data/README.md

@@ -28,12 +28,13 @@ core itself is being ractorized; see the
   (which rule of the Ractor model it violates) and a `fix`
   (what to write instead).
 - **`--fix` like RuboCop, in two tiers.** Safe corrections:
-  `.freeze` on string constants, sentinels, and containers whose
-  elements are all shareable, `Ractor.make_shareable(...)` for
-  the remaining mutable and shallow-frozen containers and for
-  Proc constants, and
-  boot-time hoisting of method-body requires. `--fix-unsafe` adds
-  semantics-affecting rewrites: magic-comment insertion,
+  `.freeze` on string constants (literals as well as the fresh
+  strings core methods return), sentinels, and containers whose
+  elements are all shareable, and boot-time hoisting of
+  method-body requires. `--fix-unsafe` adds semantics-affecting
+  rewrites: `Ractor.make_shareable(...)` for the remaining mutable
+  and shallow-frozen containers, for containers a core method
+  allocates, and for Proc constants, magic-comment insertion,
   freeze-on-memoize for class-level memoization (both `@x ||=`
   and `return @x if defined?(@x)` idioms keep their caching, the
   memoized value becomes shareable, Rails-core style;
@@ -323,14 +324,29 @@ Static, with file:line precision:
 - **Class variables**, resolved on the rubydex graph.
 - **Class-level instance variables**, unified across the class
   body, `def self.`, and `class << self`, across files; the classic
-  `@cache ||= {}` and `return @x if defined?(@x)` memoizations.
+  `@cache ||= {}` and `return @x if defined?(@x)` memoizations. A
+  memo whose write is proxied to the main Ractor
+  (`@x || on_main(self) { @x ||= ... }`) is reported as that
+  escape hatch, not as a raw memo.
 - **Constants that are not deeply shareable**: bare mutable
   literals, interpolated strings, the subtle shallow freeze
   (`[[1], [2]].freeze` still raises; Audition explains why), and
-  call results the magic comment never covers (`X.tr(":", "")`,
-  `Regexp.new`, `Regexp.union`, `format`), the shapes Rails fixed
-  last in its own ractorization. Honors `# frozen_string_literal:`
-  and `# shareable_constant_value:` magic comments.
+  call results the magic comment never covers, from the
+  return-type contracts of core methods: fresh strings
+  (`X.tr(":", "")`, `[8, 2, 0].join(".")`, `Regexp.new`,
+  `format`), fresh containers (`TYPES.keys`, `LIST.map { }`,
+  `BASE + [:x]`, `DEFAULTS.merge(...)`, `.dup`), string splitters
+  under a shallow freeze (`".*".chars.freeze`), and Method
+  objects, which no freeze makes shareable. A spec executes the
+  tables against the running Ruby. Integer arithmetic,
+  comparisons, and negation are recognized as shareable. Honors
+  `# frozen_string_literal:` and `# shareable_constant_value:`
+  magic comments.
+- **Instance memoization on classes that get frozen**: a lazy
+  `@x ||=` on a class whose initialize freezes self raises
+  FrozenError on first use; on a class with a `freeze` override
+  the memo must be warmed inside the override before `super`
+  (compute on freeze), and one left cold is reported.
 - **Sync primitives and Procs in constants** (Mutex, Queue,
   lambdas), including `Hash.new { }` default procs, which stay
   unshareable even after `.freeze`.
@@ -341,7 +357,12 @@ Static, with file:line precision:
 - **Runtime require and autoload** (serializes all Ractors through
   the main-Ractor proxy).
 - **`Ractor.new` blocks capturing outer locals** (the ArgumentError
-  at creation time), resolved through Prism's exact scope depths.
+  at creation time), resolved through Prism's exact scope depths;
+  and **blocks that `Ractor.shareable_proc` would refuse**, handed
+  to it directly or to a Rails callback macro (`before_create`,
+  `validate`, `on_load`, ...), because a captured local holds a
+  provably unshareable value (`prefix = +"Draft: "`) or is
+  assigned more than once.
 - **Hostile or removed APIs**: `Ractor.yield`/`take` (gone in 4.0),
   ActiveSupport `cattr_*`/`mattr_*` class variables (with the
   `class_attribute` migration Rails itself made) and
@@ -371,10 +392,15 @@ Dynamic, on the live object graph:
 
 - Runs scripts inside a real Ractor (via `load`, which is not
   proxied) and reports the actual exception.
-- Requires a library, then sweeps every constant it introduced with
+- Requires a library, then sweeps every constant it introduced,
+  under new namespaces and pre-existing ones alike, with
   `Ractor.shareable?`, and inspects every class and module for
   class-level ivars and class variables, with
-  `const_source_location` attribution.
+  `const_source_location` attribution. What the sweep proves
+  shareable retires the static guesses about the same objects:
+  an unproven constant warning disappears, and class-level state
+  that held only shareable values after boot keeps an info note
+  instead of an error.
 - Records every compiled extension the load pulled in,
   dependencies included, and byte-scans each for the
   `rb_ext_ractor_safe` import; silent ones are reported against
@@ -384,8 +410,14 @@ Dynamic, on the live object graph:
   the per-worker model of Ractor web servers; then hammers it from
   4 Ractors x 25 requests to surface failures that only appear
   under concurrency.
-- Boots Rails (`config/environment.rb`), eager-loads, and sweeps
-  the application's namespaces.
+- Boots Rails (`config/environment.rb`) with
+  `unshareable_proc_action` armed, so every callback block Rails
+  cannot make shareable is reported at the Proc's definition site;
+  eager-loads; on Rails 8.2 calls `ractorize!` to freeze the
+  application graph, then serves one GET / on the main Ractor
+  (where a lazy memo on a now-frozen object raises FrozenError)
+  and one inside a Ractor; and sweeps the application's
+  namespaces. Without `ractorize!` an info note says so.
 
 ## Agent skill
 

data/lib/audition/bundle_sweep.rb

@@ -80,6 +80,7 @@ module Audition
       findings = filter(static_findings(target, config),
         directives, config)
       results = dynamic_results(target)
+      findings = Reconciliation.apply(findings, results)
       findings += filter(results.flat_map(&:findings),
         directives, config)
       report = Report.new(

data/lib/audition/cli.rb

@@ -189,6 +189,7 @@ module Audition
         )
       end
 
+      findings = Reconciliation.apply(findings, results)
       all = findings +
         filter(results.flat_map(&:findings), directives, config)
 

data/lib/audition/dynamic/harness.rb

@@ -5,13 +5,15 @@
 #
 #   ruby harness.rb MODE < payload.json
 #
-# Prints exactly one JSON document on stdout and never raises.
-# Stdlib only; must stay runnable on a bare Ruby 4.0.
+# Prints exactly one Marshal document on stdout and never raises.
+# Stdlib only, and nothing beyond rbconfig loaded before the
+# target: a library the harness required first (json, once)
+# would be swept as pre-existing and its own state never seen.
+# Must stay runnable on a bare Ruby 4.0.
 
 Warning[:experimental] = false
 Thread.report_on_exception = false
 
-require "json"
 require "rbconfig"
 
 module AuditionHarness
@@ -70,19 +72,27 @@ module AuditionHarness
       else {"error" => {"class" => "ArgumentError",
                         "message" => "unknown mode #{mode}"}}
       end
-    out.puts(JSON.generate(result))
+    emit(out, result)
   rescue Exception => e
     begin
-      out.puts(JSON.generate("error" => describe_error(e)))
+      emit(out, "error" => describe_error(e))
     rescue Exception
-      out.puts('{"error":{"class":"HarnessFailure",' \
-               '"message":"unreportable error"}}')
+      emit(out, "error" => {"class" => "HarnessFailure",
+                            "message" => "unreportable error"})
     end
   end
 
+  # One Marshal document: the prober loads it back into the same
+  # strings, arrays, and hashes, binary bytes included.
+  def emit(out, result)
+    out.binmode
+    out.write(Marshal.dump(result))
+    out.flush
+  end
+
   # Exception messages can carry arbitrary bytes (C extensions,
-  # binary filenames); unscrubbed they blow up JSON.generate
-  # inside the rescue and the harness dies without output.
+  # binary filenames); scrubbed to UTF-8 so the report can print
+  # them.
   def describe_error(error)
     root = unwrap(error)
     {"class" => scrub(root.class.name.to_s),
@@ -116,12 +126,13 @@ module AuditionHarness
 
   def in_ractor(*args, &block)
     ractor = Ractor.new(*args, &block)
-    {"ok" => true, "value" => jsonable(ractor.value)}
+    {"ok" => true, "value" => plain_value(ractor.value)}
   rescue Exception => e
     {"ok" => false, "error" => describe_error(e)}
   end
 
-  def jsonable(value)
+  # Only primitives cross the pipe; anything else is described.
+  def plain_value(value)
     case value
     when Numeric, String, Symbol, true, false, nil then value
     else value.inspect[0, 200]
@@ -152,21 +163,35 @@ module AuditionHarness
     Array(payload["load_paths"]).each do |lp|
       $LOAD_PATH.unshift(lp) # audition:disable global-variables
     end
+    root = payload["root"]
+    limit = payload["max_constants"]
+    known = known_paths(payload["known_compiled"])
     baseline = module_state_snapshot
     before = Object.constants
     features = $LOADED_FEATURES.dup # audition:disable global-variables
     require_target(payload.fetch("feature"),
       Array(payload["load_paths"]))
     merge_reopened(
-      scan(Object.constants - before, root: payload["root"],
-        limit: payload["max_constants"]), baseline
+      scan(top_level(before), root: root, limit: limit, known: known),
+      baseline, root: root, limit: limit, known: known
     ).merge(
       "native_extensions" => native_extensions(
-        features, payload["root"], payload["known_compiled"]
+        features, root, payload["known_compiled"]
       )
     )
   end
 
+  # The top-level constants a load introduced, as scan roots.
+  def top_level(before)
+    (Object.constants - before).map { |name| [Object, name.to_s] }
+  end
+
+  # The target's own compiled files, realpathed like everything
+  # the sweep compares against them.
+  def known_paths(paths)
+    Array(paths).map { |path| realpath(path) }
+  end
+
   # Gem names and entry files diverge in two conventional ways:
   # dashed names ship slashed files (rspec-mocks provides
   # rspec/mocks), and squashed names ship snake_case files
@@ -206,19 +231,20 @@ module AuditionHarness
   # are inspected for class-level ivars and class variables, then
   # descended into. const_get can raise (autoload failures) and
   # anything can lie; every step is rescued and counted.
-  def scan(root_names, root: nil, limit: nil)
+  def scan(roots, root: nil, limit: nil, known: [])
     # Loaded features are realpathed by require; the target root
     # must be too, or symlinked paths (macOS /var vs /private/var)
     # break the own-vs-dependency comparison.
     root = realpath(root)
     limit = (limit || MAX_CONSTS).to_i
     unshareable = []
+    proven = []
     class_state = []
     class_vars = []
     errors = 0
     truncated = false
     seen = {}
-    queue = root_names.map { |name| [Object, name.to_s] }
+    queue = roots.map { |owner, name| [owner, name.to_s] }
     visited = 0
 
     until queue.empty?
@@ -236,7 +262,7 @@ module AuditionHarness
         next
       end
       full = owner.equal?(Object) ? name : "#{owner}::#{name}"
-      origin = origin_for(owner, name, root)
+      origin = origin_for(owner, name, root, known)
 
       if value.is_a?(Module)
         next if seen[value.object_id]
@@ -249,7 +275,10 @@ module AuditionHarness
         end
       else
         begin
-          unless Ractor.shareable?(value)
+          if Ractor.shareable?(value)
+            # Proof the static pass can retire its guesses with.
+            proven << [origin["path"], origin["line"]] if origin["path"]
+          else
             blocker, blocker_depth = blocker_for(value)
             unshareable << origin.merge(
               "const" => full, "class" => value.class.name,
@@ -264,6 +293,7 @@ module AuditionHarness
     end
 
     {"unshareable_constants" => unshareable,
+     "proven_constants" => proven,
      "class_state" => class_state,
      "class_variables" => class_vars,
      "scanned" => visited,
@@ -274,9 +304,10 @@ module AuditionHarness
 
   # Where was this constant defined, and does that location belong
   # to the audited target (as opposed to a dependency it loaded)?
-  # Unknown locations (C extensions, core) count as own so nothing
-  # gets silently downgraded.
-  def origin_for(owner, name, root)
+  # Unknown locations (core) count as own so nothing gets silently
+  # downgraded, and so does the target's own compiled extension,
+  # which RubyGems installs outside the gem's root.
+  def origin_for(owner, name, root, known = [])
     path, line = begin
       owner.const_source_location(name)
     rescue Exception
@@ -284,11 +315,23 @@ module AuditionHarness
     end
     # The separator matters: /x/app must not claim /x/app-helpers.
     own = root.nil? || path.nil? || path == root ||
+      own_compiled?(path, root, known) ||
       (path.start_with?(root + File::SEPARATOR) &&
         !excluded?(path, root))
     {"path" => path, "line" => line, "own" => own}
   end
 
+  # A compiled file the target listed, or the copy of it RubyGems
+  # built into its extensions directory, which sits outside the
+  # gem's root under a directory named after the gem
+  # (extensions/<platform>/<abi>/stringio-3.2.0/stringio.bundle).
+  def own_compiled?(path, root, known)
+    return true if known.include?(path) || known.include?(realpath(path))
+    return false unless path.match?(NATIVE) && root
+
+    path.include?(File::SEPARATOR + File.basename(root) + File::SEPARATOR)
+  end
+
   # Matches the static scanner's exclusion rule: any excluded or
   # dot-prefixed component in the root-relative path means the
   # file is not the target's own code.
@@ -397,7 +440,7 @@ module AuditionHarness
       seen[value.object_id] = true
       begin
         snap[value] = [value.instance_variables,
-          value.class_variables(false)]
+          value.class_variables(false), value.constants(false)]
       rescue Exception
         next
       end
@@ -413,17 +456,31 @@ module AuditionHarness
   MACHINERY = /\A(?:Gem|Bundler|RubyVM)(?:::|\z)/
 
   # State the boot added to pre-existing modules, reported through
-  # the same channels as freshly defined class state. No source
-  # location exists for a reopen, and unknown origins count as own.
-  def merge_reopened(result, snapshot)
+  # the same channels as freshly defined class state, and the
+  # constants it added under them, swept like top-level ones
+  # (Ractor::Dispatch lives under the core Ractor class, and the
+  # new-constants sweep never looks there). No source location
+  # exists for a reopen, and unknown origins count as own.
+  def merge_reopened(result, snapshot, root: nil, limit: nil,
+    known: [])
     origin = {"path" => nil, "line" => nil, "own" => true}
-    snapshot.each do |mod, (ivars, cvars)|
+    added = []
+    snapshot.each do |mod, (ivars, cvars, consts)|
+      full = mod.name || mod.inspect
+      next if full.match?(MACHINERY)
+
+      unless mod.equal?(Object)
+        new_consts = begin
+          mod.constants(false) - consts
+        rescue Exception
+          []
+        end
+        new_consts.each { |name| added << [mod, name.to_s] }
+      end
       new_ivars = mod.instance_variables - ivars
       new_cvars = mod.class_variables(false) - cvars
       next if new_ivars.empty? && new_cvars.empty?
 
-      full = mod.name || mod.inspect
-      next if full.match?(MACHINERY)
       if new_ivars.any?
         unshareable = new_ivars.reject do |ivar|
           safe_shareable?(mod.instance_variable_get(ivar))
@@ -439,9 +496,17 @@ module AuditionHarness
           "const" => full, "cvars" => new_cvars.map(&:to_s)
         )
       end
-    rescue Exception
-      next
     end
+    return result if added.empty?
+
+    extra = scan(added, root: root, limit: limit, known: known)
+    %w[unshareable_constants proven_constants class_state
+      class_variables].each do |key|
+      result[key] = Array(result[key]) + extra[key]
+    end
+    result["scanned"] = result["scanned"].to_i + extra["scanned"]
+    result["errors"] = result["errors"].to_i + extra["errors"]
+    result["truncated"] ||= extra["truncated"]
     result
   end
 
@@ -523,9 +588,11 @@ module AuditionHarness
     # The main-process boot defines the app's constant graph;
     # sweeping it gives rack targets the same backstop as require
     # and rails targets. A failed boot still sweeps what loaded.
+    limit = payload["max_constants"]
+    known = known_paths(payload["known_compiled"])
     out.merge!(merge_reopened(
-      scan(Object.constants - before, root: root,
-        limit: payload["max_constants"]), baseline
+      scan(top_level(before), root: root, limit: limit, known: known),
+      baseline, root: root, limit: limit, known: known
     ))
     out["native_extensions"] = native_extensions(
       features, root, payload["known_compiled"]
@@ -629,17 +696,26 @@ module AuditionHarness
   # it could reach alongside the boot error.
   def rails(payload)
     environment = payload.fetch("environment")
+    root = payload["root"]
     bundler_setup
+    # The post-freeze requests need it; loaded before the snapshot
+    # so the sweep never attributes it to the target.
+    require "stringio" # audition:disable runtime-require
     baseline = module_state_snapshot
     before = Object.constants
     features = $LOADED_FEATURES.dup # audition:disable global-variables
     started = Time.now
     boot_error = nil
+    proc_warnings = []
+    armed = arm_proc_gate(proc_warnings, preload: true)
     begin
       # Absolute requires keep the path as given; realpathing it
       # keeps own-vs-dependency attribution honest under symlinked
       # roots (macOS /var).
       require realpath(environment) # audition:disable runtime-require
+      # Boot may have replaced the deprecation behavior, and an
+      # app that only defines the shim during boot is armed here.
+      arm_proc_gate(proc_warnings, preload: false) || armed
       begin
         Rails.application.eager_load!
       rescue Exception
@@ -654,19 +730,118 @@ module AuditionHarness
       else
         {"ok" => true, "seconds" => (Time.now - started).round(1)}
       end
+    # Freezing first lets the sweep see warmed state as shareable.
+    ractorize = boot_error ? nil : ractorize_application
+    limit = payload["max_constants"]
+    known = known_paths(payload["known_compiled"])
     merge_reopened(
-      scan(Object.constants - before, root: payload["root"],
-        limit: payload["max_constants"]), baseline
+      scan(top_level(before), root: root, limit: limit, known: known),
+      baseline, root: root, limit: limit, known: known
     ).merge(
       "boot" => boot,
+      "ractorize" => ractorize,
+      "unshareable_procs" => unshareable_procs(proc_warnings, root),
       "native_extensions" => native_extensions(
-        features, payload["root"], payload["known_compiled"]
+        features, root, payload["known_compiled"]
       )
     )
   rescue Exception => e
     {"boot" => {"ok" => false, "error" => describe_error(e)}}
   end
 
+  # Rails 8.2 tries to make every callback block shareable once
+  # unshareable_proc_action is set, and reports each one it
+  # cannot as a deprecation naming the Proc. The probe arms :warn
+  # and collects those messages. Arming before boot covers apps
+  # that eager load while booting; the shim file only exists on
+  # 8.2, so older targets load nothing extra. Returns whether the
+  # gate is armed.
+  def arm_proc_gate(warnings, preload:)
+    if preload
+      begin
+        require "active_support/ractors" # audition:disable runtime-require
+        require "active_support" # audition:disable runtime-require
+      rescue LoadError
+        return false
+      end
+    end
+    return false unless defined?(ActiveSupport::Ractors) &&
+      ActiveSupport::Ractors.respond_to?(:unshareable_proc_action=)
+
+    ActiveSupport::Ractors.unshareable_proc_action = :warn
+    return true unless ActiveSupport.respond_to?(:deprecator)
+
+    deprecator = ActiveSupport.deprecator
+    collector = lambda do |message, _callstack|
+      warnings << message.to_s
+    end
+    deprecator.behavior = [collector]
+    deprecator.silenced = false if deprecator.respond_to?(:silenced=)
+    if deprecator.respond_to?(:disallowed_warnings=)
+      deprecator.disallowed_warnings = []
+    end
+    true
+  rescue Exception
+    false
+  end
+
+  # The deprecation names the Proc; its inspect carries the
+  # definition site, which is where the fix goes.
+  PROC_SITE = /#<Proc:0x\h+(?: \(lambda\))? (.+?):(\d+)>/
+
+  def unshareable_procs(warnings, root)
+    warnings.filter_map do |message|
+      next unless message.include?("Ractor shareable")
+
+      match = message.match(PROC_SITE)
+      path = match && match[1]
+      path = File.expand_path(path, root) if path && root &&
+        !path.start_with?("/")
+      shown = match ? match[0] : message.lines.last.to_s.strip
+      shown = shown.sub("#{root}/", "") if root
+      {"proc" => shown,
+       "path" => path,
+       "line" => match && Integer(match[2], 10),
+       "own" => path.nil? || own_path?(path, root)}
+    end
+  end
+
+  # Rails 8.2 adds Application#ractorize!, which deep-freezes the
+  # application graph. One GET / on the main Ractor afterwards
+  # surfaces lazy memoization on now-frozen objects (FrozenError),
+  # and one inside a Ractor surfaces state a worker cannot reach.
+  def ractorize_application
+    app = Rails.application
+    version = Rails.respond_to?(:version) ? Rails.version.to_s : nil
+    result = {"rails" => version}
+    return result.merge("available" => false) unless
+      app.respond_to?(:ractorize!)
+
+    result["available"] = true
+    begin
+      app.ractorize!
+    rescue Exception => e
+      return result.merge("ok" => false, "error" => describe_error(e))
+    end
+    result["ok"] = true
+    result["main_request"] = main_request(app)
+    if result["main_request"]["ok"]
+      result["ractor_request"] = in_ractor do
+        require "stringio" # audition:disable runtime-require
+        Rails.application.call(AuditionHarness.base_env).first
+      end
+    end
+    result
+  rescue Exception => e
+    {"available" => true, "ok" => false, "error" => describe_error(e)}
+  end
+
+  def main_request(app)
+    {"ok" => true, "status" => plain_value(app.call(base_env).first)}
+  rescue Exception => e
+    {"ok" => false, "error" => describe_error(e)}
+  end
+
   # -- capabilities ------------------------------------------------
 
   def capabilities
@@ -735,7 +910,7 @@ if $PROGRAM_NAME == __FILE__ # audition:disable global-variables
   real_stdout = $stdout.dup
   $stdout.reopen($stderr)
   mode = ARGV.fetch(0, "capabilities")
-  raw = $stdin.tty? ? "" : $stdin.read
-  payload = raw.empty? ? {} : JSON.parse(raw)
+  raw = $stdin.tty? ? "" : $stdin.binmode.read
+  payload = raw.empty? ? {} : Marshal.load(raw)
   AuditionHarness.main(mode, payload, out: real_stdout)
 end