benchtrack 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8e2ea31f63cdc62a5b3863f56d1a7f081ae877cda4645396b2d0b2ad67c910f1
+  data.tar.gz: 06f9f09c0a6d518bb7aa7aa036f25db9d60d4a85b461e877638272dbd45012a1
+SHA512:
+  metadata.gz: d93c457d1873ff66b149fa88ab072683ea6f76985b162f4a89fb5598227d3c52a9d045bf6049831bf0f055bd01ccf395f61866694f294e17a6c98e0ab94467c7
+  data.tar.gz: 9d395f2019f6fc4b0f52247bf568aa98128dafaec3d37ae914511ad9788750de220fe50c55fb9833315ee9c39a8aba32b9243063d2df29248f7a1076da389901

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Ojus Chugh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,63 @@
+# BenchTrack
+
+[![CI](https://github.com/ojuschugh1/benchtrack/actions/workflows/ci.yml/badge.svg)](https://github.com/ojuschugh1/benchtrack/actions/workflows/ci.yml)
+
+BenchTrack is pull-request-time performance regression testing for Ruby libraries. It takes an existing benchmark-ips suite, unmodified, and turns it into a paired base-versus-head comparison: both git refs are checked out into worktrees, measured in interleaved randomized blocks under one pinned Ruby, and the check fails only when a slowdown is both larger than a practical threshold and statistically supported. BenchTrack is Ruby-native and service-free: no hosted backend, no runtime gem dependencies, everything runs on the machine you invoke it on.
+
+## Install
+
+```
+gem install benchtrack
+```
+
+Requires Ruby 3.1 or newer, Linux or macOS, and git. The measured project needs benchmark-ips in its bundle; BenchTrack itself declares zero runtime dependencies.
+
+You can also run it straight from a checkout without installing anything: clone the repository and invoke `ruby -I/path/to/benchtrack/lib /path/to/benchtrack/exe/benchtrack ...`. There is nothing to build and no dependency beyond a Ruby that can load benchmark-ips for the project being measured.
+
+## Quickstart
+
+Run your suite once to check that BenchTrack can execute and parse it:
+
+```
+benchtrack run bench/ips.rb
+```
+
+Then compare two refs:
+
+```
+benchtrack compare main my-branch --suite bench/ips.rb
+```
+
+`--suite` is required and is a path relative to the repository root. Optional flags: `--blocks N` (measurement blocks, default 10), `--threshold PCT` (practical slowdown threshold in percent, default 5.0), `--seed N` (integer seed for reproducible runs, generated and recorded when absent), `--json PATH` (report path, default `benchtrack-report.json`), and `--prepare CMD` (shell command run in each worktree after bundle install).
+
+Gems with C extensions need a build step in each worktree before the suite can load them, for example `--prepare 'bundle exec rake compile'`. The command runs with the worktree as its working directory and with the Ruby being measured first on `PATH`, and a nonzero exit aborts the comparison.
+
+A suite file may call `Benchmark.ips` more than once. A report name that repeats across calls gets the call number appended, so the `json` report of the third call is labelled `json [3]`, while names that appear only once in the file are left alone.
+
+The exit status is 0 when every benchmark passes, 1 when at least one supported regression is found, and 2 on operational errors such as unresolvable refs, failed dependency installs, or unparseable suite output. Every completed comparison also writes a JSON report with per-entry statistics and an environment fingerprint, so CI can archive it and scripts can parse it.
+
+## How it measures
+
+Measurements are collected in paired randomized blocks: each block runs one base invocation and one head invocation in random order, so time-varying system noise hits both sides roughly equally instead of biasing one. Every invocation is a fresh Ruby subprocess running your suite with the benchmark-ips warmup phase, so JIT, GC, and heap state never carry over between measurements. For each benchmark entry, BenchTrack computes the per-block log ratio of head to base, then applies an exact paired sign-flip permutation test with Holm adjustment across entries. An entry fails only when the estimated slowdown exceeds the practical threshold and the adjusted p-value is below 0.05, so neither noise alone nor a trivial difference alone can fail your build.
+
+The full rationale is in [methodology.md](methodology.md). Reproducible pass and fail demonstrations, including a repeated no-change control, are in [validation/README.md](validation/README.md).
+
+## Statistical floor
+
+With n blocks, the minimum achievable two-sided permutation p-value is 2/2^n. Block counts below 6 therefore cannot produce a fail verdict: at 5 blocks the floor is 0.0625, which is above the 0.05 significance level, so the gate mathematically cannot fire. The default of 10 blocks gives a floor of about 0.002.
+
+## Roadmap
+
+None of the following exists yet; it is planned next work:
+
+- benchmark-driver adapter
+- packaged GitHub Action
+- cross-runner calibration study
+- real-gem case studies
+- historical tracking and dashboards
+- allocation tracking
+- JIT-mode matrices (YJIT/ZJIT/no-JIT)
+
+## License
+
+MIT. See [LICENSE.txt](LICENSE.txt).

data/exe/benchtrack

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "benchtrack"
+
+exit(BenchTrack::CLI.start(ARGV))

data/lib/benchtrack/cli.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+require "optparse"
+
+module BenchTrack
+  class CLI
+    EXIT_OK         = 0
+    EXIT_REGRESSION = 1
+    EXIT_ERROR      = 2
+
+    USAGE = <<~TEXT
+      usage: benchtrack <command> [options]
+
+      commands:
+        run SUITE_FILE
+            run a benchmark-ips suite once and print each entry
+
+        compare BASE HEAD --suite PATH [options]
+            measure BASE vs HEAD and fail on supported regressions
+              --suite PATH      benchmark-ips suite, relative to the repository root
+              --blocks N        measurement blocks (default 10)
+              --threshold PCT   practical slowdown threshold in percent (default 5.0)
+              --seed N          integer seed for reproducible runs (default: generated)
+              --json PATH       JSON report path (default benchtrack-report.json)
+              --prepare CMD     shell command run in each worktree after bundle install (e.g. 'bundle exec rake compile')
+    TEXT
+
+    def self.start(argv)
+      case argv.first
+      when "run"     then run(argv.drop(1))
+      when "compare" then compare(argv.drop(1))
+      else
+        warn USAGE
+        EXIT_ERROR
+      end
+    rescue OptionParser::ParseError => e
+      warn "benchtrack: #{e.message}"
+      EXIT_ERROR
+    rescue Error => e
+      warn "benchtrack: #{e.message}"
+      EXIT_ERROR
+    end
+
+    def self.run(argv)
+      suite = OptionParser.new.parse(argv).first
+      raise ConfigError, "missing required argument: SUITE_FILE" unless suite
+
+      entries = SuiteRunner.run_suite(suite, chdir: Dir.pwd, env: SuiteRunner.bundler_env(Dir.pwd))
+      print Report.run_text(entries)
+      EXIT_OK
+    end
+
+    def self.compare(argv)
+      options = { threshold_pct: 5.0, blocks: 10, json_path: "benchtrack-report.json" }
+      base, head = compare_parser(options).parse(argv)
+      raise ConfigError, "missing required argument: BASE" unless base
+      raise ConfigError, "missing required argument: HEAD" unless head
+      raise ConfigError, "missing required option: --suite" unless options[:suite]
+
+      config = Config.new(command: "compare", suite: options[:suite], base: base, head: head,
+                          threshold_pct: options[:threshold_pct], blocks: options[:blocks],
+                          seed: options[:seed] || Random.new_seed % (2**32),
+                          json_path: options[:json_path], prepare: options[:prepare])
+
+      comparison = Orchestrator.compare(config)
+      print Report.terminal(comparison)
+      write_report(Report.json(comparison), config.json_path)
+      comparison.overall == "fail" ? EXIT_REGRESSION : EXIT_OK
+    end
+
+    def self.compare_parser(options)
+      OptionParser.new do |o|
+        o.on("--suite PATH")    { |v| options[:suite] = v }
+        o.on("--blocks N")      { |v| options[:blocks] = parse_blocks(v) }
+        o.on("--threshold PCT") { |v| options[:threshold_pct] = parse_threshold(v) }
+        o.on("--seed N")        { |v| options[:seed] = parse_seed(v) }
+        o.on("--json PATH")     { |v| options[:json_path] = v }
+        o.on("--prepare CMD")   { |v| options[:prepare] = v }
+      end
+    end
+
+    def self.parse_blocks(value)
+      blocks = begin
+        Integer(value)
+      rescue ArgumentError, TypeError
+        nil
+      end
+      raise ConfigError, "invalid blocks: #{value} (must be an integer >= 1)" if blocks.nil? || blocks < 1
+
+      blocks
+    end
+
+    def self.parse_threshold(value)
+      threshold = begin
+        Float(value)
+      rescue ArgumentError, TypeError
+        nil
+      end
+      if threshold.nil? || !threshold.finite? || threshold <= 0
+        raise ConfigError, "invalid threshold: #{value} (must be a number greater than zero)"
+      end
+
+      threshold
+    end
+
+    def self.parse_seed(value)
+      Integer(value)
+    rescue ArgumentError, TypeError
+      raise ConfigError, "invalid seed: #{value} (must be an integer)"
+    end
+
+    def self.write_report(json, path)
+      File.write(path, json)
+    rescue SystemCallError, IOError => e
+      raise Error, "cannot write JSON report to #{path}: #{e.message}"
+    end
+  end
+end

data/lib/benchtrack/ips_capture.rb

@@ -0,0 +1,22 @@
+require "benchmark/ips"
+require "json"
+
+module BenchTrackIpsCapture
+  CALLS = []
+
+  def ips(*args, &block)
+    report = super
+    CALLS << report.entries.to_h { |e| [e.label.to_s, e.ips.to_f] }
+    counts = CALLS.flat_map(&:keys).tally
+    entries = CALLS.each_with_index.flat_map do |call, i|
+      call.map do |label, ips|
+        { "label" => counts[label] > 1 ? "#{label} [#{i + 1}]" : label, "ips" => ips }
+      end
+    end
+    File.write(ENV.fetch("BENCHTRACK_RESULT_PATH"),
+               JSON.generate({ "format" => 1, "entries" => entries }))
+    report
+  end
+end
+
+Benchmark.singleton_class.prepend(BenchTrackIpsCapture)

data/lib/benchtrack/measurement.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module BenchTrack
+  module Measurement
+    Pairing = Struct.new(:series, :base_only, :head_only, keyword_init: true)
+
+    module_function
+
+    def collect(blocks:, rng:)
+      runs = { base: [], head: [] }
+      blocks.times do
+        first = rng.rand(2).zero? ? :base : :head
+        second = first == :base ? :head : :base
+        runs[first] << yield(first)
+        runs[second] << yield(second)
+      end
+      runs
+    end
+
+    def pair(base_runs, head_runs)
+      base_labels = consistent_labels(:base, base_runs)
+      head_labels = consistent_labels(:head, head_runs)
+
+      base_ips = base_runs.map { |run| run.to_h { |e| [e.label, e.ips] } }
+      head_ips = head_runs.map { |run| run.to_h { |e| [e.label, e.ips] } }
+
+      series = (base_labels & head_labels).sort.map do |label|
+        PairedSeries.new(label,
+                         base_ips.map { |h| h.fetch(label) },
+                         head_ips.map { |h| h.fetch(label) })
+      end
+
+      Pairing.new(series: series,
+                  base_only: (base_labels - head_labels).sort,
+                  head_only: (head_labels - base_labels).sort)
+    end
+
+    def consistent_labels(side, runs)
+      labels = runs.first.map(&:label)
+      runs.each do |run|
+        got = run.map(&:label)
+        next if got.sort == labels.sort
+
+        diff = ((labels - got) | (got - labels)).sort
+        raise SuiteError,
+              "#{side} suite produced inconsistent labels across runs (differing: #{diff.join(", ")})"
+      end
+      labels
+    end
+  end
+end

data/lib/benchtrack/orchestrator.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require "open3"
+require "tmpdir"
+require "fileutils"
+require "digest"
+require "rbconfig"
+require "bundler"
+
+module BenchTrack
+  module Orchestrator
+    module_function
+
+    def compare(config)
+      repo = Dir.pwd
+      _, status = Open3.capture2e("git", "rev-parse", "--git-dir", chdir: repo)
+      raise GitError, "not inside a git repository: #{repo}" unless status.success?
+
+      shas = { base: resolve_ref!(config.base), head: resolve_ref!(config.head) }
+
+      Dir.mktmpdir("benchtrack") do |tmp|
+        created = []
+        begin
+          worktrees = shas.to_h do |side, sha|
+            dir = File.join(tmp, side.to_s)
+            git("worktree", "add", "--detach", dir, sha, chdir: repo)
+            created << dir
+            [side, dir]
+          end
+
+          envs = {}
+          lockfile_hashes = {}
+          worktrees.each do |side, dir|
+            envs[side] = install_bundle(side, dir, tmp)
+            lockfile_hashes[side.to_s] = lockfile_hash(dir)
+            prepare_worktree(side, dir, envs[side], config.prepare) if config.prepare
+          end
+
+          worktrees.each do |side, dir|
+            next if File.file?(File.expand_path(config.suite, dir))
+
+            raise SuiteError, "suite file not found in #{side} worktree: #{config.suite}"
+          end
+
+          invokers = worktrees.to_h do |side, dir|
+            [side, -> { SuiteRunner.run_suite(config.suite, chdir: dir, env: envs[side]) }]
+          end
+
+          runs = Measurement.collect(blocks: config.blocks, rng: Random.new(config.seed)) do |side|
+            invokers[side].call
+          end
+          pairing = Measurement.pair(runs[:base], runs[:head])
+          results = Stats.analyze(pairing.series, seed: config.seed,
+                                  threshold_pct: config.threshold_pct)
+
+          Comparison.new(
+            base_ref: config.base,
+            head_ref: config.head,
+            base_sha: shas[:base],
+            head_sha: shas[:head],
+            results: results,
+            base_only: pairing.base_only,
+            head_only: pairing.head_only,
+            overall: results.any? { |result| result.verdict == "fail" } ? "fail" : "pass",
+            fingerprint: Report.fingerprint(seed: config.seed, blocks: config.blocks,
+                                            lockfile_hashes: lockfile_hashes),
+            config: config
+          )
+        ensure
+          created.each { |dir| remove_worktree(repo, dir) }
+          prune_worktrees(repo)
+        end
+      end
+    end
+
+    def resolve_ref!(ref)
+      out, status = Open3.capture2e("git", "rev-parse", "--verify", "--quiet", "#{ref}^{commit}")
+      raise GitError, "cannot resolve git ref: #{ref}" unless status.success?
+
+      out.strip
+    end
+
+    def git(*args, chdir:)
+      out, status = Open3.capture2e("git", *args, chdir: chdir)
+      raise GitError, "git #{args.join(' ')} failed:\n#{out}" unless status.success?
+
+      out
+    end
+
+    def run_command(env, *argv, chdir:)
+      Bundler.with_unbundled_env { Open3.capture2e(env, *argv, chdir: chdir) }
+    end
+
+    def install_bundle(side, worktree, tmp)
+      gemfile = File.join(worktree, "Gemfile")
+      return {} unless File.file?(gemfile)
+
+      env = { "BUNDLE_GEMFILE" => gemfile, "BUNDLE_PATH" => File.join(tmp, "bundle-#{side}") }
+      out, status = run_command(env, RbConfig.ruby, Gem.bin_path("bundler", "bundle"), "install",
+                                chdir: worktree)
+      raise BundleError, "bundle install failed for #{side}:\n#{out}" unless status.success?
+
+      env
+    end
+
+    def prepare_worktree(side, worktree, env, command)
+      path = "#{File.dirname(RbConfig.ruby)}#{File::PATH_SEPARATOR}#{ENV['PATH']}"
+      out, status = run_command(env.merge("PATH" => path), "sh", "-c", command, chdir: worktree)
+      raise PrepareError, "prepare command failed for #{side}:\n#{out}" unless status.success?
+    end
+
+    def lockfile_hash(worktree)
+      lockfile = File.join(worktree, "Gemfile.lock")
+      return "unknown" unless File.file?(lockfile)
+
+      Digest::SHA256.hexdigest(File.read(lockfile))
+    end
+
+    def remove_worktree(repo, worktree)
+      git("worktree", "remove", "--force", worktree, chdir: repo)
+    rescue StandardError => e
+      warn "benchtrack: could not remove worktree #{worktree}: #{e.message}"
+      begin
+        FileUtils.rm_rf(worktree)
+      rescue StandardError => rm_error
+        warn "benchtrack: could not delete #{worktree}: #{rm_error.message}"
+      end
+    end
+
+    def prune_worktrees(repo)
+      git("worktree", "prune", chdir: repo)
+    rescue StandardError => e
+      warn "benchtrack: git worktree prune failed: #{e.message}"
+    end
+  end
+end

data/lib/benchtrack/report.rb

@@ -0,0 +1,155 @@
+# frozen_string_literal: true
+
+require "etc"
+require "json"
+require "time"
+require "rbconfig"
+
+module BenchTrack
+  module Report
+    ROW_FORMAT = "%-26s%-12s%-21s%-10s%s"
+
+    module_function
+
+    def fingerprint(seed:, blocks:, lockfile_hashes:)
+      {
+        "cpu_model" => probe { cpu_model },
+        "cpu_cores" => probe { Etc.nprocessors },
+        "os" => probe { os_description },
+        "kernel" => probe { "#{Etc.uname[:sysname]} #{Etc.uname[:release]}" },
+        "ruby" => probe { RUBY_DESCRIPTION },
+        "ruby_binary" => probe { RbConfig.ruby },
+        "jit" => probe { jit_state },
+        "lockfile_sha256" => lockfile_hashes,
+        "seed" => seed,
+        "blocks" => blocks,
+        "timestamp" => probe { Time.now.utc.iso8601 },
+        "benchtrack_version" => VERSION
+      }
+    end
+
+    def terminal(comparison)
+      config = comparison.config
+      lines = [
+        "benchtrack compare: #{comparison.base_ref} (#{comparison.base_sha.to_s[0, 7]}) " \
+        "vs #{comparison.head_ref} (#{comparison.head_sha.to_s[0, 7]})",
+        "blocks: #{config.blocks}   threshold: #{config.threshold_pct}%   " \
+        "alpha: #{Stats::ALPHA}   seed: #{config.seed}",
+        ""
+      ]
+
+      unless comparison.results.empty?
+        lines << format(ROW_FORMAT, "label", "change", "95% CI", "holm p", "verdict")
+        comparison.results.each { |result| lines << table_row(result) }
+        lines << ""
+      end
+
+      unless comparison.base_only.empty?
+        lines << "unmatched labels: base only: #{comparison.base_only.join(', ')}"
+      end
+      unless comparison.head_only.empty?
+        lines << "unmatched labels: head only: #{comparison.head_only.join(', ')}"
+      end
+      lines << "no benchmark labels were paired; nothing was compared" if comparison.results.empty?
+      lines << "overall: #{display_verdict(comparison.overall)}"
+      lines.join("\n") + "\n"
+    end
+
+    def json(comparison)
+      config = comparison.config
+      report = {
+        "benchtrack_version" => VERSION,
+        "overall_verdict" => comparison.overall,
+        "refs" => {
+          "base" => comparison.base_ref,
+          "head" => comparison.head_ref,
+          "base_sha" => comparison.base_sha,
+          "head_sha" => comparison.head_sha
+        },
+        "config" => { "threshold_pct" => config.threshold_pct, "alpha" => Stats::ALPHA,
+                      "prepare" => config.prepare },
+        "entries" => comparison.results.map { |result| entry_hash(result) },
+        "unmatched_labels" => {
+          "base_only" => comparison.base_only,
+          "head_only" => comparison.head_only
+        },
+        "environment" => comparison.fingerprint
+      }
+      JSON.generate(report)
+    end
+
+    def run_text(entries)
+      width = entries.map { |entry| entry.label.length }.max || 0
+      lines = entries.map { |entry| format("%-*s  %.1f i/s", width, entry.label, entry.ips) }
+      lines.join("\n") + "\n"
+    end
+
+    def entry_hash(result)
+      {
+        "label" => result.label,
+        "effect_log_ratio" => result.effect,
+        "percent_change" => result.percent_change,
+        "ci_log_ratio" => [result.ci_low, result.ci_high],
+        "p_value" => result.p_value,
+        "holm_p" => result.holm_p,
+        "verdict" => result.verdict
+      }
+    end
+
+    def table_row(result)
+      low_pct = (Math.exp(result.ci_low) - 1) * 100
+      high_pct = (Math.exp(result.ci_high) - 1) * 100
+      format(ROW_FORMAT,
+             result.label,
+             format("%+.1f%%", result.percent_change),
+             format("[%+.1f%%, %+.1f%%]", low_pct, high_pct),
+             format("%.3f", result.holm_p),
+             display_verdict(result.verdict))
+    end
+
+    def display_verdict(verdict)
+      verdict == "fail" ? "FAIL" : verdict
+    end
+
+    def probe
+      yield
+    rescue StandardError
+      "unknown"
+    end
+
+    def cpu_model
+      if darwin?
+        command_output("sysctl", "-n", "machdep.cpu.brand_string")
+      else
+        line = File.foreach("/proc/cpuinfo").find { |l| l.start_with?("model name") }
+        line.split(":", 2).last.strip
+      end
+    end
+
+    def os_description
+      if darwin?
+        "#{command_output('sw_vers', '-productName')} #{command_output('sw_vers', '-productVersion')}"
+      else
+        line = File.foreach("/etc/os-release").find { |l| l.start_with?("PRETTY_NAME=") }
+        line.split("=", 2).last.strip.delete_prefix('"').delete_suffix('"')
+      end
+    end
+
+    def jit_state
+      return "yjit" if defined?(RubyVM::YJIT) && RubyVM::YJIT.enabled?
+      return "rjit" if defined?(RubyVM::RJIT) && RubyVM::RJIT.enabled?
+      return "mjit" if defined?(RubyVM::MJIT) && RubyVM::MJIT.enabled?
+      "none"
+    end
+
+    def darwin?
+      RUBY_PLATFORM.include?("darwin")
+    end
+
+    def command_output(*argv)
+      output = IO.popen(argv, err: File::NULL, &:read).strip
+      raise Error, "no output from #{argv.first}" if !$?.success? || output.empty?
+      output
+    end
+  end
+end

data/lib/benchtrack/stats.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+module BenchTrack
+  module Stats
+    ALPHA       = 0.05
+    ENUM_LIMIT  = 20
+    MC_ROUNDS   = 10_000
+    BOOT_ROUNDS = 10_000
+
+    module_function
+
+    def log_ratios(series)
+      series.base.zip(series.head).map { |base, head| Math.log(head / base) }
+    end
+
+    def mean(xs)
+      xs.sum / xs.length
+    end
+
+    def permutation_pvalue(xs, rng)
+      n = xs.length
+      observed = 0.0
+      xs.each { |x| observed += x }
+      observed = observed.abs
+
+      if n <= ENUM_LIMIT
+        total = 1 << n
+        count = 0
+        total.times do |mask|
+          sum = 0.0
+          n.times { |i| sum += mask[i] == 1 ? -xs[i] : xs[i] }
+          count += 1 if sum.abs >= observed
+        end
+        count.fdiv(total)
+      else
+        hits = 0
+        MC_ROUNDS.times do
+          sum = 0.0
+          n.times { |i| sum += rng.rand(2).zero? ? xs[i] : -xs[i] }
+          hits += 1 if sum.abs >= observed
+        end
+        (hits + 1.0) / (MC_ROUNDS + 1)
+      end
+    end
+
+    def bootstrap_ci(xs, rng)
+      n = xs.length
+      means = Array.new(BOOT_ROUNDS) { mean(Array.new(n) { xs[rng.rand(n)] }) }
+      means.sort!
+      [means[((BOOT_ROUNDS - 1) * 0.025).round], means[((BOOT_ROUNDS - 1) * 0.975).round]]
+    end
+
+    def holm(pvalues)
+      return pvalues.dup if pvalues.length < 2
+
+      m = pvalues.length
+      order = pvalues.each_index.sort_by { |i| [pvalues[i], i] }
+      adjusted = Array.new(m)
+      running_max = 0.0
+      order.each_with_index do |original, rank|
+        candidate = (m - rank) * pvalues[original]
+        running_max = candidate if candidate > running_max
+        adjusted[original] = [running_max, 1.0].min
+      end
+      adjusted
+    end
+
+    def verdict(slowdown_pct, holm_p, threshold_pct)
+      slowdown_pct > threshold_pct && holm_p < ALPHA ? "fail" : "pass"
+    end
+
+    def analyze(series_list, seed:, threshold_pct:)
+      rng = Random.new(seed)
+      partials = series_list.sort_by(&:label).map do |series|
+        perm_seed = rng.rand(2**32)
+        boot_seed = rng.rand(2**32)
+        xs = log_ratios(series)
+        p_value = permutation_pvalue(xs, Random.new(perm_seed))
+        ci_low, ci_high = bootstrap_ci(xs, Random.new(boot_seed))
+        { label: series.label, effect: mean(xs), p_value: p_value,
+          ci_low: ci_low, ci_high: ci_high }
+      end
+
+      holm_ps = holm(partials.map { |partial| partial[:p_value] })
+
+      partials.zip(holm_ps).map do |partial, holm_p|
+        effect = partial[:effect]
+        slowdown_pct = (1 - Math.exp(effect)) * 100
+        EntryResult.new(
+          label: partial[:label],
+          effect: effect,
+          ci_low: partial[:ci_low],
+          ci_high: partial[:ci_high],
+          p_value: partial[:p_value],
+          holm_p: holm_p,
+          percent_change: (Math.exp(effect) - 1) * 100,
+          slowdown_pct: slowdown_pct,
+          verdict: verdict(slowdown_pct, holm_p, threshold_pct)
+        )
+      end
+    end
+  end
+end

data/lib/benchtrack/suite_runner.rb

@@ -0,0 +1,77 @@
+# frozen_string_literal: true
+
+require "open3"
+require "json"
+require "tmpdir"
+require "rbconfig"
+require "bundler"
+
+module BenchTrack
+  module SuiteRunner
+    SHIM = File.expand_path("ips_capture.rb", __dir__)
+
+    module_function
+
+    def run_suite(suite_path, chdir:, env: {})
+      unless File.file?(File.expand_path(suite_path, chdir))
+        raise SuiteError, "suite file not found: #{suite_path}"
+      end
+
+      Dir.mktmpdir("benchtrack") do |tmp|
+        result_path = File.join(tmp, "result.json")
+        argv = [RbConfig.ruby]
+        argv << "-rbundler/setup" if env.key?("BUNDLE_GEMFILE")
+        argv.push("-r", SHIM, suite_path)
+        out, status = Bundler.with_unbundled_env do
+          Open3.capture2e(env.merge("BENCHTRACK_RESULT_PATH" => result_path), *argv, chdir: chdir)
+        end
+        unless status.success?
+          raise SuiteError, "suite exited with status #{status.exitstatus || status}:\n#{out}"
+        end
+        unless File.file?(result_path)
+          raise SuiteError, "suite produced no result file (no Benchmark.ips call?)\nsuite output:\n#{out}"
+        end
+        parse_result(File.read(result_path), output: out)
+      end
+    end
+
+    def bundler_env(dir, bundle_path: nil)
+      gemfile = File.expand_path("Gemfile", dir)
+      gemfile = ENV["BUNDLE_GEMFILE"] unless File.file?(gemfile)
+      return {} if gemfile.nil? || gemfile.empty? || !File.file?(gemfile)
+
+      env = { "BUNDLE_GEMFILE" => File.expand_path(gemfile) }
+      env["BUNDLE_PATH"] = bundle_path if bundle_path
+      env
+    end
+
+    def parse_result(text, output: "")
+      suffix = output.empty? ? "" : "\nsuite output:\n#{output}"
+      data = begin
+        JSON.parse(text)
+      rescue JSON::ParserError => e
+        raise SuiteError, "invalid benchmark result JSON: #{e.message}#{suffix}"
+      end
+
+      unless data.is_a?(Hash) && data["format"] == 1 && data["entries"].is_a?(Array)
+        raise SuiteError, "unrecognized benchmark result format#{suffix}"
+      end
+
+      seen = {}
+      data["entries"].map do |item|
+        label = item.is_a?(Hash) ? item["label"] : nil
+        ips   = item.is_a?(Hash) ? item["ips"] : nil
+        unless label.is_a?(String) && !label.empty?
+          raise SuiteError, "benchmark entry has a missing or empty label#{suffix}"
+        end
+        unless ips.is_a?(Numeric) && ips.to_f.finite? && ips.to_f.positive?
+          raise SuiteError, "benchmark entry #{label.inspect} has invalid ips #{ips.inspect}#{suffix}"
+        end
+        raise SuiteError, "duplicate benchmark label #{label.inspect}#{suffix}" if seen[label]
+
+        seen[label] = true
+        Entry.new(label, ips.to_f)
+      end
+    end
+  end
+end

data/lib/benchtrack/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module BenchTrack
+  VERSION = "0.1.0"
+end

data/lib/benchtrack.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require_relative "benchtrack/version"
+
+module BenchTrack
+  Error        = Class.new(StandardError)
+  ConfigError  = Class.new(Error)
+  SuiteError   = Class.new(Error)
+  GitError     = Class.new(Error)
+  BundleError  = Class.new(Error)
+  PrepareError = Class.new(Error)
+
+  Entry        = Struct.new(:label, :ips)
+  Config       = Struct.new(:command, :suite, :base, :head, :threshold_pct,
+                            :blocks, :seed, :json_path, :prepare, keyword_init: true)
+  PairedSeries = Struct.new(:label, :base, :head)
+  EntryResult  = Struct.new(:label, :effect, :ci_low, :ci_high, :p_value, :holm_p,
+                            :percent_change, :slowdown_pct, :verdict, keyword_init: true)
+  Comparison   = Struct.new(:base_ref, :head_ref, :base_sha, :head_sha, :results,
+                            :base_only, :head_only, :overall, :fingerprint, :config,
+                            keyword_init: true)
+end
+
+require_relative "benchtrack/cli"
+require_relative "benchtrack/suite_runner"
+require_relative "benchtrack/orchestrator"
+require_relative "benchtrack/measurement"
+require_relative "benchtrack/stats"
+require_relative "benchtrack/report"

metadata

@@ -0,0 +1,58 @@
+--- !ruby/object:Gem::Specification
+name: benchtrack
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Ojus Chugh
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2026-09-13 00:00:00.000000000 Z
+dependencies: []
+description: Turns an existing benchmark-ips suite into a paired base-versus-head
+  performance comparison over git worktrees. Fails CI only when a slowdown is both
+  practically large and statistically supported.
+email: 
+executables:
+- benchtrack
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- exe/benchtrack
+- lib/benchtrack.rb
+- lib/benchtrack/cli.rb
+- lib/benchtrack/ips_capture.rb
+- lib/benchtrack/measurement.rb
+- lib/benchtrack/orchestrator.rb
+- lib/benchtrack/report.rb
+- lib/benchtrack/stats.rb
+- lib/benchtrack/suite_runner.rb
+- lib/benchtrack/version.rb
+homepage: https://github.com/ojuschugh1/benchtrack
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/ojuschugh1/benchtrack
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.3.1
+signing_key: 
+specification_version: 4
+summary: Pull-request-time performance regression testing for Ruby libraries
+test_files: []