duel 0.1.0

data/docs/how-it-works.md

@@ -0,0 +1,207 @@
+# How Duel works
+
+This page explains how Duel decides that a change made your code slower,
+what it measures, how it explains a regression, where the baseline comes
+from, and how it handles JIT compilers.
+
+## How Duel tells a regression from noise
+
+A policy of zero tolerance only works if the guard never cries wolf.
+After one false alarm, people learn to rerun until the build is green.
+So Duel is built to fail only when your code really got slower, and
+never on noise, so that it can gate every pull request:
+
+- **CPU time, not elapsed time.** Duel times each benchmark by its
+  thread's CPU time, which other processes on the machine don't
+  inflate. A sample is the fastest of several runs, because interference
+  only ever adds time. Garbage collection is kept out of samples, and
+  its cost shows in the allocation count instead, which is exact.
+- **Side by side, many times over.** The two versions run in worker
+  processes of their own, three of each by default, at the same time on
+  the same machine. They take turns: one sample from each, in
+  alternating order, and each pair from a different pair of processes.
+  That way, a busy moment weighs on both versions alike, and no one
+  process's luck decides anything. (Where things land in a process's
+  memory can make the very same code a few percent faster or slower for
+  the process's whole life.) There are no stored numbers and no
+  thresholds to maintain, and any machine will do.
+- **A confidence interval, not an average.** A timing is slower only
+  when a 99% distribution-free confidence interval for the median of
+  the paired ratios (candidate / baseline) lies entirely above the
+  limit, which is 1.02 by default. No one outlier sample can decide it.
+  Duel takes samples in rounds of 10 pairs, up to 60, until the interval
+  settles the question. A process benchmark's sample is the fastest of
+  a few fresh processes, and it can also stop after any pair once the
+  interval shows that it isn't slower, which only ever ends a
+  measurement before it could find a slowdown.
+  Duel reports a timing that's still too close to the limit to call, and
+  doesn't fail on it.
+- **Measured again before failing.** Duel measures whatever looks
+  slower again from scratch, up to three times, in fresh processes. It
+  fails only if every measurement finds the same metric worse. A
+  measurement whose timings are all still too close to call neither
+  confirms a slowdown nor clears it, so Duel measures once more in its
+  place, up to three more times. Every measurement that counts still has
+  to find it worse, so identical code, which only rarely looks slower in
+  any one measurement, still has to look slower in four or more.
+- **Slower in time, whichever timing shows it.** A slowdown near the
+  limits can be slower in CPU time in one measurement and in
+  instructions in the next, with the other timing too close to call
+  each time. Duel fails on that too, if in every measurement a timing
+  is slower and none is the same or faster. Any timing can be the
+  slower one, not only one in particular, so Duel measures once more
+  before it fails, which keeps identical code from failing more often
+  than it otherwise would.
+- **Exact counts.** Allocations (on CRuby), and any counted metric of
+  your own, fail on any increase: one more allocation per two
+  iterations.
+- **Across the board, too.** A slowdown that's too small for any one
+  benchmark to settle, such as 1% everywhere, still shows in all of the
+  benchmarks together. For each metric that's judged by ratio, every
+  benchmark's median ratio is a sample. The candidate fails when a
+  confidence interval for their median lies above a quarter of the
+  metric's margin: 1.005, for CPU time's 1.02. This takes 8 benchmarks,
+  so with fewer, Duel doesn't report it. Duel measures again before it
+  fails, as it does for the rest.
+
+## What Duel measures
+
+Duel judges every benchmark on the following metrics:
+
+| Metric | What it is | A regression |
+|---|---|---|
+| `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
+| `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
+| `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
+| `peak_memory` | The most resident memory, above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
+| Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
+
+Instruction counts barely vary from run to run, so they catch slowdowns
+that are too small for CPU time to settle. CPU time still catches what
+instructions miss, such as cache misses.
+
+Instruction counts come from the CPU's counters, where the operating
+system lets Duel read them:
+
+- On Linux, they're the benchmark thread's, through `perf_event_open`,
+  which many virtual machines, including CI runners, don't offer. So
+  Linux doesn't count instructions for benchmarks with threads of their
+  own.
+- On macOS, they're the whole process's.
+- Elsewhere, and on JRuby and TruffleRuby, there's no `instructions`
+  metric, and CPU time judges alone.
+
+## Where the extra time went
+
+Duel shows what regressed per iteration, on each version, with what a
+metric said of it if it had something to say. For more information, see
+[Add your own metrics](configuration.md#add-your-own-metrics). When something is slower
+in time, Duel profiles both versions over the same iterations with
+[StackProf](https://github.com/tmm1/stackprof), if it's in your bundle
+(`duel init` and duel-rails's generator add it to your `Gemfile`), and
+shows where the extra CPU time went:
+
+```
+REGRESSION parse: a large document: worse instructions, cpu in every measurement
+  instructions: 1.521e+06 → 1.629e+06 an iteration (1.07x)
+  cpu: 112.4µs → 121.5µs an iteration (1.08x)
+  where the extra CPU went (as a share of the baseline's: in each frame itself, and with what it called):
+     +6.8%   +7.1%  MyParser::Lexer#next_token
+     +0.4%   +0.4%  String#byteslice
+```
+
+A profile covers only the work that Duel measures. Work inside
+`untimed`, and what an `around_iteration` hook does before and after
+each iteration, is left out. What an `after_iterations` hook does is in
+it, because Duel measures that with the iterations. The garbage collector is held off
+during each sample, as it is when Duel measures. Collecting what each
+sample left behind isn't the benchmark's code, so it isn't ranked with
+the frames. If it changed, Duel shows it on a line of its own, `in
+garbage collection, apart from the code`. On macOS, the signal that a CPU-time interval
+has passed arrives only when the process calls into the kernel, so a
+CPU-time profile would blame whatever code makes system calls. There,
+Duel profiles elapsed time instead, which a benchmark that's busy on the
+CPU spends the same way.
+
+When no one timing is slower in every measurement, but every
+measurement has one that's slower (see
+[How Duel tells a regression from noise](#how-duel-tells-a-regression-from-noise)),
+the report names the timings that were:
+
+```
+REGRESSION parse: a large document: slower in every measurement, by one of cpu, instructions
+```
+
+On each benchmark's line, counts such as `allocations` are per
+iteration, as they are in the explanation of a regression.
+
+## Where the baseline comes from
+
+The main branch is the one that `origin/HEAD` names. Without
+`origin/HEAD`, Duel uses `origin/main`, and then `origin/master`. In a
+repository without a remote, it uses `main`, and then `master`. To name
+another branch, set `c.main_branch`.
+
+Duel checks out a git baseline as a git worktree next to your project,
+under a hidden name that's as long as your project's directory name, for
+example `.3f9c2a1b` next to `my_parser`. Duel keeps it there for the
+next run. If your project has an `ext/` directory, Duel builds the
+baseline with `bundle exec rake compile`. To remove the baseline, git's
+record of it, and the rest of `tmp/duel`, run `duel --clean`.
+
+If your project is in a subdirectory of its repository, as in a
+monorepo, run `duel` in that subdirectory. Duel checks out the whole
+repository next to the repository's directory, and the baseline is the
+same subdirectory in that checkout. The paths in `c.watch` are from your
+project's directory.
+
+If `HEAD` and the main branch have no commit in common, Duel says so.
+This happens in a shallow clone, which doesn't have the history that the
+baseline comes from. To fix it, fetch the whole history. For example, on
+GitHub Actions, set `fetch-depth: 0` for `actions/checkout`.
+
+The baseline's path is as long and as deep as your project's, in a
+subdirectory too, because what code does with its own file paths, such
+as looking for a file that a request might be for, takes longer on a
+longer or deeper path. A
+baseline at a longer path would measure slower, and hide a regression
+that small. If Duel can't write next to your project, or something else
+already has that name, it uses `tmp/duel/base` instead. Whenever the two
+versions' paths differ in length, Duel says so.
+
+If nothing under `lib/` or `ext/` changed since the baseline, there's
+nothing to compare, and Duel says so. To compare anyway, use `--force`.
+
+Duel also builds the candidate before every run. Builds are incremental,
+so a build that's already done costs little. A stale or missing build,
+or a C extension that your code quietly does without, never decides a
+comparison.
+
+Duel also checks that each version's workers load only native extensions
+that were built in that version's directory. A version without a build
+of its own would quietly load another build, such as the other
+version's, or an installed gem's, and the comparison wouldn't be of its
+code. If that happens, Duel stops and tells you to set `c.build`. An
+installed gem's extension is never taken for a version's build, even
+when the gems are installed in the project's directory, as
+`vendor/bundle` (where setup-ruby's `bundler-cache` puts them).
+
+## JIT compilers
+
+If YJIT is on in a worker, Duel runs each benchmark for two seconds
+before it measures it. A small library's methods are compiled within
+Duel's first samples, but in a big app, YJIT compiles a method only after
+120 calls. YJIT can be on because you turned it on (`RUBY_YJIT_ENABLE=1`,
+for example), or because your code did as it loaded, as Rails 7.2 and
+later do. To warm up for a different time, use `--warm`.
+
+### JRuby and TruffleRuby
+
+JRuby's and TruffleRuby's JIT compilers keep compiling for seconds, so
+Duel runs each benchmark for five seconds before it measures it
+(`--warm`). Their timings stay noisier than CRuby's, so Duel often
+leaves small timing regressions on them unsettled.
+
+Neither has an allocation count, an instruction count, or profiles, and
+each has peak memory only on Linux. StackProf runs on CRuby only: Duel
+doesn't use it on JRuby or TruffleRuby, even where it's installed.

data/docs/json.md

@@ -0,0 +1,306 @@
+# Read Duel's report as JSON
+
+To read Duel's results from a program, such as a dashboard, a bot, or a
+script that tracks benchmarks over time, write its report as JSON with
+`--json`:
+
+```sh
+duel --json tmp/duel.json
+```
+
+The JSON report has the same results as the
+[Markdown report](../README.md#report-on-a-pull-request), and more of
+each: every timing's ratio, interval and verdict, each version's medians
+an iteration, and every count. A relative path is relative to where
+`duel` runs, and Duel makes the file's directory if it needs to. Each
+run replaces the file.
+
+Duel writes the report whatever the outcome: a comparison that passed or
+failed, nothing to compare, or a comparison that couldn't run. It writes
+nothing when the command line isn't valid, such as an unknown option,
+and nothing for `--clean`, `--version`, or `duel init`, which aren't
+comparisons. So a missing report means that `duel` didn't get as far as
+comparing, or crashed.
+
+## Versions of the schema
+
+The report's `schema` key is the version of its schema, now `1`. A new
+key can appear in any release without a new version, so ignore the keys
+that you don't know. Any other change, such as a key removed or renamed,
+or a value that means something else, comes with a new version. Check
+`schema` before you read the rest, and stop if it isn't one that you
+know.
+
+The report uses only plain JSON types: objects, arrays, strings, numbers,
+`true`, `false`, and `null`. Numbers are numbers, never strings. A key
+that doesn't apply, such as a skipped benchmark's `iterations`, is
+`null`, not missing, so every object of a kind has the same keys.
+
+## An example
+
+A run that found one regression, abridged to two benchmarks:
+
+```json
+{
+  "schema": 1,
+  "duel_version": "0.1.0",
+  "status": "failed",
+  "exit": 1,
+  "message": null,
+  "settings": {
+    "confidence": 0.99,
+    "limit": 1.02,
+    "filter": "",
+    "jobs": 4,
+    "processes": 3,
+    "timeout": 60.0,
+    "warm": null,
+    "force": false
+  },
+  "seconds": 14.2,
+  "baseline": {
+    "ref": "origin/main",
+    "commit": "3f9c2a1d5e8b7c6a4f3e2d1c0b9a8f7e6d5c4b3a",
+    "root": "/home/runner/work/my_parser/.3f9c2a1b",
+    "ruby": "ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux]"
+  },
+  "candidate": {
+    "commit": "8e4d0b7f1a2c3e4d5b6a7f8e9d0c1b2a3f4e5d6c",
+    "uncommitted_changes": false,
+    "root": "/home/runner/work/my_parser/my_parser",
+    "ruby": "ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux]"
+  },
+  "plugins": [],
+  "benchmarks": [
+    {
+      "name": "parse: a large document",
+      "verdict": "regressed",
+      "passed": false,
+      "regressed": ["instructions", "cpu", "allocations"],
+      "unsettled": false,
+      "iterations": 9,
+      "seconds": 3.1,
+      "confirmations": 3,
+      "timed": {
+        "instructions": {
+          "ratio": 1.071,
+          "interval": [1.069, 1.073],
+          "limit": 1.01,
+          "pairs": 10,
+          "verdict": "slower",
+          "baseline": 1521000.0,
+          "candidate": 1629000.0,
+          "unit": null,
+          "plural": null,
+          "across": true
+        },
+        "cpu": {
+          "ratio": 1.081,
+          "interval": [1.062, 1.097],
+          "limit": 1.02,
+          "pairs": 10,
+          "verdict": "slower",
+          "baseline": 112400.0,
+          "candidate": 121500.0,
+          "unit": "ns",
+          "plural": "ns",
+          "across": true
+        }
+      },
+      "counted": {
+        "allocations": {
+          "baseline": 5790.0,
+          "candidate": 5791.0,
+          "unit": null,
+          "plural": null,
+          "worse": true
+        }
+      },
+      "peak_memory": null,
+      "details": {},
+      "explanation": [
+        "instructions: 1.521e+06 → 1.629e+06 an iteration (1.07x)",
+        "cpu: 112.4µs → 121.5µs an iteration (1.08x)",
+        "allocations: 5790 → 5791 an iteration",
+        "where the extra CPU went (as a share of the baseline's: in each frame itself, and with what it called):",
+        "   +6.8%   +7.1%  MyParser::Lexer#next_token"
+      ],
+      "error": null,
+      "skipped": null
+    },
+    {
+      "name": "parse: an empty document",
+      "verdict": "same",
+      "passed": true,
+      "regressed": [],
+      "unsettled": false,
+      "iterations": 2400,
+      "seconds": 0.8,
+      "confirmations": 0,
+      "timed": {
+        "cpu": {
+          "ratio": 1.002,
+          "interval": [0.994, 1.009],
+          "limit": 1.02,
+          "pairs": 10,
+          "verdict": "same",
+          "baseline": 410.5,
+          "candidate": 411.2,
+          "unit": "ns",
+          "plural": "ns",
+          "across": true
+        }
+      },
+      "counted": {
+        "allocations": {
+          "baseline": 3.0,
+          "candidate": 3.0,
+          "unit": null,
+          "plural": null,
+          "worse": false
+        }
+      },
+      "peak_memory": null,
+      "details": {},
+      "explanation": [],
+      "error": null,
+      "skipped": null
+    }
+  ],
+  "across": {
+    "cpu": {
+      "ratio": 1.004,
+      "interval": [0.998, 1.012],
+      "limit": 1.005,
+      "benchmarks": 12,
+      "verdict": "undecided",
+      "confirmed": false
+    }
+  },
+  "errors": []
+}
+```
+
+A run with nothing to compare has the same keys, with nothing compared:
+
+```json
+{
+  "schema": 1,
+  "duel_version": "0.1.0",
+  "status": "nothing_to_compare",
+  "exit": 0,
+  "message": "duel: lib, ext unchanged since the baseline, so there's nothing to compare (--force to anyway)",
+  "settings": {"confidence": 0.99, "limit": 1.02, "filter": "", "jobs": 4, "processes": 3, "timeout": 60.0, "warm": null, "force": false},
+  "seconds": null,
+  "baseline": null,
+  "candidate": null,
+  "plugins": [],
+  "benchmarks": [],
+  "across": {},
+  "errors": []
+}
+```
+
+For example, to list the benchmarks that regressed, with
+[jq](https://jqlang.org):
+
+```sh
+jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
+```
+
+## The report
+
+| Key | What it is |
+|---|---|
+| `schema` | The schema's version: `1`. |
+| `duel_version` | The version of Duel that wrote the report. |
+| `status` | `"passed"`, `"failed"`, `"nothing_to_compare"`, or `"error"` (Duel couldn't compare). |
+| `exit` | The exit status of `duel`: `0` for passed or nothing to compare, `1` for failed, `2` for an error. |
+| `message` | What Duel said, for nothing to compare or an error, such as `"duel: no release tag (v*) before HEAD to compare against"`. Otherwise `null`. |
+| `settings` | The settings that the run used (see below). |
+| `seconds` | How long the comparison took. `null` without a comparison. |
+| `baseline`, `candidate` | The two versions (see below). `null` without a comparison. |
+| `plugins` | The names of the plugins in use, such as `["rails", "activerecord"]`. |
+| `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
+| `across` | The verdicts across the board, by metric (see below). |
+| `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
+
+A run fails when a benchmark regressed, a benchmark failed, or a metric
+was slower across the board and confirmed (`"confirmed": true`).
+
+### Settings
+
+| Key | What it is |
+|---|---|
+| `confidence` | The confidence of each interval, such as `0.99`. |
+| `limit` | The ratio above which a timing is slower (`--limit`). A metric can have its own limit, which its timings show. |
+| `filter` | The regular expression that picked the benchmarks (`--filter`), as source. `""` matches every one. |
+| `jobs` | Benchmarks measured at once (`--jobs`). |
+| `processes` | Worker processes for each version (`--processes`). |
+| `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
+| `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default, which depends on the Ruby. |
+| `force` | Whether `--force` was given. |
+
+### The versions
+
+| Key | What it is |
+|---|---|
+| `ref` | The baseline's git ref: the one `--base` named, the release tag for `--base last-release`, or the main branch, such as `"origin/main"`, by default, where the baseline is the commit where `HEAD` left that branch. `null` for a baseline directory. |
+| `commit` | The full hash of the commit compared. `null` for a baseline directory, or for a candidate outside a git repository. |
+| `uncommitted_changes` | The candidate only: whether what `c.watch` names has changes that aren't committed. `null` outside a git repository. |
+| `root` | The tree that was compared, as an absolute path. |
+| `ruby` | The version's Ruby, as its `RUBY_DESCRIPTION`. |
+
+### Benchmarks
+
+| Key | What it is |
+|---|---|
+| `name` | The benchmark's name. |
+| `verdict` | `"regressed"`, `"error"`, `"unsettled"` (a timing too close to its limit to settle), `"faster"`, `"changed"` (a count changed), `"skipped"`, or `"same"`, as in the Markdown report's table. |
+| `passed` | Whether it regressed on nothing and didn't fail. |
+| `regressed` | The metrics that it regressed on, in every measurement, such as `["cpu", "allocations"]`. Peak memory is `"peak_memory"`. |
+| `unsettled` | Whether a timing was too close to its limit to settle. |
+| `iterations` | The iterations in each sample. `null` for a benchmark that failed or was skipped. |
+| `seconds` | How long its first measurement took. `null` for a benchmark that failed or was skipped. |
+| `confirmations` | How many times it was measured again, from scratch, because it looked worse. |
+| `timed` | Its timings, by metric (see below). |
+| `counted` | Its counts, by metric (see below). |
+| `peak_memory` | Each version's peak memory, as `{"baseline": bytes, "candidate": bytes, "unit": "B", "worse": true or false}`, for a benchmark that measures it. Otherwise `null`. |
+| `details` | What metrics said of the last samples, by metric and then by version, such as `{"allocations": {"baseline": "2 String", "candidate": "3 String"}}`. |
+| `explanation` | For a regression, the lines that explain it, as the terminal report and the Markdown report show them. Otherwise `[]`. |
+| `error` | Why it failed, with any backtrace. Otherwise `null`. |
+| `skipped` | Why it was skipped, such as a benchmark that isn't on the baseline. Otherwise `null`. |
+
+Each timing in `timed`, such as `cpu`:
+
+| Key | What it is |
+|---|---|
+| `ratio` | The median of the paired ratios, the candidate's to the baseline's. Above 1 is the candidate slower. |
+| `interval` | The confidence interval for that median, as `[low, high]`. |
+| `limit` | The ratio above which it's slower. |
+| `pairs` | The pairs of samples measured. |
+| `verdict` | `"slower"`, `"faster"`, `"same"`, or `"undecided"` (too close to the limit to settle). |
+| `baseline`, `candidate` | Each version's median for one iteration, in `unit`. |
+| `unit`, `plural` | The metric's unit, singular and plural, such as `"ns"`, or `null` for a plain number. |
+| `across` | Whether it counts across the board. |
+
+Each count in `counted`, such as `allocations`:
+
+| Key | What it is |
+|---|---|
+| `baseline`, `candidate` | Each version's fewest for one iteration, in `unit`. |
+| `unit`, `plural` | The metric's unit, singular and plural, such as `"query"` and `"queries"`, or `null` for a plain number. |
+| `worse` | Whether the candidate's count is worse, by more than the metric tolerates. |
+
+### Across the board
+
+Each metric in `across`, such as `cpu`:
+
+| Key | What it is |
+|---|---|
+| `ratio` | The median of every benchmark's ratio. |
+| `interval` | The confidence interval for that median, as `[low, high]`, or `null` with too few benchmarks to tell. |
+| `limit` | The ratio above which it's slower across the board. |
+| `benchmarks` | The benchmarks that it's from. |
+| `verdict` | `"slower"`, `"faster"`, `"same"`, or `"undecided"`. |
+| `confirmed` | Whether it was slower again when every benchmark was measured a second time, which fails the run. |