duel 0.1.0

data/README.md

@@ -0,0 +1,354 @@
+# ⚔️ Duel: Performance regression guards for Ruby
+
+[![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
+[![Test](https://github.com/sferik/duel-ruby/actions/workflows/test.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/test.yml)
+[![Lint](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml)
+[![Mutant](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml)
+[![Typecheck](https://github.com/sferik/duel-ruby/actions/workflows/typecheck.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/typecheck.yml)
+[![Plugins](https://github.com/sferik/duel-ruby/actions/workflows/plugins.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugins.yml)
+[![Gem Version](https://badge.fury.io/rb/duel.svg)](https://rubygems.org/gems/duel)
+
+Duel runs your benchmarks on two versions of your code side by side and fails
+if the new code is slower. Add it to your app's CI and it will prevent your
+code from becoming slower.
+
+```sh
+$ duel
+Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02)
+  parse: a small document                            0.6s n=412    cpu 0.998 [0.991, 1.006] same  allocations 20
+! parse: a large document                            1.1s n=9      cpu 1.081 [1.062, 1.097] slower  allocations 5790
+...
+REGRESSION parse: a large document: worse cpu in every measurement
+  cpu: 112.4µs → 121.5µs an iteration (1.08x)
+12 benchmarks in 14s: 1 regressed, 0 errors, 0 with a timing too close to the limit to settle
+```
+
+## Never slower
+
+Duel exists to enforce a policy that the WebKit project sets out on its
+[Performance](https://webkit.org/performance/) page:
+
+> The way to make a program faster is to never let it get slower.
+>
+> We have a zero-tolerance policy for performance regressions. If a patch
+> lands that regresses performance according to our benchmarks, then the
+> person responsible must either back the patch out of the tree or drop
+> everything immediately and fix the regression.
+>
+> Common excuses people give when they regress performance are, "But the
+> new way is cleaner!" or "The new way is more correct." We don't care.
+> No performance regressions are allowed, regardless of the reason. There
+> is no justification for regressing performance. None.
+
+WebKit backs those words with its benchmarks, and with the discipline to
+treat a slowdown as seriously as a broken build. Duel brings that idea
+to Ruby. If you adopt the policy, Duel tells a pull request that it made
+your code slower before the pull request is merged. All credit for the
+idea belongs to the WebKit project, and any shortcomings in carrying it
+out are Duel's.
+
+## Install Duel
+
+1. Add Duel to the `development` group of your `Gemfile`:
+
+   ```ruby
+   # Gemfile
+   group :development do
+     gem "duel"
+   end
+   ```
+
+2. Run `bundle install`.
+
+3. Set Duel up in your project:
+
+   ```sh
+   bundle exec duel init
+   ```
+
+   It adds StackProf to your `Gemfile`, for profiles of regressions, so
+   run `bundle install` again.
+
+### Set up a project
+
+`duel init` sets Duel up in a gem or an app, and writes the following:
+
+| File | What it's for |
+|---|---|
+| `.duel.rb` | Duel's configuration. It builds a project that has an `ext/` directory with `rake compile`, and runs each version on its own Ruby when the project has a `.ruby-version`. Without a gemspec to name the library, it names the only file in `lib/`, or leaves the library for you to set. See [Configure Duel](docs/configuration.md). |
+| `duel/<library>_duel.rb` | An example benchmark to replace with your own, with advice on writing benchmarks that tell a regression from noise. See [Write a benchmark](#write-a-benchmark). |
+| `.github/workflows/perf.yml` | The GitHub Actions workflow in [Run Duel in CI](docs/ci.md#catch-slow-creep). For a project in a subdirectory of its repository, the workflow is at the repository's root, named after the subdirectory, and runs in it. |
+| `Rakefile` | If there is one, `duel init` adds Duel's Rake task to it, as `rake duel`. See [Run Duel in CI](#run-duel-in-ci). |
+| `Gemfile` | Unless it mentions StackProf already, `duel init` adds `gem "stackprof", platforms: :mri` to its `development` group (or, without one, to its end), so Duel can show where a regression spent its extra time. StackProf runs on CRuby only, so the gem isn't installed on JRuby or TruffleRuby. |
+
+`duel init` doesn't overwrite a file that exists. To overwrite it, use
+`--force`. To see what `duel init` would do without writing anything,
+use `--dry-run`. When it's done, `duel init` lists the next steps.
+
+For a Rails app, use
+[duel-rails](https://github.com/sferik/duel-ruby/tree/main/plugins/rails)
+instead: add it to your `Gemfile`, and run
+`bin/rails generate duel:install`.
+
+### Adopt Duel with an agent
+
+[AGENT_PROMPT.md](AGENT_PROMPT.md) is a prompt that tells a coding agent
+how to add Duel to your project. The agent does the following:
+
+* Picks the code that's worth guarding.
+* Writes benchmarks that don't vary from run to run.
+* Checks that the benchmarks pass against the unchanged code, and that
+  they catch a deliberate slowdown.
+* Adds the CI workflow.
+
+To use the prompt, paste it into the agent, or point the agent at the
+file. The prompt also ships with the gem. After you install Duel, it's
+in the directory that `bundle info duel --path` prints.
+
+## Write a benchmark
+
+By convention, add benchmarks in files named `*_duel.rb` in the `duel/`
+directory.
+
+```ruby
+# duel/parser_duel.rb
+duel "parse: a small document" do
+  MyParser.parse("[1, 2, 3]")
+end
+```
+
+Duel times the block that you pass to `duel`, and runs it as many times
+as a sample needs. To do work that you don't want timed, put it in a
+`once` block. Duel runs it once, the first time that the benchmark runs,
+before it measures anything:
+
+```ruby
+duel "parse: a large document" do
+  once { @input = File.read("test/fixtures/large.json") } # not timed
+  MyParser.parse(@input)                                  # timed
+end
+```
+
+If your code handles many items at once, give the block a parameter.
+Duel then runs the block once per sample, and passes it the number of
+iterations. To prepare each sample's work without timing it, wrap the
+preparation in `untimed`, which pauses measuring:
+
+```ruby
+duel "queue: drain the queue" do |n|
+  queue = untimed { MyQueue.new(Array.new(n) { |i| i }) } # not timed
+  queue.drain                                             # timed
+end
+```
+
+Benchmarks can do much more, such as run code around their samples or
+iterations, undo what each iteration changes, and time a whole process,
+such as your app's boot. For more information, see
+[Write benchmarks](docs/benchmarks.md).
+
+## Run Duel
+
+Run `duel` from your project's root. By default, it compares your
+working tree with the commit where `HEAD` left the main branch:
+
+```sh
+duel                       # against where HEAD left the main branch
+duel --base v1.4.0         # against any git ref
+duel --base last-release   # against the newest release tag before HEAD
+duel --filter "^parse:"    # only benchmarks whose names match (a regular expression)
+duel --base ../old         # against a tree on disk, already built
+```
+
+To see every option, such as `--processes`, `--timeout`, and `--limit`,
+run `duel --help`. To write the report in Markdown too, use
+`--markdown`. For more information, see
+[Report on a pull request](#report-on-a-pull-request). To write it as
+JSON, for a program to read, use `--json`. For more information, see
+[Read Duel's report as JSON](docs/json.md).
+
+Duel runs one comparison at a time in a project, because two runs would
+share the project's `tmp/duel` directory and its baseline's checkout. A
+second run fails and names the process that's running the first. A run
+that crashes leaves nothing behind that blocks the next run.
+
+Where it's hard to pass options, such as behind a Rake task or in a CI
+expression, set these environment variables instead:
+
+| Variable | Option |
+|---|---|
+| `DUEL_BASE` | `--base` |
+| `DUEL_FILTER` | `--filter` |
+| `DUEL_FORCE=1` | `--force` |
+
+An option on the command line takes precedence over its variable, and
+an empty variable counts as unset.
+
+For where the baseline comes from, how Duel checks it out and builds it,
+and how it compares a project in a subdirectory of its repository, see
+[Where the baseline comes from](docs/how-it-works.md#where-the-baseline-comes-from).
+
+### Timeouts and exit status
+
+A benchmark that hangs, or that waits on a server that died, doesn't
+hang the run. If a worker takes more than a minute to load or to answer,
+Duel stops the worker and reports the benchmark as an error. To change
+the limit, use `--timeout`.
+
+`duel` exits with one of the following statuses:
+
+| Status | Meaning |
+|---|---|
+| `0` | Nothing got slower, or there was nothing to compare. |
+| `1` | Something got slower, or a benchmark failed. |
+| `2` | Duel couldn't compare. |
+
+## Run Duel in CI
+
+Run `bundle exec duel` as a step in your CI job. Its exit status passes
+or fails the job, and its output shows what got slower, by how much, and
+where the extra time went. For example, on GitLab:
+
+```yaml
+duel:
+  script: bundle exec duel
+  variables:
+    GIT_DEPTH: 0 # the baseline comes from history
+```
+
+To run Duel from Rake, define a task with `Duel::RakeTask`. Any
+arguments after the task's name are passed to `duel`:
+
+```ruby
+require "duel/rake_task"
+Duel::RakeTask.new(:perf)
+Duel::RakeTask.new(:perf_since_release, "--base", "last-release")
+```
+
+`duel init` writes a GitHub Actions workflow,
+`.github/workflows/perf.yml`. It compares each pull request with the
+main branch, each push to `main` with the commit before it, and, on a
+schedule, your last release, to catch slow creep. It also comments on
+each pull request with Duel's report. For the workflow, and for a branch
+that upgrades Ruby, see [Run Duel in CI](docs/ci.md).
+
+### Report on a pull request
+
+On GitHub Actions, Duel adds its report to the job's summary, in
+Markdown, whatever the outcome: a comparison, nothing to compare, or an
+error. To write the same Markdown to a file, for example to post it as a
+comment, use `--markdown`:
+
+```sh
+duel --markdown tmp/duel.md
+```
+
+The report starts with the verdict and the two versions that Duel
+compared, with each version's Ruby when they differ. A table lists the
+benchmarks that changed, regressions first, with each timing's ratio and
+its interval, and the counts that changed. Each regression is explained
+as it is in the terminal, with what its metrics said and where the extra
+CPU time went. Then come the benchmarks whose timings were too close to
+call, and the benchmarks that failed. The benchmarks that didn't change
+are collapsed, so the report stays short with hundreds of benchmarks:
+
+````markdown
+### Duel failed: 1 regression
+
+- **Baseline:** `3f9c2a1`, where HEAD left `origin/main`
+- **Candidate:** `8e4d0b7`
+
+12 benchmarks in 14s. A timing is slower when a 99.0% confidence interval for its ratio, the candidate's to the baseline's, is above 1.02.
+
+| Benchmark | Verdict | Timings (ratio [interval]) | Counts an iteration |
+|---|---|---|---|
+| `parse: a large document` | **regressed** | instructions 1.071 [1.069, 1.073] slower<br>cpu 1.081 [1.062, 1.097] slower | allocations 5790 → 5791 |
+
+#### Regressions
+
+**`parse: a large document`**: worse instructions, cpu, allocations in every measurement
+
+```text
+instructions: 1.521e+06 → 1.629e+06 an iteration (1.07x)
+cpu: 112.4µs → 121.5µs an iteration (1.08x)
+allocations: 5790 → 5791 an iteration
+where the extra CPU went (as a share of the baseline's: in each frame itself, and with what it called):
+   +6.8%   +7.1%  MyParser::Lexer#next_token
+   +0.4%   +0.4%  String#byteslice
+```
+
+<details>
+<summary>11 benchmarks unchanged</summary>
+
+...
+
+</details>
+````
+
+The workflow that `duel init` writes posts this report as a comment on
+the pull request, and updates the same comment on every push. For more
+information, see
+[Comment on a pull request](docs/ci.md#comment-on-a-pull-request).
+
+### Show that a project is guarded
+
+To show that Duel guards your project, add its badge to your README,
+next to your other badges:
+
+```markdown
+[![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
+```
+
+The badge links to Duel. It doesn't show whether a comparison passed:
+your CI's status does that.
+
+## Plugins
+
+A plugin packages what a server or a library needs, such as a server for
+each worker, hooks, and metrics, for any project to use. The following
+plugins are available:
+
+| Plugin | Build | What it adds |
+|---|---|---|
+| [Rails](https://github.com/sferik/duel-ruby/tree/main/plugins/rails) | [![duel-rails](https://github.com/sferik/duel-ruby/actions/workflows/plugin-rails.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-rails.yml) | For a Rails app: each version booted like production, with its own bundle and database, and benchmarks of its requests, judged on its queries, views, and cache use too. |
+| [ActiveRecord](https://github.com/sferik/duel-ruby/tree/main/plugins/activerecord) | [![duel-activerecord](https://github.com/sferik/duel-ruby/actions/workflows/plugin-activerecord.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-activerecord.yml) | The queries, transactions, and records that each benchmark took, from Active Record's notifications (for example, an N+1), with any database. |
+| [PostgreSQL](https://github.com/sferik/duel-ruby/tree/main/plugins/postgresql) | [![duel-postgresql](https://github.com/sferik/duel-ruby/actions/workflows/plugin-postgresql.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-postgresql.yml) | A PostgreSQL server for each worker, and the queries, rows, blocks, and time that each benchmark took of it, by statement. |
+| [MySQL](https://github.com/sferik/duel-ruby/tree/main/plugins/mysql) | [![duel-mysql](https://github.com/sferik/duel-ruby/actions/workflows/plugin-mysql.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-mysql.yml) | A MySQL server for each worker, and the queries, rows (returned and examined), and time that each benchmark took of it, by statement. |
+| [Redis](https://github.com/sferik/duel-ruby/tree/main/plugins/redis) | [![duel-redis](https://github.com/sferik/duel-ruby/actions/workflows/plugin-redis.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-redis.yml) | A Redis server for each worker, and the Redis CPU time, commands, and round trips that each benchmark took. |
+| [Memcached](https://github.com/sferik/duel-ruby/tree/main/plugins/memcached) | [![duel-memcached](https://github.com/sferik/duel-ruby/actions/workflows/plugin-memcached.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-memcached.yml) | A memcached server for each worker, and the operations, misses, and bytes that each benchmark took of it. |
+| [GVL](https://github.com/sferik/duel-ruby/tree/main/plugins/gvl) | [![duel-gvl](https://github.com/sferik/duel-ruby/actions/workflows/plugin-gvl.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-gvl.yml) | How long the threads of a benchmark with threads of its own waited for the GVL, on CRuby. |
+| [Retained](https://github.com/sferik/duel-ruby/tree/main/plugins/retained) | [![duel-retained](https://github.com/sferik/duel-ruby/actions/workflows/plugin-retained.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/plugin-retained.yml) | The objects that each benchmark left alive (for example, a leak, or a cache that grows with every call), on CRuby. |
+
+To use a plugin, add its gem to your `Gemfile`, and call `c.use` in
+`.duel.rb` with the plugin's name and any of its options:
+
+```ruby
+# .duel.rb
+Duel.configure do |c|
+  c.use :redis
+  c.use :activerecord, query_time: true
+end
+```
+
+For what `c.use` does, and how a plugin's options work, see
+[Use plugins](docs/configuration.md#use-plugins). To write your own
+plugin, see [Write a Duel plugin](docs/plugins.md).
+
+## Learn more
+
+The [Duel guide](docs/README.md) covers the rest:
+
+- [Write benchmarks](docs/benchmarks.md): hooks, undoing each
+  iteration, sharing code, choosing what a benchmark is judged on, and
+  benchmarking a whole process.
+- [Configure Duel](docs/configuration.md): `.duel.rb`, preparing each
+  worker and each version, each version's own Ruby, your own metrics,
+  and plugins.
+- [Run Duel in CI](docs/ci.md): the GitHub Actions workflow, slow creep,
+  and comments on pull requests.
+- [How Duel works](docs/how-it-works.md): how it tells a regression from
+  noise, what it measures, profiles, where the baseline comes from, and
+  JIT compilers.
+- [Troubleshooting](docs/troubleshooting.md): what to do when Duel
+  can't compare, or reports what you don't expect.
+- [Write a Duel plugin](docs/plugins.md): plugin conventions, hooks,
+  metrics, and tests.

data/docs/README.md

@@ -0,0 +1,23 @@
+# Duel guide
+
+The [README](../README.md) shows how to install Duel, write a first
+benchmark, run it, and run it in CI. This guide covers the rest.
+
+| Page | What it covers |
+|---|---|
+| [Write benchmarks](benchmarks.md) | Hooks that run code around a benchmark, starting every sample alike, undoing each iteration, measuring work that a benchmark leaves for later, sharing code between benchmarks, choosing what a benchmark is judged on, and benchmarking a whole process. |
+| [Configure Duel](configuration.md) | `.duel.rb` and its settings, preparing each worker and each version, running code in each worker, running each version on its own Ruby, adding your own metrics, and using plugins. |
+| [Run Duel in CI](ci.md) | The GitHub Actions workflow that `duel init` writes, catching slow creep, commenting on a pull request, and installing the baseline's Ruby on CI. |
+| [How Duel works](how-it-works.md) | How Duel tells a regression from noise, what it measures, where the extra time went, where the baseline comes from, and JIT compilers, including JRuby and TruffleRuby. |
+| [Read Duel's report as JSON](json.md) | Writing the report as JSON with `--json`, its schema and how it's versioned, and an example. |
+| [Troubleshooting](troubleshooting.md) | What Duel says when it can't compare, or reports what you don't expect, and what to do about it. |
+| [Write a Duel plugin](plugins.md) | Plugin conventions, adding to the DSL, hooks and metrics in a plugin, explaining a sample, and testing a plugin. |
+
+## Public API
+
+What the README and this guide document is Duel's public API, which
+follows [Semantic Versioning](https://semver.org).
+
+> [!IMPORTANT]
+> Anything marked `@api private` in the code or in its signatures
+> (`sig/`) is Duel's own, and can change in any release.

data/docs/benchmarks.md

@@ -0,0 +1,289 @@
+# Write benchmarks
+
+This page covers what a benchmark can do beyond timing a block: run
+code around its samples and iterations, share code with other
+benchmarks, choose what it's judged on, and benchmark a whole process.
+For a first benchmark, with `once`, `untimed`, and a block that takes
+the number of iterations, see
+[Write a benchmark](../README.md#write-a-benchmark).
+
+## Run code around a benchmark
+
+A *hook* runs code around a benchmark's samples or iterations, such as
+setting up a client once, or emptying a queue before every sample. Duel
+has the following hooks:
+
+| Hook | When it runs, in each worker | Measured |
+|---|---|---|
+| `setup` | Once, before the benchmark's samples | No |
+| `teardown` | Once, after the benchmark's samples | No |
+| `before_sample` | Before every sample, warm-up samples included | No |
+| `around_iteration` | Around each iteration, which it's given to run | Only the iteration |
+| `after_iterations` | After each sample's iterations, or after each iteration with `around_iteration` | Yes |
+
+You can give a hook at three levels, with the same name at each:
+
+- In `.duel.rb`, for every benchmark: `c.before_sample { ... }`.
+- At the top of a file of benchmarks, for every benchmark in the file:
+  `before_sample { ... }`.
+- As an option of one benchmark, with a lambda:
+  `duel "name", before_sample: -> { ... } do ... end`.
+
+```ruby
+# duel/client_duel.rb
+setup { @client = MyClient.new(timeout: 1) }
+teardown { @client.close }
+
+duel("client: one request") { @client.get("/") }
+duel("client: ten requests", setup: -> { @client.warm }) { 10.times { @client.get("/") } }
+```
+
+When several levels give the same hook, Duel runs `.duel.rb`'s first,
+then the file's, then the benchmark's own, each in the order that it was
+given. `teardown` runs in the reverse order, so what's set up last is
+torn down first. `around_iteration` hooks nest, outermost first: the
+hook from `.duel.rb` runs around the file's, which runs around the
+benchmark's.
+
+A file's hooks and a benchmark's own run in the benchmark's context, so
+they share its instance variables. Hooks in `.duel.rb` run where they're
+written, which suits a plugin's hooks. `setup`, `teardown`, and
+`before_sample` receive the benchmark, with its `name`, `only`, and
+`peak_memory`, if they take a parameter.
+
+Inside a benchmark's own block, `once { ... }` runs its block the first
+time that the benchmark runs, and isn't timed, as in
+[Write a benchmark](../README.md#write-a-benchmark). It's for what one
+benchmark needs, in its own block, where the `setup` hook is for what
+it needs before its block runs, or for what every benchmark in a file
+needs.
+
+A process benchmark (`duel_process`) doesn't run in a worker, so it
+runs no hooks. See [Benchmark a whole process](#benchmark-a-whole-process).
+
+## Start every sample alike
+
+A worker measures each benchmark over many samples. If a benchmark
+changes something that later samples see, such as rows that it inserts
+or items that it pushes onto a queue, later samples measure different
+work. A `before_sample` hook puts back what earlier samples changed, so
+that every sample starts alike. Duel doesn't measure it:
+
+```ruby
+# duel/queue_duel.rb
+setup { @queue = MyQueue.new }
+before_sample { @queue.clear } # not measured
+
+duel("queue: push") { @queue.push(1) }
+```
+
+To start every sample of every benchmark alike, for example by emptying
+a database's tables, give the hook in `.duel.rb`:
+
+```ruby
+Duel.configure do |c|
+  c.before_sample { |benchmark| MyDatabase.truncate_all } # not measured
+end
+```
+
+## Undo each iteration
+
+Some work changes what the next iteration sees, such as a request that
+creates a record. To undo each iteration without measuring the undo,
+use an `around_iteration` hook. Duel runs the hook around each
+iteration, and the hook calls `iteration.call` exactly once. Duel
+measures only the iteration: nothing else that the hook does is timed
+or counted.
+
+```ruby
+rolled_back = lambda do |iteration|
+  Post.transaction do
+    iteration.call
+    raise ActiveRecord::Rollback
+  end
+end
+
+duel "posts: create", around_iteration: rolled_back do
+  Post.create!(title: "Hello")
+end
+```
+
+`iteration.call` returns what the iteration returns. A benchmark that
+runs inside an `around_iteration` hook, from any level, can't take a
+block parameter, because the hook runs once for each iteration.
+
+To leave out what the hook does, Duel pauses every metric after each
+iteration and resumes it before the next one. That reads each metric
+twice per iteration, and part of that reading lands inside the
+iteration. Duel learns how much in each worker, before it measures the
+benchmark, and subtracts it from every iteration. Pausing allocates
+nothing, so counts stay exact.
+
+A metric that's costly to read, such as one that queries a database
+server's statistics, would take a round trip to the server twice per
+iteration. So Duel doesn't pause a metric that says it's costly (see
+[Account for the cost of reading](configuration.md#account-for-the-cost-of-reading)).
+It reads that metric only when a sample starts and stops, and subtracts
+what the hook adds to it instead. Duel learns that in each worker,
+before it measures the benchmark, from a few samples of their own, in
+which it pauses the metric around each iteration. The hook's own work,
+such as a transaction's `BEGIN` and `ROLLBACK`, isn't counted, and
+neither is work of the hook's that depends on the iteration, such as
+deleting what the iteration wrote. The plugins for servers, such as
+[PostgreSQL](https://github.com/sferik/duel-ruby/tree/main/plugins/postgresql),
+say that their metrics are costly.
+
+The pausing still adds a little to each iteration, the same on both
+versions, and it takes time that Duel doesn't measure. On an Apple M1
+Max, with macOS and Ruby 3.4, an empty iteration with an
+`around_iteration` hook measures less than 100 ns of CPU time and a few
+hundred instructions more than an empty iteration without one, and no
+more allocations. It also takes from 2 µs to 50 µs longer to run,
+depending on how busy the machine is, mostly to read the CPU's
+instruction counter, which takes a system call on macOS. Use
+`around_iteration` for iterations that take tens of microseconds or
+more, such as a database write. For cheaper work, give the block a
+parameter, and prepare each sample's work in `untimed`.
+
+## Measure the work that a benchmark leaves for later
+
+Some of the work that a benchmark starts runs later, such as jobs that
+it enqueues, or messages that it broadcasts. To measure that work as
+part of the benchmark, give an `after_iterations` hook that does it
+now:
+
+```ruby
+Duel.configure do |c|
+  c.after_iterations { MyJobs.run_pending } # measured: the jobs are the benchmark's work
+end
+```
+
+`after_iterations` is the one hook that Duel measures. It runs inside
+the timed part of each sample, after the sample's iterations, and Duel
+measures it with them. For a benchmark with an `around_iteration` hook,
+it runs inside each iteration, after the iteration's own work, so that
+what the hook undoes after the iteration is still there for the pending
+work. Its work counts as the benchmark's, in every metric and in a
+regression's profile.
+
+## Share code between benchmarks
+
+Put code that several files share in `duel/duel_helper.rb`, and require
+it from each file, the same way that tests require `test_helper.rb`.
+Every benchmark can call the methods that the helper defines at its top
+level. Instance variables that those methods set belong to the
+benchmark that calls them:
+
+```ruby
+# duel/duel_helper.rb
+def document(size) = "[" + (["1"] * size).join(",") + "]"
+
+def build_parser = @parser = MyParser.new(strict: true)
+
+# duel/parser_duel.rb
+require_relative "duel_helper"
+
+duel("parse: a thousand numbers") { MyParser.parse(document(1000)) }
+
+duel "parse: strictly" do
+  once { build_parser }
+  @parser.parse("[1]")
+end
+```
+
+A helper can have any name except those of the methods that Duel gives
+benchmarks: `duel`, `duel_process`, `once`, `untimed`, the hooks
+(`setup`, `teardown`, `before_sample`, `around_iteration`, and
+`after_iterations`), and any that a plugin used in `.duel.rb` adds.
+Duel's method would hide a helper of the same name, so the worker
+fails, and names the helper's file and line.
+
+## Choose what a benchmark is judged on
+
+By default, Duel judges each benchmark on its timings and its counts:
+
+- Timings: CPU time, instructions, and the timings of any metric of your
+  own.
+- Counts: allocations, and any other exact counts.
+
+To change that, pass these options to `duel`:
+
+- `only: :count`: for work that's mostly waiting, too slow and light to
+  time. Duel judges the benchmark on its counts alone, over a few
+  iterations.
+- `only: :timing`: for work done by threads that the benchmark starts,
+  which make its counts vary from run to run. Duel judges the benchmark
+  on its timings alone, and takes the process's CPU time instead of the
+  benchmark thread's.
+- `peak_memory: true`: also judges the benchmark's peak memory, which is
+  the most resident memory that its measured work reached, above where
+  it started. Duel takes each version's lowest peak from three fresh
+  workers. Like every other measure, peak memory leaves out what
+  `once` and `untimed` blocks leave behind.
+
+```ruby
+duel "server: 10 workers", only: :timing, peak_memory: true do |n|
+  # ...
+end
+```
+
+## Benchmark a whole process
+
+To guard how long your app takes to boot, or your library to load, use
+`duel_process`. Each sample starts fresh processes for each version,
+in that version's directory and with its environment, and the two
+versions take turns, as they do for every other benchmark. Give
+`duel_process` a command, or Ruby code to run with `ruby:`:
+
+```ruby
+# duel/boot_duel.rb
+duel_process "boot: the app", "bin/rails", "runner", "1"
+duel_process "boot: require the library", ruby: %(require "my_gem")
+```
+
+Duel runs a command the way that `Process.spawn` does, so a single
+string with spaces runs in a shell. It runs Ruby code on the version's
+Ruby, with the version's `lib/` first on the load path.
+
+Duel judges a process benchmark on the following:
+
+- `cpu`: the CPU time that the process took, including the CPU time of
+  the processes that it started and waited for.
+- `peak_memory`: the most resident memory that the process reached,
+  when Duel's runner is CRuby. Duel takes each version's lowest peak.
+
+Each sample is the fastest of a few processes for each version, with
+the two versions' processes taking turns: the lowest CPU time, and the
+lowest peak memory. Interference from the rest of the machine only ever
+adds time, so the fastest of a few is steadier than any one, and fewer
+pairs settle the verdict. A sample takes as many processes as take
+about a second of CPU time, up to 3, judging by a first pair of
+processes that Duel doesn't measure. A short process, whose time varies
+the most, gets 3, and an app that takes a second or more to boot gets
+1, since its pairs take long enough as they are. To set the number
+yourself, pass `runs:`:
+
+```ruby
+duel_process "boot: require the library", ruby: %(require "my_gem"), runs: 5
+```
+
+Duel starts and measures the processes from a small Ruby process of its
+own, one for each worker, that lasts as long as the worker. It does
+nothing else while a process runs, so the CPU time that it reads is the
+process's alone, and its own start-up is paid once, not for every
+process.
+
+A process that exits with a status other than 0 is an error, and Duel
+shows the end of its output. A process benchmark has no profile.
+
+Each process takes a while, so Duel judges a process benchmark after
+every pair of samples, and stops as soon as it finds that the
+candidate isn't slower. It finds a process benchmark slower only after
+a round of 10 pairs, as it does any benchmark, so stopping early never
+makes the same code look slower.
+
+A process benchmark doesn't run in a worker, so nothing that runs there
+applies to it: hooks, from `.duel.rb` or its file, and the metrics that
+plugins add. `duel_process` doesn't take the options of `duel`
+(`only:`, `peak_memory:`, and the hooks, such as `around_iteration:`),
+and raises an `ArgumentError` if it's given one.