duel 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. checksums.yaml +7 -0
  2. data/AGENT_PROMPT.md +211 -0
  3. data/CHANGELOG.md +281 -0
  4. data/LICENSE +21 -0
  5. data/README.md +354 -0
  6. data/docs/README.md +23 -0
  7. data/docs/benchmarks.md +289 -0
  8. data/docs/ci.md +141 -0
  9. data/docs/configuration.md +429 -0
  10. data/docs/how-it-works.md +207 -0
  11. data/docs/json.md +306 -0
  12. data/docs/plugins.md +408 -0
  13. data/docs/troubleshooting.md +105 -0
  14. data/exe/duel +6 -0
  15. data/lib/duel/across.rb +48 -0
  16. data/lib/duel/baseline.rb +166 -0
  17. data/lib/duel/cli.rb +264 -0
  18. data/lib/duel/config.rb +343 -0
  19. data/lib/duel/context.rb +630 -0
  20. data/lib/duel/init.rb +211 -0
  21. data/lib/duel/json_report.rb +107 -0
  22. data/lib/duel/lock.rb +38 -0
  23. data/lib/duel/metric.rb +496 -0
  24. data/lib/duel/native.rb +162 -0
  25. data/lib/duel/process_sample.rb +144 -0
  26. data/lib/duel/profile.rb +116 -0
  27. data/lib/duel/rake_task.rb +25 -0
  28. data/lib/duel/resource.rb +209 -0
  29. data/lib/duel/rubies.rb +149 -0
  30. data/lib/duel/runner.rb +1114 -0
  31. data/lib/duel/stats.rb +73 -0
  32. data/lib/duel/summary.rb +237 -0
  33. data/lib/duel/templates/benchmark_duel.rb.erb +34 -0
  34. data/lib/duel/templates/duel.rb.erb +35 -0
  35. data/lib/duel/templates/perf.yml.erb +97 -0
  36. data/lib/duel/testing.rb +241 -0
  37. data/lib/duel/version.rb +5 -0
  38. data/lib/duel/worker.rb +211 -0
  39. data/lib/duel.rb +413 -0
  40. data/sig/duel/across.rbs +21 -0
  41. data/sig/duel/baseline.rbs +61 -0
  42. data/sig/duel/cli.rbs +95 -0
  43. data/sig/duel/config.rbs +181 -0
  44. data/sig/duel/context.rbs +387 -0
  45. data/sig/duel/init.rbs +97 -0
  46. data/sig/duel/json_report.rbs +47 -0
  47. data/sig/duel/lock.rbs +12 -0
  48. data/sig/duel/metric.rbs +351 -0
  49. data/sig/duel/native.rbs +98 -0
  50. data/sig/duel/process_sample.rbs +54 -0
  51. data/sig/duel/profile.rbs +64 -0
  52. data/sig/duel/rake_task.rbs +9 -0
  53. data/sig/duel/resource.rbs +106 -0
  54. data/sig/duel/rubies.rbs +45 -0
  55. data/sig/duel/runner.rbs +530 -0
  56. data/sig/duel/stats.rbs +26 -0
  57. data/sig/duel/summary.rbs +87 -0
  58. data/sig/duel/testing.rbs +91 -0
  59. data/sig/duel/version.rbs +3 -0
  60. data/sig/duel/worker.rbs +74 -0
  61. data/sig/duel.rbs +276 -0
  62. data/sig/manifest.yaml +4 -0
  63. metadata +124 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 24277600f238129c16a52fb4115ccb6a6bd340602e7a38187622bfe1550adc1d
4
+ data.tar.gz: 489836bc05d935ef59e369149df8ec47e06279005691fd084ea9b587c489f175
5
+ SHA512:
6
+ metadata.gz: 2ce8583f3c82f785cd25b95339c9fd7a751e42b7221d0a8b9af793351560945b8ca4d4001c7725db93c3a4d77506baf3cf5de37b5decd4c055f1ca17313de997
7
+ data.tar.gz: 81e190402ee6b2eb3705b41fece6ee45e12bc3a6f71645ac1467d7bbe5ed015195d9291fb4b651888b69b759fa4f109f88029e7b7c0272260c2326af11692b65
data/AGENT_PROMPT.md ADDED
@@ -0,0 +1,211 @@
1
+ # Adopting Duel: a prompt for a coding agent
2
+
3
+ Give this to a coding agent (paste it, or point it here), in the project
4
+ you want to guard.
5
+
6
+ ---
7
+
8
+ Add performance regression guards to this project with Duel
9
+ (https://github.com/sferik/duel-ruby), so a pull request that makes the
10
+ code slower fails CI.
11
+
12
+ First read Duel's README, and the guide in its `docs/` directory, in
13
+ full. After installing, find them with `bundle info duel --path`. Follow its conventions exactly, and don't guess
14
+ at the API.
15
+
16
+ 1. **Install and set up.** Add `gem "duel"` to the Gemfile's
17
+ development group (unless the Gemfile has it already), and run
18
+ `bundle install`. Then, unless the project is a Rails app (see step
19
+ 4), run `bundle exec duel init`, and read what it wrote: `.duel.rb`,
20
+ an example benchmark in `duel/`, the GitHub Actions workflow
21
+ (`.github/workflows/perf.yml`), Duel's Rake task, if there's a
22
+ Rakefile, and `gem "stackprof", platforms: :mri` in the Gemfile, for
23
+ profiles of regressions. It lists what it did and what's next: run
24
+ `bundle install` again for StackProf.
25
+
26
+ 2. **Find what's worth guarding.** Read the code and its tests to find
27
+ the work users depend on being fast:
28
+ - The main public entry points.
29
+ - Anything called in a loop or once per request.
30
+ - Parsers, serializers and encoders.
31
+ - Anything the docs or changelog make performance claims about.
32
+ - Any existing benchmarks (`benchmark/`, `bench/`, benchmark-ips
33
+ scripts).
34
+
35
+ Prefer focused benchmarks over broad ones: 10–30 for a sizable
36
+ library, fewer for a small one with few hot paths. Cover typical
37
+ inputs and the small and large extremes, as separate benchmarks.
38
+ Before writing any, list what you chose and why.
39
+
40
+ 3. **Write the benchmarks** in `duel/*_duel.rb`, in place of the
41
+ example that `duel init` wrote. Shared helpers go in
42
+ `duel/duel_helper.rb`, and fixtures in `duel/fixtures/`.
43
+ - Name each one "area: what it does", e.g. `"parse: a large document"`.
44
+ `--filter` takes a regular expression and matches anywhere in a
45
+ name, so `^parse:` picks out an area. Keep names under about 50
46
+ characters, or they push the report's columns out of line.
47
+ - Time only the work itself. Building inputs, reading fixtures and
48
+ connecting go in `once { }`. For per-batch work, take `|n|` and wrap
49
+ the per-batch preparation in `untimed`.
50
+ - Make every iteration identical. Use fixed inputs, seed any
51
+ randomness (`Random.new(42)`), and warm caches and memoization in
52
+ `once`. Work that writes (records, files, a queue) changes what
53
+ the next iteration sees: undo each iteration with an
54
+ `around_iteration` hook (rolling back a transaction, say), and put
55
+ back what a sample changed with a `before_sample` hook. Duel fails
56
+ on even one extra allocation per two iterations, so anything that
57
+ varies from run to run is a false alarm waiting to happen. That
58
+ includes what differs between the two versions' workers, like a
59
+ server's socket path or a worker's directory: don't time work on
60
+ it.
61
+ - Use the sizes the code really sees. Read it for limits and
62
+ constants (a cap on how many items it ever handles, say) rather
63
+ than guessing.
64
+ - Time one iteration before writing a benchmark around it. Aim for
65
+ microseconds to a few milliseconds per iteration, and split
66
+ anything slower: a size the code never sees can take seconds, and
67
+ an iteration that takes over a minute (`--timeout`) is an error.
68
+ - Use only API the main branch already has. A benchmark the baseline
69
+ can't run is skipped, so it guards nothing until it's merged.
70
+ - Use `only: :count` for work that's mostly waiting, including
71
+ anything that goes through a socket or makes system calls, even on
72
+ the same machine: its timings are too noisy to settle. Use
73
+ `only: :timing` for work done on threads of its own, and
74
+ `peak_memory: true` where memory use matters.
75
+ - Prefer one thread. Switching between threads costs microseconds,
76
+ which swamps a small cost and leaves its timings unsettled: guard
77
+ threaded code with a single thread exercising it where you can.
78
+ - Don't change anything under `lib/` or `ext/`.
79
+
80
+ 4. **Configure only if needed.** `duel init` wrote a `.duel.rb` for
81
+ what it could tell of the project: a gemspec, `ext/`, and a
82
+ `.ruby-version`. A conventional gem needs nothing more. If the
83
+ project isn't one (a non-standard layout, a build step other than
84
+ `rake compile`), change it, using only the settings that Duel's
85
+ `docs/configuration.md` documents.
86
+
87
+ If the code talks to Redis, PostgreSQL, MySQL or memcached, or uses
88
+ Active Record, threads, or long-lived objects, use the matching
89
+ plugin (duel-redis, duel-postgresql, duel-mysql, duel-memcached,
90
+ duel-activerecord, duel-gvl, duel-retained): add its gem, and
91
+ `c.use` it by name in `.duel.rb` (`c.use :redis`). Don't point
92
+ benchmarks at a shared server.
93
+
94
+ If the project is a Rails app, use duel-rails instead of `duel init`
95
+ (which says so) or setting Duel up by hand: add it to the Gemfile's
96
+ development group, run `bin/rails generate duel:install`, read what
97
+ it made, and run `bundle install` again for what it added to the
98
+ Gemfile (StackProf, and a database server's plugin). Then:
99
+ - Benchmark the requests that matter most with `duel_request`, not
100
+ only the GET routes the generator found: the pages users load
101
+ most, and the forms and API calls that write. Give each request
102
+ what it sends (`params:`, `json:`, `headers:`), and sign in with a
103
+ block where the request needs a user.
104
+ - Give the requests data to work with: `fixtures: :all` (or the
105
+ fixtures the requests need), or `seeds: true`, in `.duel.rb`. A
106
+ page with no records to show guards almost nothing.
107
+ - Benchmark the app's slowest models and services with plain `duel`
108
+ blocks too, with their data made in `once`. Roll back those that
109
+ write with `around_iteration: Duel::Rails::ROLLBACK`.
110
+ - Check `config/environments/benchmark.rb`: stub anything a request
111
+ would call over the network.
112
+
113
+ If the project is itself a Duel plugin, don't require or use it in
114
+ its own `.duel.rb`. That would load the candidate's plugin into
115
+ Duel's own workers, under the version being measured. Start any
116
+ server its benchmarks need with `c.prepare(:worker)` directly, with
117
+ every option the plugin itself starts it with: one left out can
118
+ change what the server does (and what the plugin reads of it).
119
+
120
+ If the project's boot time or load time matters (an app, or a library
121
+ that's slow to require), add a `duel_process` benchmark of it, such as
122
+ `duel_process "boot: the app", "bin/rails", "runner", "1"`.
123
+
124
+ If the project pins its Ruby (`.ruby-version`, `.tool-versions`, or
125
+ `mise.toml`), set `c.ruby = :version_file`, so that a branch that
126
+ upgrades Ruby is compared on each version's own Ruby. `duel init`
127
+ does this, and adds the steps in Duel's `docs/ci.md` that install
128
+ the baseline's Ruby to the workflow, for a `.ruby-version`. For another version
129
+ file, do both yourself (see step 6). Duel uses the Ruby that runs it
130
+ when that's the one asked for, so CI needs nothing more until a
131
+ branch changes the Ruby.
132
+
133
+ If the project is in a subdirectory of its repository (a monorepo),
134
+ put `.duel.rb` and `duel/` in that subdirectory, and run `duel` from
135
+ there. Run `duel init` there too: it puts the workflow at the
136
+ repository's root, and runs it in the subdirectory.
137
+
138
+ Make sure the linter checks `.duel.rb`. It's a dotfile, so RuboCop
139
+ leaves it out unless its config includes it (`AllCops: Include`,
140
+ merged with the defaults by `inherit_mode`).
141
+
142
+ 5. **Check that the guards hold.** While you're still writing
143
+ benchmarks, run a few at a time (`bundle exec duel --force --filter
144
+ <name>`): a whole run waits on its slowest benchmark, and shows
145
+ nothing of one until it's measured. Duel runs one comparison at a
146
+ time in a project: a second run fails, naming the first one's pid.
147
+
148
+ If the project has a C extension, Duel checks that each version's
149
+ workers load their own build, and fails, saying to set `c.build`, if
150
+ one doesn't (a baseline with no build of its own would load the
151
+ candidate's).
152
+
153
+ Then do all three:
154
+ - Run `bundle exec duel --force`. With `lib/` unchanged, it compares
155
+ the code with itself and must pass. Run it three times: a noisy
156
+ benchmark can pass twice by luck. A benchmark that fails or
157
+ errors is broken. One reported unsettled in more than one run is
158
+ noisy: fix it or drop it. (Across many benchmarks, one unsettled
159
+ once is normal: it doesn't fail the run.) Benchmarks that run a
160
+ lot of other code (a framework's whole query, say) to exercise a
161
+ little of the project's are the noisiest: guard the project's part
162
+ directly instead.
163
+ - Make a deliberate regression in code that a few benchmarks
164
+ exercise: an extra allocation in a hot method, then some extra
165
+ work in a loop (in a C extension, the extra work). Confirm that
166
+ `bundle exec duel` exits 1 and names those benchmarks. Then revert
167
+ the change completely, check that `git diff lib/ ext/` is empty,
168
+ and rebuild the extension if you changed one.
169
+ - Run `bundle exec duel --force --filter <pattern>` for a single
170
+ benchmark, and for one area (`--filter "^parse:"`), to confirm
171
+ your names can be picked out. Without `--force`, an unchanged
172
+ `lib/` has nothing to compare.
173
+
174
+ 6. **Check CI.** `duel init` wrote Duel's GitHub Actions
175
+ workflow (`.github/workflows/perf.yml`). Keep `fetch-depth: 0`:
176
+ Duel's baseline comes from git history. Keep its check for a push
177
+ with no commit before it in the history (a branch's first push, or a
178
+ force push), which would otherwise fail the job. Keep its last step,
179
+ which comments on a pull request with Duel's report (Duel also adds
180
+ the report to the job's summary). Check that `ruby-version` is one
181
+ the project supports, unless it has a `.ruby-version`. With
182
+ `c.ruby = :version_file` and another version file than
183
+ `.ruby-version`, add the steps in Duel's `docs/ci.md` that install the
184
+ baseline's Ruby for a pull request. Add the project's other CI setup that its
185
+ tests need (system packages, services) to the workflow as well. Keep
186
+ the scheduled `last-release` run only if the project tags its
187
+ releases (`v*` by default, or else set `c.release_tags`), or leave
188
+ the schedule out. Until the first release is tagged, that run says
189
+ there's nothing to compare against, and passes. Once it's pushed,
190
+ run the workflow by hand: it compares the code with itself there,
191
+ which shows that the benchmarks run on CI's machines (with their
192
+ packages and services).
193
+
194
+ 7. **Show the badge.** If the project's README already shows badges (CI
195
+ status, gem version), add Duel's to that row, with exactly this
196
+ markdown:
197
+
198
+ ```markdown
199
+ [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
200
+ ```
201
+
202
+ If the README has no badges, don't add a row just for Duel's.
203
+
204
+ 8. **Report back** with:
205
+ - The benchmarks you wrote and what each one guards.
206
+ - What you chose not to benchmark, and why.
207
+ - The output of the self-comparison and of the deliberate-regression
208
+ check.
209
+ - Anything you couldn't guard reliably.
210
+
211
+ Don't commit. Leave the changes for me to review.
data/CHANGELOG.md ADDED
@@ -0,0 +1,281 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.1.0] - 2026-10-24
11
+
12
+ The first release. Duel is open source under the MIT license, and
13
+ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
14
+
15
+ ### Added
16
+
17
+ - `duel` and `Duel::RakeTask`: run every benchmark on a baseline (where
18
+ HEAD left the main branch, a git ref, the last release tag, or a
19
+ directory) and the candidate, side by side in worker processes of their
20
+ own, and fail if the candidate is slower at any of them.
21
+ - `duel init`: sets Duel up in a gem or an app that isn't a Rails app.
22
+ It writes `.duel.rb` (with `c.build` for `ext/`, and
23
+ `c.ruby = :version_file` for a `.ruby-version`), an example benchmark
24
+ with advice on writing good ones, and the GitHub Actions workflow in
25
+ docs/ci.md, adds Duel's Rake task to the Rakefile, and adds
26
+ `gem "stackprof", platforms: :mri` to the Gemfile, for profiles of
27
+ regressions (in its `development` group, if it has one, and not if it
28
+ mentions StackProf already). It overwrites nothing without `--force`,
29
+ and writes nothing with `--dry-run`. The workflow compares pull
30
+ requests, pushes, and, nightly, the last release, and comments on each
31
+ pull request with Duel's report, updating one comment on every push.
32
+ - A report in Markdown, for a pull request. On GitHub Actions
33
+ (`GITHUB_STEP_SUMMARY`), Duel adds it to the job's summary, and
34
+ `--markdown PATH` writes it to a file, to post as a comment. It has the
35
+ verdict, the two versions (and their Rubies, where they differ), a
36
+ table of the benchmarks that changed, regressions first, with each
37
+ timing's ratio and interval and the counts that changed, each
38
+ regression explained as the terminal report explains it, and the
39
+ benchmarks that were unsettled or failed. Those that didn't change are
40
+ collapsed. A run with nothing to compare, or that couldn't compare,
41
+ says so in Markdown too.
42
+ - `--json PATH`: writes the report as JSON, for a program to read, with
43
+ a versioned schema (`"schema": 1`, documented in docs/json.md). It
44
+ has the run's status and exit status, its settings, the two versions
45
+ (each one's ref, commit, tree and Ruby), the plugins in use, and every
46
+ benchmark: its verdict, each timing's ratio, interval, limit, pairs,
47
+ verdict and each version's median an iteration, each count on both
48
+ versions an iteration, peak memory, what metrics said, each
49
+ regression's explanation, and why it failed or was skipped. Then the
50
+ verdicts across the board, and the errors. It's written whatever the
51
+ outcome, nothing to compare and couldn't compare included, but not for
52
+ a command line that isn't valid.
53
+ - Timings (CPU time, and any timed metric of a plugin's) are judged by a
54
+ 99% distribution-free confidence interval for the median of paired
55
+ ratios, and whatever looks slower is measured again in fresh processes
56
+ before it fails. A measurement whose timings are all too close to
57
+ call neither confirms a slowdown nor clears it, so Duel measures once
58
+ more in its place, up to three more times. A benchmark that's slower
59
+ in time in every measurement fails even when no one timing is slower
60
+ in all of them (CPU time in one, instructions in the next), as long as
61
+ none says the same or faster, after one measurement more.
62
+ Allocations and counted metrics fail on any increase.
63
+ - Each version runs in several worker processes (three by default,
64
+ `--processes`), paired in turn, so no one process's layout in memory
65
+ biases a comparison.
66
+ - A git baseline is checked out next to the project, at a path as long
67
+ and as deep as the project's, so work a version does on its own paths
68
+ doesn't make the baseline measure slower. Duel says so when the two
69
+ versions' paths still differ in length. A project in a subdirectory
70
+ of its repository (a monorepo's) is compared with the same
71
+ subdirectory of a checkout of the whole repository. Without a merge
72
+ base with the main branch (unrelated histories, or a shallow clone),
73
+ or in a repository without commits, Duel says why.
74
+ - The candidate is built (`c.build`) before every run, as a baseline is,
75
+ so a stale or missing build never decides a comparison. Duel checks
76
+ that each version's workers load only native extensions built in that
77
+ version's tree, and fails, saying to set `c.build`, when one loads
78
+ another version's build or an installed gem's. Gems installed in the
79
+ project's directory (`vendor/bundle`, say) are installed gems, not
80
+ builds.
81
+ - One run at a time in a project: a second `duel` fails, naming the
82
+ first one's pid. The lock is an flock, so a run that crashed leaves
83
+ nothing behind.
84
+ - With YJIT on in a worker (an app, Rails's say, may turn it on as it
85
+ boots), each benchmark warms up for two seconds before it's measured,
86
+ unless `--warm` says otherwise.
87
+ - Instruction counts from the CPU's counters (the benchmark thread's on
88
+ Linux, and the process's on macOS), judged with a limit of their own
89
+ (1%).
90
+ - Peak memory, for benchmarks that ask (`peak_memory: true`): each
91
+ version's least in fresh workers.
92
+ - A StackProf profile of both versions with every regression in time,
93
+ and where the extra CPU went, on CRuby with StackProf in the bundle.
94
+ StackProf is optional, and Duel doesn't use it on JRuby or
95
+ TruffleRuby, even where it's installed. Profiles cover what Duel measures: they
96
+ leave out work inside `untimed` and an `around_iteration` hook's own
97
+ work, keep what `after_iterations` does, hold the garbage collector
98
+ off as samples do, and show what garbage collection gained on a line
99
+ of its own, not ranked with the code. On macOS, they sample elapsed
100
+ time, because CPU-time samples there pile up on code that makes
101
+ system calls.
102
+ - Conventions: benchmarks in `duel/**/*_duel.rb`, the gemspec's library
103
+ required from each version, `rake compile` for a baseline of a
104
+ project with `ext/`, and the main branch from `origin/HEAD` (or
105
+ `origin/main`, `origin/master`, and without a remote, `main` or
106
+ `master`).
107
+ - Across the board: every benchmark's median ratio, judged together, so
108
+ a slowdown too small for any one benchmark still fails. It takes 8
109
+ benchmarks, so with fewer, the report leaves it out.
110
+ - A worker that doesn't load or answer within `--timeout` (60s) is
111
+ killed, and its benchmark is an error. Samples are held to 64 MB of
112
+ allocations while the garbage collector is held off. `duel --clean`
113
+ removes what runs leave behind.
114
+ - `duel "name" do ... end`, in files named `*_duel.rb`, defines a
115
+ benchmark: its block is timed, run as many times as a sample needs
116
+ (or, given `|n|`, once a sample for n iterations). `once { }` in it
117
+ runs its block once, untimed, before anything's measured, and
118
+ `untimed { }` pauses measuring. Duel's own work in an iteration allocates nothing,
119
+ so a benchmark that allocates nothing counts no allocations, and an
120
+ `untimed` block adds none. `only: :count` (work that's mostly waiting)
121
+ or `only: :timing` (work on threads of its own) narrows what it's
122
+ judged on, and `peak_memory: true` adds its peak memory. `.duel.rb`
123
+ covers what the conventions don't: building, loading, preparing each
124
+ worker, and metrics of your own, which together make a plugin (a
125
+ `duel-*` gem that `c.use` uses by name). Nothing else is built in.
126
+ - Hooks, with one set of names at every level: in `.duel.rb`
127
+ (`c.setup { }`), at the top of a file of benchmarks (`setup { }`), and
128
+ as an option of one benchmark (`duel "x", setup: -> { }`). They're
129
+ `setup` and `teardown` (once in each worker, around a benchmark's
130
+ samples), `before_sample` (before every sample, not measured, so every
131
+ sample starts alike), `around_iteration` (around each iteration, which
132
+ it's given to run, and which alone is measured), and
133
+ `after_iterations` (after each sample's iterations, measured with
134
+ them: work that they left for later, such as jobs that they
135
+ enqueued). `.duel.rb`'s run first, then the file's, then the
136
+ benchmark's own (teardown's in reverse, and around_iteration's nested,
137
+ outermost first). A file's and a benchmark's own run in the
138
+ benchmark's context. What pausing and resuming the metrics around
139
+ each iteration leaves is learned in each worker and taken away, and
140
+ pausing allocates nothing.
141
+ - Helpers that files of benchmarks share, in `duel/duel_helper.rb`. A
142
+ helper named as one of the methods that Duel gives benchmarks
143
+ (`once`, `untimed`, a hook, or a DSL method) fails the worker, naming
144
+ its file and line.
145
+ - `duel_process "name", command` (or `ruby: code`): a benchmark of a
146
+ whole process, booting an app or requiring a library, say. Each sample
147
+ is the fastest of a few fresh processes for each version (as many as
148
+ take about a second of CPU time, up to 3, or `runs:`), the versions'
149
+ processes in turn, in its tree and with its environment, judged on its
150
+ CPU time (its children's too) and its peak memory (with a CRuby
151
+ runner: each version's lowest). The processes are started and
152
+ measured by a small Ruby process of Duel's, one for each worker, that
153
+ lasts as long as the worker, so its start-up is paid once and not for
154
+ every process. It's judged after every pair of samples,
155
+ and stops as soon as it's settled that it isn't slower, but it's found
156
+ slower only after a round of 10 pairs, so stopping early can't make
157
+ identical code look slower. It runs in no worker, so it takes none of
158
+ `duel`'s options (a hook such as `around_iteration:`, `only:`, or
159
+ `peak_memory:` raises an `ArgumentError`), and runs no hooks.
160
+ - `c.prepare(:worker) { |w| ... }` prepares each worker with a
161
+ `Duel::Resource`: a directory of the worker's own, `spawn` (logged
162
+ there, and stopped with the worker, by its `stop_signal:`), `run`,
163
+ `wait_until` (failing if a process exits, or after `--timeout`),
164
+ `failed`, `at_stop`, the version's `root` and `ruby`, and `env`, the
165
+ version's environment, which the commands that it starts get under
166
+ their own. The block returns the worker's environment, and two
167
+ prepare blocks that set a variable differently fail.
168
+ - `c.prepare(:version) { |v| ... }`: prepares each version once for a
169
+ run, in the runner, before its workers start (installing its bundle,
170
+ say). It gets a `Duel::Resource` for the version and returns the
171
+ environment that the version's workers and process benchmarks get.
172
+ - `c.ruby` and `duel --ruby`: each version's Ruby. `:version_file` (or
173
+ `--ruby version-file`) runs each version on the Ruby that its own
174
+ `.ruby-version`, `.tool-versions`, or `mise.toml` names, found with
175
+ rbenv, chruby, mise, or asdf. A path, or a block given the version's
176
+ tree, says which otherwise. A version file that asks for the Ruby that
177
+ runs `duel` (as CI's setup step installs it, where no manager looks)
178
+ gets that Ruby. The report shows each version's `RUBY_DESCRIPTION`
179
+ when they differ, and a Ruby that isn't installed fails the run.
180
+ - `c.in_worker { |w| }`: set-up that runs only in workers, once the
181
+ benchmarks have loaded. What it gives `w.at_stop` runs when the
182
+ worker ends. It returns what the block made in each (a connection,
183
+ say), as `value`, the worker's.
184
+ - `w.cached(key) { |dir| }`: what a prepare block builds once, for every
185
+ worker of a run and later runs (a database's initial files, say).
186
+ - A run's workers start at once, each prepared, and stop at once.
187
+ - Metrics of running totals: `c.metric("name") { total }`, or several
188
+ by their settings, and a sample measures what they grew by, less what
189
+ reading a counted one adds to it (learned in each worker, unless
190
+ `calibrate: false`). A `Duel::Metric` subclass covers the rest: its
191
+ `measures` name what it measures, with their settings, its
192
+ `start_sample` gets the sample (`n`, `benchmark`), and it can
193
+ `pause_sample` and `resume_sample` around `untimed` blocks its own
194
+ way.
195
+ - `open { }` and `close { }` on what `c.metric` returns, and `open` and
196
+ `close` methods of a `Duel::Metric` subclass: a metric's own setup and
197
+ teardown in each worker. `details`, `open`, `close`, and `costly`
198
+ return the metric, to chain them.
199
+ - `costly` on what `c.metric` returns, and `self.costly?` on a
200
+ `Duel::Metric` subclass: a metric that's costly to read (a round trip
201
+ to a server, say) isn't paused around each iteration inside an
202
+ `around_iteration` hook, but read only as each sample starts and
203
+ stops. What the hook adds to it is learned in each worker, from three
204
+ samples of 10 iterations in which it's paused around each iteration,
205
+ and subtracted, so the hook's own work (a transaction's `BEGIN` and
206
+ `ROLLBACK`, say) still isn't counted, and the metric measures what
207
+ pausing would have. The plugins that read a server (duel-postgresql,
208
+ duel-mysql, duel-redis and duel-memcached) say so.
209
+ - Settings for metrics: `floor` (on either version), `limit`, `unit_ns`,
210
+ `sizes_samples`, `tolerance` (per iteration) and `allowance` (per
211
+ sample). `c.measure(name, ...)` changes any measure's, Duel's own
212
+ too, or skips it. Two metrics that measure the same name fail.
213
+ - A regression is explained an iteration, on each version
214
+ (`cpu: 112.4µs → 121.5µs an iteration`), with what its metric says of
215
+ it (`Metric#details`), and results carry each version's raw medians.
216
+ Each benchmark's line shows its counts an iteration too.
217
+ - `unit:` for a measure (a time unit shows it as time, and gives
218
+ `unit_ns`). Settings there aren't, and kinds there aren't, fail.
219
+ `Metric#settings` has a metric's settings after `c.measure`'s changes,
220
+ every one filled in. Running totals' `details { |stretches, sample| }`
221
+ explains them, and their block can return `[totals, context]` to pass
222
+ the context on (typed, with Steep). `Reading#total` reads one total.
223
+ Units show with regressions, and on the summary line unless the
224
+ measure's name ends in them. `Duel::Testing.measure` returns what
225
+ metrics said (`#details`) with what they measured.
226
+ - `kind: :ratio` (judged by paired ratios, as CPU time is) and
227
+ `:counted`, with `cpu_floor:` (a floor that's a share of CPU time) and
228
+ `across: false` (not judged across the board).
229
+ - `c.use :redis`: uses a plugin by name. It requires `duel/redis` and
230
+ calls the `configure` method of the module that the file registers
231
+ with `Duel.plugin(:redis, self)`, given the configuration and the
232
+ options. A plugin is configured once, however many times it's used, so
233
+ a plugin that needs another (duel-rails, duel-activerecord) uses it
234
+ too, and an app's own use of it, with options, comes first. A plugin
235
+ that isn't in the bundle fails, naming the gem to add. The report, in
236
+ the terminal and in Markdown, lists the plugins used. Workers load
237
+ Duel and every Duel plugin that the runner loaded (the `duel` and
238
+ `duel-*` gems) from where the runner did, ahead of the version's own
239
+ bundle, so a version whose bundle lacks them (or has other versions of
240
+ them) is measured with the same.
241
+ - A plugin's own DSL methods: those of its module's `DSL` module, which
242
+ files of benchmarks get once `.duel.rb` uses the plugin. A name that's
243
+ taken, by the DSL, another plugin, or every object, raises
244
+ `Duel::Error` instead of replacing it. A benchmark that the method
245
+ defines, and a hook that it adds, belong to the file of benchmarks
246
+ that called it, which gets that file's hooks.
247
+ - `duel/testing`, for plugins' tests: `Duel::Testing.prepare` (which
248
+ runs the configuration's `c.prepare(:version)` blocks, then its
249
+ `c.prepare(:worker)` blocks, as a run does), `measure`, `in_worker`
250
+ (which ends as a worker ends, closing its metrics), `metrics`,
251
+ `compare` and `project`. `measure(config, **hooks)` measures the
252
+ block as a benchmark with the configuration's hooks and its own, as
253
+ `duel` takes them. With an `around_iteration` hook, the block is one
254
+ iteration, run `n` times, each inside the hook, costly metrics and
255
+ all. The block gets the benchmark's context, with `untimed`.
256
+ - `Duel::Testing.sample`, `reading` and `details`: stand-ins for a
257
+ sample and for readings of running totals, to test or benchmark what a
258
+ metric says of a sample. docs/plugins.md documents what `details` is
259
+ given and when it runs.
260
+ - `Metric#setting(:allowance)`, for a metric of one measure, and
261
+ `Metric#within?(more, n)` (and `Metric.within?`): whether a count is
262
+ within its tolerance and allowance, as Duel judges it.
263
+ - A unit that's a word comes with its plural (`unit: %w[row rows]`: 1
264
+ row, 2 rows), and a symbol (`"B"`) is one String.
265
+ - `Duel.measuring?`: whether a benchmark is being measured right now
266
+ (not in setup or `untimed`), and `Duel.unmeasured!(name)`, for a
267
+ plugin's helper to fail where what it does would be counted.
268
+ - Running totals are typed from their block (Integer totals stay
269
+ Integer), and `Duel::Testing.measure` ends the worker it starts.
270
+ - `DUEL_BASE`, `DUEL_FILTER` and `DUEL_FORCE` set `--base`, `--filter`
271
+ and `--force` where options are hard to pass (behind a rake task, or
272
+ in a CI expression). An invalid regular expression for `--filter` or
273
+ `DUEL_FILTER` is a usage error, with exit status 2.
274
+ - Mutation tested: mutant kills every change to Duel's code, in CI.
275
+ - RBS signatures in sig/, checked with Steep.
276
+ - The public API is what the README and the guide in docs/ document.
277
+ Anything marked `@api private`, in lib/ and sig/, may change in any
278
+ release.
279
+
280
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.0...HEAD
281
+ [0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Erik Berlin
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.