duel 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/AGENT_PROMPT.md +211 -0
- data/CHANGELOG.md +281 -0
- data/LICENSE +21 -0
- data/README.md +354 -0
- data/docs/README.md +23 -0
- data/docs/benchmarks.md +289 -0
- data/docs/ci.md +141 -0
- data/docs/configuration.md +429 -0
- data/docs/how-it-works.md +207 -0
- data/docs/json.md +306 -0
- data/docs/plugins.md +408 -0
- data/docs/troubleshooting.md +105 -0
- data/exe/duel +6 -0
- data/lib/duel/across.rb +48 -0
- data/lib/duel/baseline.rb +166 -0
- data/lib/duel/cli.rb +264 -0
- data/lib/duel/config.rb +343 -0
- data/lib/duel/context.rb +630 -0
- data/lib/duel/init.rb +211 -0
- data/lib/duel/json_report.rb +107 -0
- data/lib/duel/lock.rb +38 -0
- data/lib/duel/metric.rb +496 -0
- data/lib/duel/native.rb +162 -0
- data/lib/duel/process_sample.rb +144 -0
- data/lib/duel/profile.rb +116 -0
- data/lib/duel/rake_task.rb +25 -0
- data/lib/duel/resource.rb +209 -0
- data/lib/duel/rubies.rb +149 -0
- data/lib/duel/runner.rb +1114 -0
- data/lib/duel/stats.rb +73 -0
- data/lib/duel/summary.rb +237 -0
- data/lib/duel/templates/benchmark_duel.rb.erb +34 -0
- data/lib/duel/templates/duel.rb.erb +35 -0
- data/lib/duel/templates/perf.yml.erb +97 -0
- data/lib/duel/testing.rb +241 -0
- data/lib/duel/version.rb +5 -0
- data/lib/duel/worker.rb +211 -0
- data/lib/duel.rb +413 -0
- data/sig/duel/across.rbs +21 -0
- data/sig/duel/baseline.rbs +61 -0
- data/sig/duel/cli.rbs +95 -0
- data/sig/duel/config.rbs +181 -0
- data/sig/duel/context.rbs +387 -0
- data/sig/duel/init.rbs +97 -0
- data/sig/duel/json_report.rbs +47 -0
- data/sig/duel/lock.rbs +12 -0
- data/sig/duel/metric.rbs +351 -0
- data/sig/duel/native.rbs +98 -0
- data/sig/duel/process_sample.rbs +54 -0
- data/sig/duel/profile.rbs +64 -0
- data/sig/duel/rake_task.rbs +9 -0
- data/sig/duel/resource.rbs +106 -0
- data/sig/duel/rubies.rbs +45 -0
- data/sig/duel/runner.rbs +530 -0
- data/sig/duel/stats.rbs +26 -0
- data/sig/duel/summary.rbs +87 -0
- data/sig/duel/testing.rbs +91 -0
- data/sig/duel/version.rbs +3 -0
- data/sig/duel/worker.rbs +74 -0
- data/sig/duel.rbs +276 -0
- data/sig/manifest.yaml +4 -0
- metadata +124 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 24277600f238129c16a52fb4115ccb6a6bd340602e7a38187622bfe1550adc1d
|
|
4
|
+
data.tar.gz: 489836bc05d935ef59e369149df8ec47e06279005691fd084ea9b587c489f175
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 2ce8583f3c82f785cd25b95339c9fd7a751e42b7221d0a8b9af793351560945b8ca4d4001c7725db93c3a4d77506baf3cf5de37b5decd4c055f1ca17313de997
|
|
7
|
+
data.tar.gz: 81e190402ee6b2eb3705b41fece6ee45e12bc3a6f71645ac1467d7bbe5ed015195d9291fb4b651888b69b759fa4f109f88029e7b7c0272260c2326af11692b65
|
data/AGENT_PROMPT.md
ADDED
|
@@ -0,0 +1,211 @@
|
|
|
1
|
+
# Adopting Duel: a prompt for a coding agent
|
|
2
|
+
|
|
3
|
+
Give this to a coding agent (paste it, or point it here), in the project
|
|
4
|
+
you want to guard.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Add performance regression guards to this project with Duel
|
|
9
|
+
(https://github.com/sferik/duel-ruby), so a pull request that makes the
|
|
10
|
+
code slower fails CI.
|
|
11
|
+
|
|
12
|
+
First read Duel's README, and the guide in its `docs/` directory, in
|
|
13
|
+
full. After installing, find them with `bundle info duel --path`. Follow its conventions exactly, and don't guess
|
|
14
|
+
at the API.
|
|
15
|
+
|
|
16
|
+
1. **Install and set up.** Add `gem "duel"` to the Gemfile's
|
|
17
|
+
development group (unless the Gemfile has it already), and run
|
|
18
|
+
`bundle install`. Then, unless the project is a Rails app (see step
|
|
19
|
+
4), run `bundle exec duel init`, and read what it wrote: `.duel.rb`,
|
|
20
|
+
an example benchmark in `duel/`, the GitHub Actions workflow
|
|
21
|
+
(`.github/workflows/perf.yml`), Duel's Rake task, if there's a
|
|
22
|
+
Rakefile, and `gem "stackprof", platforms: :mri` in the Gemfile, for
|
|
23
|
+
profiles of regressions. It lists what it did and what's next: run
|
|
24
|
+
`bundle install` again for StackProf.
|
|
25
|
+
|
|
26
|
+
2. **Find what's worth guarding.** Read the code and its tests to find
|
|
27
|
+
the work users depend on being fast:
|
|
28
|
+
- The main public entry points.
|
|
29
|
+
- Anything called in a loop or once per request.
|
|
30
|
+
- Parsers, serializers and encoders.
|
|
31
|
+
- Anything the docs or changelog make performance claims about.
|
|
32
|
+
- Any existing benchmarks (`benchmark/`, `bench/`, benchmark-ips
|
|
33
|
+
scripts).
|
|
34
|
+
|
|
35
|
+
Prefer focused benchmarks over broad ones: 10–30 for a sizable
|
|
36
|
+
library, fewer for a small one with few hot paths. Cover typical
|
|
37
|
+
inputs and the small and large extremes, as separate benchmarks.
|
|
38
|
+
Before writing any, list what you chose and why.
|
|
39
|
+
|
|
40
|
+
3. **Write the benchmarks** in `duel/*_duel.rb`, in place of the
|
|
41
|
+
example that `duel init` wrote. Shared helpers go in
|
|
42
|
+
`duel/duel_helper.rb`, and fixtures in `duel/fixtures/`.
|
|
43
|
+
- Name each one "area: what it does", e.g. `"parse: a large document"`.
|
|
44
|
+
`--filter` takes a regular expression and matches anywhere in a
|
|
45
|
+
name, so `^parse:` picks out an area. Keep names under about 50
|
|
46
|
+
characters, or they push the report's columns out of line.
|
|
47
|
+
- Time only the work itself. Building inputs, reading fixtures and
|
|
48
|
+
connecting go in `once { }`. For per-batch work, take `|n|` and wrap
|
|
49
|
+
the per-batch preparation in `untimed`.
|
|
50
|
+
- Make every iteration identical. Use fixed inputs, seed any
|
|
51
|
+
randomness (`Random.new(42)`), and warm caches and memoization in
|
|
52
|
+
`once`. Work that writes (records, files, a queue) changes what
|
|
53
|
+
the next iteration sees: undo each iteration with an
|
|
54
|
+
`around_iteration` hook (rolling back a transaction, say), and put
|
|
55
|
+
back what a sample changed with a `before_sample` hook. Duel fails
|
|
56
|
+
on even one extra allocation per two iterations, so anything that
|
|
57
|
+
varies from run to run is a false alarm waiting to happen. That
|
|
58
|
+
includes what differs between the two versions' workers, like a
|
|
59
|
+
server's socket path or a worker's directory: don't time work on
|
|
60
|
+
it.
|
|
61
|
+
- Use the sizes the code really sees. Read it for limits and
|
|
62
|
+
constants (a cap on how many items it ever handles, say) rather
|
|
63
|
+
than guessing.
|
|
64
|
+
- Time one iteration before writing a benchmark around it. Aim for
|
|
65
|
+
microseconds to a few milliseconds per iteration, and split
|
|
66
|
+
anything slower: a size the code never sees can take seconds, and
|
|
67
|
+
an iteration that takes over a minute (`--timeout`) is an error.
|
|
68
|
+
- Use only API the main branch already has. A benchmark the baseline
|
|
69
|
+
can't run is skipped, so it guards nothing until it's merged.
|
|
70
|
+
- Use `only: :count` for work that's mostly waiting, including
|
|
71
|
+
anything that goes through a socket or makes system calls, even on
|
|
72
|
+
the same machine: its timings are too noisy to settle. Use
|
|
73
|
+
`only: :timing` for work done on threads of its own, and
|
|
74
|
+
`peak_memory: true` where memory use matters.
|
|
75
|
+
- Prefer one thread. Switching between threads costs microseconds,
|
|
76
|
+
which swamps a small cost and leaves its timings unsettled: guard
|
|
77
|
+
threaded code with a single thread exercising it where you can.
|
|
78
|
+
- Don't change anything under `lib/` or `ext/`.
|
|
79
|
+
|
|
80
|
+
4. **Configure only if needed.** `duel init` wrote a `.duel.rb` for
|
|
81
|
+
what it could tell of the project: a gemspec, `ext/`, and a
|
|
82
|
+
`.ruby-version`. A conventional gem needs nothing more. If the
|
|
83
|
+
project isn't one (a non-standard layout, a build step other than
|
|
84
|
+
`rake compile`), change it, using only the settings that Duel's
|
|
85
|
+
`docs/configuration.md` documents.
|
|
86
|
+
|
|
87
|
+
If the code talks to Redis, PostgreSQL, MySQL or memcached, or uses
|
|
88
|
+
Active Record, threads, or long-lived objects, use the matching
|
|
89
|
+
plugin (duel-redis, duel-postgresql, duel-mysql, duel-memcached,
|
|
90
|
+
duel-activerecord, duel-gvl, duel-retained): add its gem, and
|
|
91
|
+
`c.use` it by name in `.duel.rb` (`c.use :redis`). Don't point
|
|
92
|
+
benchmarks at a shared server.
|
|
93
|
+
|
|
94
|
+
If the project is a Rails app, use duel-rails instead of `duel init`
|
|
95
|
+
(which says so) or setting Duel up by hand: add it to the Gemfile's
|
|
96
|
+
development group, run `bin/rails generate duel:install`, read what
|
|
97
|
+
it made, and run `bundle install` again for what it added to the
|
|
98
|
+
Gemfile (StackProf, and a database server's plugin). Then:
|
|
99
|
+
- Benchmark the requests that matter most with `duel_request`, not
|
|
100
|
+
only the GET routes the generator found: the pages users load
|
|
101
|
+
most, and the forms and API calls that write. Give each request
|
|
102
|
+
what it sends (`params:`, `json:`, `headers:`), and sign in with a
|
|
103
|
+
block where the request needs a user.
|
|
104
|
+
- Give the requests data to work with: `fixtures: :all` (or the
|
|
105
|
+
fixtures the requests need), or `seeds: true`, in `.duel.rb`. A
|
|
106
|
+
page with no records to show guards almost nothing.
|
|
107
|
+
- Benchmark the app's slowest models and services with plain `duel`
|
|
108
|
+
blocks too, with their data made in `once`. Roll back those that
|
|
109
|
+
write with `around_iteration: Duel::Rails::ROLLBACK`.
|
|
110
|
+
- Check `config/environments/benchmark.rb`: stub anything a request
|
|
111
|
+
would call over the network.
|
|
112
|
+
|
|
113
|
+
If the project is itself a Duel plugin, don't require or use it in
|
|
114
|
+
its own `.duel.rb`. That would load the candidate's plugin into
|
|
115
|
+
Duel's own workers, under the version being measured. Start any
|
|
116
|
+
server its benchmarks need with `c.prepare(:worker)` directly, with
|
|
117
|
+
every option the plugin itself starts it with: one left out can
|
|
118
|
+
change what the server does (and what the plugin reads of it).
|
|
119
|
+
|
|
120
|
+
If the project's boot time or load time matters (an app, or a library
|
|
121
|
+
that's slow to require), add a `duel_process` benchmark of it, such as
|
|
122
|
+
`duel_process "boot: the app", "bin/rails", "runner", "1"`.
|
|
123
|
+
|
|
124
|
+
If the project pins its Ruby (`.ruby-version`, `.tool-versions`, or
|
|
125
|
+
`mise.toml`), set `c.ruby = :version_file`, so that a branch that
|
|
126
|
+
upgrades Ruby is compared on each version's own Ruby. `duel init`
|
|
127
|
+
does this, and adds the steps in Duel's `docs/ci.md` that install
|
|
128
|
+
the baseline's Ruby to the workflow, for a `.ruby-version`. For another version
|
|
129
|
+
file, do both yourself (see step 6). Duel uses the Ruby that runs it
|
|
130
|
+
when that's the one asked for, so CI needs nothing more until a
|
|
131
|
+
branch changes the Ruby.
|
|
132
|
+
|
|
133
|
+
If the project is in a subdirectory of its repository (a monorepo),
|
|
134
|
+
put `.duel.rb` and `duel/` in that subdirectory, and run `duel` from
|
|
135
|
+
there. Run `duel init` there too: it puts the workflow at the
|
|
136
|
+
repository's root, and runs it in the subdirectory.
|
|
137
|
+
|
|
138
|
+
Make sure the linter checks `.duel.rb`. It's a dotfile, so RuboCop
|
|
139
|
+
leaves it out unless its config includes it (`AllCops: Include`,
|
|
140
|
+
merged with the defaults by `inherit_mode`).
|
|
141
|
+
|
|
142
|
+
5. **Check that the guards hold.** While you're still writing
|
|
143
|
+
benchmarks, run a few at a time (`bundle exec duel --force --filter
|
|
144
|
+
<name>`): a whole run waits on its slowest benchmark, and shows
|
|
145
|
+
nothing of one until it's measured. Duel runs one comparison at a
|
|
146
|
+
time in a project: a second run fails, naming the first one's pid.
|
|
147
|
+
|
|
148
|
+
If the project has a C extension, Duel checks that each version's
|
|
149
|
+
workers load their own build, and fails, saying to set `c.build`, if
|
|
150
|
+
one doesn't (a baseline with no build of its own would load the
|
|
151
|
+
candidate's).
|
|
152
|
+
|
|
153
|
+
Then do all three:
|
|
154
|
+
- Run `bundle exec duel --force`. With `lib/` unchanged, it compares
|
|
155
|
+
the code with itself and must pass. Run it three times: a noisy
|
|
156
|
+
benchmark can pass twice by luck. A benchmark that fails or
|
|
157
|
+
errors is broken. One reported unsettled in more than one run is
|
|
158
|
+
noisy: fix it or drop it. (Across many benchmarks, one unsettled
|
|
159
|
+
once is normal: it doesn't fail the run.) Benchmarks that run a
|
|
160
|
+
lot of other code (a framework's whole query, say) to exercise a
|
|
161
|
+
little of the project's are the noisiest: guard the project's part
|
|
162
|
+
directly instead.
|
|
163
|
+
- Make a deliberate regression in code that a few benchmarks
|
|
164
|
+
exercise: an extra allocation in a hot method, then some extra
|
|
165
|
+
work in a loop (in a C extension, the extra work). Confirm that
|
|
166
|
+
`bundle exec duel` exits 1 and names those benchmarks. Then revert
|
|
167
|
+
the change completely, check that `git diff lib/ ext/` is empty,
|
|
168
|
+
and rebuild the extension if you changed one.
|
|
169
|
+
- Run `bundle exec duel --force --filter <pattern>` for a single
|
|
170
|
+
benchmark, and for one area (`--filter "^parse:"`), to confirm
|
|
171
|
+
your names can be picked out. Without `--force`, an unchanged
|
|
172
|
+
`lib/` has nothing to compare.
|
|
173
|
+
|
|
174
|
+
6. **Check CI.** `duel init` wrote Duel's GitHub Actions
|
|
175
|
+
workflow (`.github/workflows/perf.yml`). Keep `fetch-depth: 0`:
|
|
176
|
+
Duel's baseline comes from git history. Keep its check for a push
|
|
177
|
+
with no commit before it in the history (a branch's first push, or a
|
|
178
|
+
force push), which would otherwise fail the job. Keep its last step,
|
|
179
|
+
which comments on a pull request with Duel's report (Duel also adds
|
|
180
|
+
the report to the job's summary). Check that `ruby-version` is one
|
|
181
|
+
the project supports, unless it has a `.ruby-version`. With
|
|
182
|
+
`c.ruby = :version_file` and another version file than
|
|
183
|
+
`.ruby-version`, add the steps in Duel's `docs/ci.md` that install the
|
|
184
|
+
baseline's Ruby for a pull request. Add the project's other CI setup that its
|
|
185
|
+
tests need (system packages, services) to the workflow as well. Keep
|
|
186
|
+
the scheduled `last-release` run only if the project tags its
|
|
187
|
+
releases (`v*` by default, or else set `c.release_tags`), or leave
|
|
188
|
+
the schedule out. Until the first release is tagged, that run says
|
|
189
|
+
there's nothing to compare against, and passes. Once it's pushed,
|
|
190
|
+
run the workflow by hand: it compares the code with itself there,
|
|
191
|
+
which shows that the benchmarks run on CI's machines (with their
|
|
192
|
+
packages and services).
|
|
193
|
+
|
|
194
|
+
7. **Show the badge.** If the project's README already shows badges (CI
|
|
195
|
+
status, gem version), add Duel's to that row, with exactly this
|
|
196
|
+
markdown:
|
|
197
|
+
|
|
198
|
+
```markdown
|
|
199
|
+
[](https://github.com/sferik/duel-ruby)
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
If the README has no badges, don't add a row just for Duel's.
|
|
203
|
+
|
|
204
|
+
8. **Report back** with:
|
|
205
|
+
- The benchmarks you wrote and what each one guards.
|
|
206
|
+
- What you chose not to benchmark, and why.
|
|
207
|
+
- The output of the self-comparison and of the deliberate-regression
|
|
208
|
+
check.
|
|
209
|
+
- Anything you couldn't guard reliably.
|
|
210
|
+
|
|
211
|
+
Don't commit. Leave the changes for me to review.
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.1.0] - 2026-10-24
|
|
11
|
+
|
|
12
|
+
The first release. Duel is open source under the MIT license, and
|
|
13
|
+
needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
|
|
17
|
+
- `duel` and `Duel::RakeTask`: run every benchmark on a baseline (where
|
|
18
|
+
HEAD left the main branch, a git ref, the last release tag, or a
|
|
19
|
+
directory) and the candidate, side by side in worker processes of their
|
|
20
|
+
own, and fail if the candidate is slower at any of them.
|
|
21
|
+
- `duel init`: sets Duel up in a gem or an app that isn't a Rails app.
|
|
22
|
+
It writes `.duel.rb` (with `c.build` for `ext/`, and
|
|
23
|
+
`c.ruby = :version_file` for a `.ruby-version`), an example benchmark
|
|
24
|
+
with advice on writing good ones, and the GitHub Actions workflow in
|
|
25
|
+
docs/ci.md, adds Duel's Rake task to the Rakefile, and adds
|
|
26
|
+
`gem "stackprof", platforms: :mri` to the Gemfile, for profiles of
|
|
27
|
+
regressions (in its `development` group, if it has one, and not if it
|
|
28
|
+
mentions StackProf already). It overwrites nothing without `--force`,
|
|
29
|
+
and writes nothing with `--dry-run`. The workflow compares pull
|
|
30
|
+
requests, pushes, and, nightly, the last release, and comments on each
|
|
31
|
+
pull request with Duel's report, updating one comment on every push.
|
|
32
|
+
- A report in Markdown, for a pull request. On GitHub Actions
|
|
33
|
+
(`GITHUB_STEP_SUMMARY`), Duel adds it to the job's summary, and
|
|
34
|
+
`--markdown PATH` writes it to a file, to post as a comment. It has the
|
|
35
|
+
verdict, the two versions (and their Rubies, where they differ), a
|
|
36
|
+
table of the benchmarks that changed, regressions first, with each
|
|
37
|
+
timing's ratio and interval and the counts that changed, each
|
|
38
|
+
regression explained as the terminal report explains it, and the
|
|
39
|
+
benchmarks that were unsettled or failed. Those that didn't change are
|
|
40
|
+
collapsed. A run with nothing to compare, or that couldn't compare,
|
|
41
|
+
says so in Markdown too.
|
|
42
|
+
- `--json PATH`: writes the report as JSON, for a program to read, with
|
|
43
|
+
a versioned schema (`"schema": 1`, documented in docs/json.md). It
|
|
44
|
+
has the run's status and exit status, its settings, the two versions
|
|
45
|
+
(each one's ref, commit, tree and Ruby), the plugins in use, and every
|
|
46
|
+
benchmark: its verdict, each timing's ratio, interval, limit, pairs,
|
|
47
|
+
verdict and each version's median an iteration, each count on both
|
|
48
|
+
versions an iteration, peak memory, what metrics said, each
|
|
49
|
+
regression's explanation, and why it failed or was skipped. Then the
|
|
50
|
+
verdicts across the board, and the errors. It's written whatever the
|
|
51
|
+
outcome, nothing to compare and couldn't compare included, but not for
|
|
52
|
+
a command line that isn't valid.
|
|
53
|
+
- Timings (CPU time, and any timed metric of a plugin's) are judged by a
|
|
54
|
+
99% distribution-free confidence interval for the median of paired
|
|
55
|
+
ratios, and whatever looks slower is measured again in fresh processes
|
|
56
|
+
before it fails. A measurement whose timings are all too close to
|
|
57
|
+
call neither confirms a slowdown nor clears it, so Duel measures once
|
|
58
|
+
more in its place, up to three more times. A benchmark that's slower
|
|
59
|
+
in time in every measurement fails even when no one timing is slower
|
|
60
|
+
in all of them (CPU time in one, instructions in the next), as long as
|
|
61
|
+
none says the same or faster, after one measurement more.
|
|
62
|
+
Allocations and counted metrics fail on any increase.
|
|
63
|
+
- Each version runs in several worker processes (three by default,
|
|
64
|
+
`--processes`), paired in turn, so no one process's layout in memory
|
|
65
|
+
biases a comparison.
|
|
66
|
+
- A git baseline is checked out next to the project, at a path as long
|
|
67
|
+
and as deep as the project's, so work a version does on its own paths
|
|
68
|
+
doesn't make the baseline measure slower. Duel says so when the two
|
|
69
|
+
versions' paths still differ in length. A project in a subdirectory
|
|
70
|
+
of its repository (a monorepo's) is compared with the same
|
|
71
|
+
subdirectory of a checkout of the whole repository. Without a merge
|
|
72
|
+
base with the main branch (unrelated histories, or a shallow clone),
|
|
73
|
+
or in a repository without commits, Duel says why.
|
|
74
|
+
- The candidate is built (`c.build`) before every run, as a baseline is,
|
|
75
|
+
so a stale or missing build never decides a comparison. Duel checks
|
|
76
|
+
that each version's workers load only native extensions built in that
|
|
77
|
+
version's tree, and fails, saying to set `c.build`, when one loads
|
|
78
|
+
another version's build or an installed gem's. Gems installed in the
|
|
79
|
+
project's directory (`vendor/bundle`, say) are installed gems, not
|
|
80
|
+
builds.
|
|
81
|
+
- One run at a time in a project: a second `duel` fails, naming the
|
|
82
|
+
first one's pid. The lock is an flock, so a run that crashed leaves
|
|
83
|
+
nothing behind.
|
|
84
|
+
- With YJIT on in a worker (an app, Rails's say, may turn it on as it
|
|
85
|
+
boots), each benchmark warms up for two seconds before it's measured,
|
|
86
|
+
unless `--warm` says otherwise.
|
|
87
|
+
- Instruction counts from the CPU's counters (the benchmark thread's on
|
|
88
|
+
Linux, and the process's on macOS), judged with a limit of their own
|
|
89
|
+
(1%).
|
|
90
|
+
- Peak memory, for benchmarks that ask (`peak_memory: true`): each
|
|
91
|
+
version's least in fresh workers.
|
|
92
|
+
- A StackProf profile of both versions with every regression in time,
|
|
93
|
+
and where the extra CPU went, on CRuby with StackProf in the bundle.
|
|
94
|
+
StackProf is optional, and Duel doesn't use it on JRuby or
|
|
95
|
+
TruffleRuby, even where it's installed. Profiles cover what Duel measures: they
|
|
96
|
+
leave out work inside `untimed` and an `around_iteration` hook's own
|
|
97
|
+
work, keep what `after_iterations` does, hold the garbage collector
|
|
98
|
+
off as samples do, and show what garbage collection gained on a line
|
|
99
|
+
of its own, not ranked with the code. On macOS, they sample elapsed
|
|
100
|
+
time, because CPU-time samples there pile up on code that makes
|
|
101
|
+
system calls.
|
|
102
|
+
- Conventions: benchmarks in `duel/**/*_duel.rb`, the gemspec's library
|
|
103
|
+
required from each version, `rake compile` for a baseline of a
|
|
104
|
+
project with `ext/`, and the main branch from `origin/HEAD` (or
|
|
105
|
+
`origin/main`, `origin/master`, and without a remote, `main` or
|
|
106
|
+
`master`).
|
|
107
|
+
- Across the board: every benchmark's median ratio, judged together, so
|
|
108
|
+
a slowdown too small for any one benchmark still fails. It takes 8
|
|
109
|
+
benchmarks, so with fewer, the report leaves it out.
|
|
110
|
+
- A worker that doesn't load or answer within `--timeout` (60s) is
|
|
111
|
+
killed, and its benchmark is an error. Samples are held to 64 MB of
|
|
112
|
+
allocations while the garbage collector is held off. `duel --clean`
|
|
113
|
+
removes what runs leave behind.
|
|
114
|
+
- `duel "name" do ... end`, in files named `*_duel.rb`, defines a
|
|
115
|
+
benchmark: its block is timed, run as many times as a sample needs
|
|
116
|
+
(or, given `|n|`, once a sample for n iterations). `once { }` in it
|
|
117
|
+
runs its block once, untimed, before anything's measured, and
|
|
118
|
+
`untimed { }` pauses measuring. Duel's own work in an iteration allocates nothing,
|
|
119
|
+
so a benchmark that allocates nothing counts no allocations, and an
|
|
120
|
+
`untimed` block adds none. `only: :count` (work that's mostly waiting)
|
|
121
|
+
or `only: :timing` (work on threads of its own) narrows what it's
|
|
122
|
+
judged on, and `peak_memory: true` adds its peak memory. `.duel.rb`
|
|
123
|
+
covers what the conventions don't: building, loading, preparing each
|
|
124
|
+
worker, and metrics of your own, which together make a plugin (a
|
|
125
|
+
`duel-*` gem that `c.use` uses by name). Nothing else is built in.
|
|
126
|
+
- Hooks, with one set of names at every level: in `.duel.rb`
|
|
127
|
+
(`c.setup { }`), at the top of a file of benchmarks (`setup { }`), and
|
|
128
|
+
as an option of one benchmark (`duel "x", setup: -> { }`). They're
|
|
129
|
+
`setup` and `teardown` (once in each worker, around a benchmark's
|
|
130
|
+
samples), `before_sample` (before every sample, not measured, so every
|
|
131
|
+
sample starts alike), `around_iteration` (around each iteration, which
|
|
132
|
+
it's given to run, and which alone is measured), and
|
|
133
|
+
`after_iterations` (after each sample's iterations, measured with
|
|
134
|
+
them: work that they left for later, such as jobs that they
|
|
135
|
+
enqueued). `.duel.rb`'s run first, then the file's, then the
|
|
136
|
+
benchmark's own (teardown's in reverse, and around_iteration's nested,
|
|
137
|
+
outermost first). A file's and a benchmark's own run in the
|
|
138
|
+
benchmark's context. What pausing and resuming the metrics around
|
|
139
|
+
each iteration leaves is learned in each worker and taken away, and
|
|
140
|
+
pausing allocates nothing.
|
|
141
|
+
- Helpers that files of benchmarks share, in `duel/duel_helper.rb`. A
|
|
142
|
+
helper named as one of the methods that Duel gives benchmarks
|
|
143
|
+
(`once`, `untimed`, a hook, or a DSL method) fails the worker, naming
|
|
144
|
+
its file and line.
|
|
145
|
+
- `duel_process "name", command` (or `ruby: code`): a benchmark of a
|
|
146
|
+
whole process, booting an app or requiring a library, say. Each sample
|
|
147
|
+
is the fastest of a few fresh processes for each version (as many as
|
|
148
|
+
take about a second of CPU time, up to 3, or `runs:`), the versions'
|
|
149
|
+
processes in turn, in its tree and with its environment, judged on its
|
|
150
|
+
CPU time (its children's too) and its peak memory (with a CRuby
|
|
151
|
+
runner: each version's lowest). The processes are started and
|
|
152
|
+
measured by a small Ruby process of Duel's, one for each worker, that
|
|
153
|
+
lasts as long as the worker, so its start-up is paid once and not for
|
|
154
|
+
every process. It's judged after every pair of samples,
|
|
155
|
+
and stops as soon as it's settled that it isn't slower, but it's found
|
|
156
|
+
slower only after a round of 10 pairs, so stopping early can't make
|
|
157
|
+
identical code look slower. It runs in no worker, so it takes none of
|
|
158
|
+
`duel`'s options (a hook such as `around_iteration:`, `only:`, or
|
|
159
|
+
`peak_memory:` raises an `ArgumentError`), and runs no hooks.
|
|
160
|
+
- `c.prepare(:worker) { |w| ... }` prepares each worker with a
|
|
161
|
+
`Duel::Resource`: a directory of the worker's own, `spawn` (logged
|
|
162
|
+
there, and stopped with the worker, by its `stop_signal:`), `run`,
|
|
163
|
+
`wait_until` (failing if a process exits, or after `--timeout`),
|
|
164
|
+
`failed`, `at_stop`, the version's `root` and `ruby`, and `env`, the
|
|
165
|
+
version's environment, which the commands that it starts get under
|
|
166
|
+
their own. The block returns the worker's environment, and two
|
|
167
|
+
prepare blocks that set a variable differently fail.
|
|
168
|
+
- `c.prepare(:version) { |v| ... }`: prepares each version once for a
|
|
169
|
+
run, in the runner, before its workers start (installing its bundle,
|
|
170
|
+
say). It gets a `Duel::Resource` for the version and returns the
|
|
171
|
+
environment that the version's workers and process benchmarks get.
|
|
172
|
+
- `c.ruby` and `duel --ruby`: each version's Ruby. `:version_file` (or
|
|
173
|
+
`--ruby version-file`) runs each version on the Ruby that its own
|
|
174
|
+
`.ruby-version`, `.tool-versions`, or `mise.toml` names, found with
|
|
175
|
+
rbenv, chruby, mise, or asdf. A path, or a block given the version's
|
|
176
|
+
tree, says which otherwise. A version file that asks for the Ruby that
|
|
177
|
+
runs `duel` (as CI's setup step installs it, where no manager looks)
|
|
178
|
+
gets that Ruby. The report shows each version's `RUBY_DESCRIPTION`
|
|
179
|
+
when they differ, and a Ruby that isn't installed fails the run.
|
|
180
|
+
- `c.in_worker { |w| }`: set-up that runs only in workers, once the
|
|
181
|
+
benchmarks have loaded. What it gives `w.at_stop` runs when the
|
|
182
|
+
worker ends. It returns what the block made in each (a connection,
|
|
183
|
+
say), as `value`, the worker's.
|
|
184
|
+
- `w.cached(key) { |dir| }`: what a prepare block builds once, for every
|
|
185
|
+
worker of a run and later runs (a database's initial files, say).
|
|
186
|
+
- A run's workers start at once, each prepared, and stop at once.
|
|
187
|
+
- Metrics of running totals: `c.metric("name") { total }`, or several
|
|
188
|
+
by their settings, and a sample measures what they grew by, less what
|
|
189
|
+
reading a counted one adds to it (learned in each worker, unless
|
|
190
|
+
`calibrate: false`). A `Duel::Metric` subclass covers the rest: its
|
|
191
|
+
`measures` name what it measures, with their settings, its
|
|
192
|
+
`start_sample` gets the sample (`n`, `benchmark`), and it can
|
|
193
|
+
`pause_sample` and `resume_sample` around `untimed` blocks its own
|
|
194
|
+
way.
|
|
195
|
+
- `open { }` and `close { }` on what `c.metric` returns, and `open` and
|
|
196
|
+
`close` methods of a `Duel::Metric` subclass: a metric's own setup and
|
|
197
|
+
teardown in each worker. `details`, `open`, `close`, and `costly`
|
|
198
|
+
return the metric, to chain them.
|
|
199
|
+
- `costly` on what `c.metric` returns, and `self.costly?` on a
|
|
200
|
+
`Duel::Metric` subclass: a metric that's costly to read (a round trip
|
|
201
|
+
to a server, say) isn't paused around each iteration inside an
|
|
202
|
+
`around_iteration` hook, but read only as each sample starts and
|
|
203
|
+
stops. What the hook adds to it is learned in each worker, from three
|
|
204
|
+
samples of 10 iterations in which it's paused around each iteration,
|
|
205
|
+
and subtracted, so the hook's own work (a transaction's `BEGIN` and
|
|
206
|
+
`ROLLBACK`, say) still isn't counted, and the metric measures what
|
|
207
|
+
pausing would have. The plugins that read a server (duel-postgresql,
|
|
208
|
+
duel-mysql, duel-redis and duel-memcached) say so.
|
|
209
|
+
- Settings for metrics: `floor` (on either version), `limit`, `unit_ns`,
|
|
210
|
+
`sizes_samples`, `tolerance` (per iteration) and `allowance` (per
|
|
211
|
+
sample). `c.measure(name, ...)` changes any measure's, Duel's own
|
|
212
|
+
too, or skips it. Two metrics that measure the same name fail.
|
|
213
|
+
- A regression is explained an iteration, on each version
|
|
214
|
+
(`cpu: 112.4µs → 121.5µs an iteration`), with what its metric says of
|
|
215
|
+
it (`Metric#details`), and results carry each version's raw medians.
|
|
216
|
+
Each benchmark's line shows its counts an iteration too.
|
|
217
|
+
- `unit:` for a measure (a time unit shows it as time, and gives
|
|
218
|
+
`unit_ns`). Settings there aren't, and kinds there aren't, fail.
|
|
219
|
+
`Metric#settings` has a metric's settings after `c.measure`'s changes,
|
|
220
|
+
every one filled in. Running totals' `details { |stretches, sample| }`
|
|
221
|
+
explains them, and their block can return `[totals, context]` to pass
|
|
222
|
+
the context on (typed, with Steep). `Reading#total` reads one total.
|
|
223
|
+
Units show with regressions, and on the summary line unless the
|
|
224
|
+
measure's name ends in them. `Duel::Testing.measure` returns what
|
|
225
|
+
metrics said (`#details`) with what they measured.
|
|
226
|
+
- `kind: :ratio` (judged by paired ratios, as CPU time is) and
|
|
227
|
+
`:counted`, with `cpu_floor:` (a floor that's a share of CPU time) and
|
|
228
|
+
`across: false` (not judged across the board).
|
|
229
|
+
- `c.use :redis`: uses a plugin by name. It requires `duel/redis` and
|
|
230
|
+
calls the `configure` method of the module that the file registers
|
|
231
|
+
with `Duel.plugin(:redis, self)`, given the configuration and the
|
|
232
|
+
options. A plugin is configured once, however many times it's used, so
|
|
233
|
+
a plugin that needs another (duel-rails, duel-activerecord) uses it
|
|
234
|
+
too, and an app's own use of it, with options, comes first. A plugin
|
|
235
|
+
that isn't in the bundle fails, naming the gem to add. The report, in
|
|
236
|
+
the terminal and in Markdown, lists the plugins used. Workers load
|
|
237
|
+
Duel and every Duel plugin that the runner loaded (the `duel` and
|
|
238
|
+
`duel-*` gems) from where the runner did, ahead of the version's own
|
|
239
|
+
bundle, so a version whose bundle lacks them (or has other versions of
|
|
240
|
+
them) is measured with the same.
|
|
241
|
+
- A plugin's own DSL methods: those of its module's `DSL` module, which
|
|
242
|
+
files of benchmarks get once `.duel.rb` uses the plugin. A name that's
|
|
243
|
+
taken, by the DSL, another plugin, or every object, raises
|
|
244
|
+
`Duel::Error` instead of replacing it. A benchmark that the method
|
|
245
|
+
defines, and a hook that it adds, belong to the file of benchmarks
|
|
246
|
+
that called it, which gets that file's hooks.
|
|
247
|
+
- `duel/testing`, for plugins' tests: `Duel::Testing.prepare` (which
|
|
248
|
+
runs the configuration's `c.prepare(:version)` blocks, then its
|
|
249
|
+
`c.prepare(:worker)` blocks, as a run does), `measure`, `in_worker`
|
|
250
|
+
(which ends as a worker ends, closing its metrics), `metrics`,
|
|
251
|
+
`compare` and `project`. `measure(config, **hooks)` measures the
|
|
252
|
+
block as a benchmark with the configuration's hooks and its own, as
|
|
253
|
+
`duel` takes them. With an `around_iteration` hook, the block is one
|
|
254
|
+
iteration, run `n` times, each inside the hook, costly metrics and
|
|
255
|
+
all. The block gets the benchmark's context, with `untimed`.
|
|
256
|
+
- `Duel::Testing.sample`, `reading` and `details`: stand-ins for a
|
|
257
|
+
sample and for readings of running totals, to test or benchmark what a
|
|
258
|
+
metric says of a sample. docs/plugins.md documents what `details` is
|
|
259
|
+
given and when it runs.
|
|
260
|
+
- `Metric#setting(:allowance)`, for a metric of one measure, and
|
|
261
|
+
`Metric#within?(more, n)` (and `Metric.within?`): whether a count is
|
|
262
|
+
within its tolerance and allowance, as Duel judges it.
|
|
263
|
+
- A unit that's a word comes with its plural (`unit: %w[row rows]`: 1
|
|
264
|
+
row, 2 rows), and a symbol (`"B"`) is one String.
|
|
265
|
+
- `Duel.measuring?`: whether a benchmark is being measured right now
|
|
266
|
+
(not in setup or `untimed`), and `Duel.unmeasured!(name)`, for a
|
|
267
|
+
plugin's helper to fail where what it does would be counted.
|
|
268
|
+
- Running totals are typed from their block (Integer totals stay
|
|
269
|
+
Integer), and `Duel::Testing.measure` ends the worker it starts.
|
|
270
|
+
- `DUEL_BASE`, `DUEL_FILTER` and `DUEL_FORCE` set `--base`, `--filter`
|
|
271
|
+
and `--force` where options are hard to pass (behind a rake task, or
|
|
272
|
+
in a CI expression). An invalid regular expression for `--filter` or
|
|
273
|
+
`DUEL_FILTER` is a usage error, with exit status 2.
|
|
274
|
+
- Mutation tested: mutant kills every change to Duel's code, in CI.
|
|
275
|
+
- RBS signatures in sig/, checked with Steep.
|
|
276
|
+
- The public API is what the README and the guide in docs/ document.
|
|
277
|
+
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
278
|
+
release.
|
|
279
|
+
|
|
280
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.0...HEAD
|
|
281
|
+
[0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Erik Berlin
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|