duel 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENT_PROMPT.md +68 -31
- data/CHANGELOG.md +158 -1
- data/README.md +16 -13
- data/docs/README.md +1 -1
- data/docs/benchmarks.md +32 -3
- data/docs/ci.md +365 -167
- data/docs/configuration.md +11 -4
- data/docs/how-it-works.md +109 -35
- data/docs/json.md +16 -10
- data/docs/plugins.md +3 -1
- data/docs/troubleshooting.md +6 -6
- data/lib/duel/across.rb +100 -9
- data/lib/duel/baseline.rb +26 -2
- data/lib/duel/cli.rb +18 -7
- data/lib/duel/config.rb +4 -1
- data/lib/duel/init.rb +55 -1
- data/lib/duel/json_report.rb +2 -2
- data/lib/duel/metric.rb +32 -1
- data/lib/duel/rake_task.rb +10 -1
- data/lib/duel/rubies.rb +39 -0
- data/lib/duel/runner.rb +337 -49
- data/lib/duel/stats.rb +76 -0
- data/lib/duel/summary.rb +2 -2
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +3 -0
- data/lib/duel/templates/duel.rb.erb +7 -3
- data/lib/duel/templates/perf-comment.yml.erb +68 -4
- data/lib/duel/templates/perf.yml.erb +136 -38
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +5 -4
- data/sig/duel/across.rbs +24 -2
- data/sig/duel/baseline.rbs +12 -0
- data/sig/duel/cli.rbs +4 -0
- data/sig/duel/init.rbs +23 -0
- data/sig/duel/metric.rbs +19 -1
- data/sig/duel/rake_task.rbs +3 -0
- data/sig/duel/rubies.rbs +13 -0
- data/sig/duel/runner.rbs +113 -7
- data/sig/duel/stats.rbs +25 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a3bd63fc0a6b4eb33d7fe48f733446e7b35c3e961dfdc42e85531fdc680ca53c
|
|
4
|
+
data.tar.gz: d44709c7629208381a9e24235f2401dc53c69c11fd540c8b8ab5c4e02710bf7a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 44fd8dbe608296d4af112bd27525f38d544662e858e39abfc4913583fa1a0d5d74cd9c20c3745dcc226ef1c6f89c6deb9f2f6780bb457627da18e0b7bf82373e
|
|
7
|
+
data.tar.gz: 6984d61ce88c4042e8447a9ec03886bb68fa39d4e2d5bdef448a6b4dd1cd7ef6d258a940aac56b3bd08892907fc4d626cd4f91dcbbcbca634b29c78227c7f860
|
data/AGENT_PROMPT.md
CHANGED
|
@@ -17,10 +17,11 @@ at the API.
|
|
|
17
17
|
development group (unless the Gemfile has it already), and run
|
|
18
18
|
`bundle install`. Then, unless the project is a Rails app (see step
|
|
19
19
|
4), run `bundle exec duel init`, and read what it wrote: `.duel.rb`,
|
|
20
|
-
an example benchmark in `duel/`,
|
|
21
|
-
(`.github/workflows/perf.yml
|
|
22
|
-
|
|
23
|
-
|
|
20
|
+
an example benchmark in `duel/`, two GitHub Actions workflows
|
|
21
|
+
(`.github/workflows/perf.yml`, and `perf-comment.yml`, which comments
|
|
22
|
+
on pull requests), Duel's Rake task, if there's a Rakefile, and
|
|
23
|
+
`gem "stackprof", platforms: :mri` in the Gemfile, for profiles of
|
|
24
|
+
regressions. It lists what it did and what's next: run
|
|
24
25
|
`bundle install` again for StackProf.
|
|
25
26
|
|
|
26
27
|
2. **Find what's worth guarding.** Read the code and its tests to find
|
|
@@ -58,6 +59,22 @@ at the API.
|
|
|
58
59
|
includes what differs between the two versions' workers, like a
|
|
59
60
|
server's socket path or a worker's directory: don't time work on
|
|
60
61
|
it.
|
|
62
|
+
- Build inputs without the code you're measuring. Each version's
|
|
63
|
+
workers run the benchmarks with that version's code, so an input
|
|
64
|
+
made with the project's own encoder (its `dump`, `to_json` or
|
|
65
|
+
`serialize`) differs wherever the two versions' output does, and
|
|
66
|
+
the benchmark then compares different work. Write the input out as
|
|
67
|
+
a fixture, or build it from literals or Ruby's standard library.
|
|
68
|
+
See "Give both versions the same work" in Duel's
|
|
69
|
+
`docs/benchmarks.md`.
|
|
70
|
+
- Leave process-wide settings alone. Every worker loads every
|
|
71
|
+
benchmark file, so a mode, patch or global option that changes the
|
|
72
|
+
whole process (an `optimize` call, a changed default) changes every
|
|
73
|
+
benchmark that runs after it, and the default path goes unmeasured.
|
|
74
|
+
Never turn one on in `duel_helper.rb` or at a file's top level.
|
|
75
|
+
Turn it on only in the benchmarks that are about it, and put it
|
|
76
|
+
back in `teardown`, or measure it in a process of its own with
|
|
77
|
+
`duel_process`.
|
|
61
78
|
- Use the sizes the code really sees. Read it for limits and
|
|
62
79
|
constants (a cap on how many items it ever handles, say) rather
|
|
63
80
|
than guessing.
|
|
@@ -135,9 +152,12 @@ at the API.
|
|
|
135
152
|
there. Run `duel init` there too: it puts the workflow at the
|
|
136
153
|
repository's root, and runs it in the subdirectory.
|
|
137
154
|
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
155
|
+
Don't change the project's own configuration beyond what `duel init`
|
|
156
|
+
did: its linter's config, its README (but for step 7), its other CI
|
|
157
|
+
workflows. Those are the maintainers' choices. If the linter flags
|
|
158
|
+
the benchmarks, fix the benchmarks. If it can't be satisfied without
|
|
159
|
+
a config change (it leaves `.duel.rb` out, say, as RuboCop does with
|
|
160
|
+
dotfiles), say so in your report instead.
|
|
141
161
|
|
|
142
162
|
5. **Check that the guards hold.** While you're still writing
|
|
143
163
|
benchmarks, run a few at a time (`bundle exec duel --force --filter
|
|
@@ -171,29 +191,39 @@ at the API.
|
|
|
171
191
|
your names can be picked out. Without `--force`, a tree with
|
|
172
192
|
nothing changed since the main branch has nothing to compare.
|
|
173
193
|
|
|
174
|
-
6. **Check CI.** `duel init` wrote
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
`
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
194
|
+
6. **Check CI.** `duel init` wrote two GitHub Actions workflows.
|
|
195
|
+
- `.github/workflows/perf.yml` runs Duel. Keep `fetch-depth: 0`:
|
|
196
|
+
Duel's baseline comes from git history. Keep its check for a push
|
|
197
|
+
with no commit before it in the history (a branch's first push, or
|
|
198
|
+
a force push), which would otherwise fail the job. The Duel job
|
|
199
|
+
runs the branch's code, so it only reads: don't give it, or the
|
|
200
|
+
workflow, any write permission.
|
|
201
|
+
- `.github/workflows/perf-comment.yml` comments on a pull request
|
|
202
|
+
with Duel's report once the first has run (`c.comment` in
|
|
203
|
+
`.duel.rb` chooses when, and Duel also adds the report to the
|
|
204
|
+
job's summary). It runs none of the pull request's code, and shows
|
|
205
|
+
a fork's report as text. Keep it as it is.
|
|
206
|
+
- `duel init` writes the jobs for pushes (a comment on a pushed
|
|
207
|
+
commit, and a `duel/passed/` branch that marks the last commit
|
|
208
|
+
Duel passed) only where `.duel.rb` sets `c.fail_on =
|
|
209
|
+
:pull_requests` or `c.push_base = :last_passed`, and the badge's
|
|
210
|
+
job only with `--badge`. Each needs `contents: write`, so don't
|
|
211
|
+
set those settings, or add those jobs, unless I ask.
|
|
212
|
+
|
|
213
|
+
Check that `ruby-version` is one the project supports, unless it has
|
|
214
|
+
a `.ruby-version`. With `c.ruby = :version_file` and another version
|
|
215
|
+
file than `.ruby-version`, add the steps in Duel's `docs/ci.md` that
|
|
216
|
+
install the baseline's Ruby, reading that file instead. If
|
|
217
|
+
`.duel.rb` sets `c.main_branch` or `c.release_tags`, set those
|
|
218
|
+
steps' `MAIN_BRANCH` or `RELEASE_TAGS` to match. Add the project's
|
|
219
|
+
other CI setup that its tests need (system packages, services) to
|
|
220
|
+
the Duel job as well. Keep the scheduled `last-release` run only if
|
|
221
|
+
the project tags its releases (`v*` by default, or else set
|
|
222
|
+
`c.release_tags`), or leave the schedule out. Until the first
|
|
223
|
+
release is tagged, that run says there's nothing to compare against,
|
|
224
|
+
and passes. Once it's pushed, run the workflow by hand: it compares
|
|
225
|
+
the code with itself there, which shows that the benchmarks run on
|
|
226
|
+
CI's machines (with their packages and services).
|
|
197
227
|
|
|
198
228
|
7. **Show the badge.** If the project's README already shows badges (CI
|
|
199
229
|
status, gem version), add Duel's to that row, with exactly this
|
|
@@ -203,7 +233,10 @@ at the API.
|
|
|
203
233
|
[](https://github.com/sferik/duel-ruby)
|
|
204
234
|
```
|
|
205
235
|
|
|
206
|
-
If the README has no badges, don't add a row just for Duel's.
|
|
236
|
+
If the README has no badges, don't add a row just for Duel's. This
|
|
237
|
+
badge is always green: one that turns red when Duel fails a push
|
|
238
|
+
needs the badge's job (`duel init --badge`), so add that only if I
|
|
239
|
+
ask.
|
|
207
240
|
|
|
208
241
|
8. **Report back** with:
|
|
209
242
|
- The benchmarks you wrote and what each one guards.
|
|
@@ -211,5 +244,9 @@ at the API.
|
|
|
211
244
|
- The output of the self-comparison and of the deliberate-regression
|
|
212
245
|
check.
|
|
213
246
|
- Anything you couldn't guard reliably.
|
|
247
|
+
- Every file you changed outside `duel/` and `.duel.rb`, and why.
|
|
248
|
+
- The label that lets a pull request that's slower on purpose pass
|
|
249
|
+
(`c.accept_label`, `duel: accept` in the `.duel.rb` that
|
|
250
|
+
`duel init` writes), which the repository needs to have.
|
|
214
251
|
|
|
215
252
|
Don't commit. Leave the changes for me to review.
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,161 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.6] - 2026-11-10
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- The comment workflow that `duel init` writes runs the Duel job again
|
|
15
|
+
when the label that accepts a regression (`c.accept_label`) is added
|
|
16
|
+
to a pull request whose run failed, or taken off one whose run
|
|
17
|
+
passed. Before, the label did nothing until someone ran the job again
|
|
18
|
+
by hand. The job runs on `pull_request_target`, checks nothing out,
|
|
19
|
+
and has only `actions: write`.
|
|
20
|
+
- On a schedule, the workflow compares with the last commit since the
|
|
21
|
+
last release that merged a pull request with the label, instead of
|
|
22
|
+
the release, so a slowdown accepted in its pull request doesn't fail
|
|
23
|
+
every nightly run until the next release. Duel reads that commit from
|
|
24
|
+
`DUEL_ACCEPTED`, and the report says what it compared with.
|
|
25
|
+
- The JSON report says, for each timing and across the board, how many
|
|
26
|
+
worker processes of each version the interval is over (`processes`),
|
|
27
|
+
or `null` where it's over pairs or benchmarks.
|
|
28
|
+
- The nightly bias check (`tasks/bias_check.rb`) also fails where more
|
|
29
|
+
of a metric's intervals of identical code leave out 1 than their
|
|
30
|
+
confidence allows, and runs on JRuby too, and on a pull request that
|
|
31
|
+
changes how Duel measures or judges.
|
|
32
|
+
|
|
33
|
+
- On JRuby, Duel counts the bytes each benchmark allocates
|
|
34
|
+
(`allocated_bytes`), as the JVM counts them for each thread (or every
|
|
35
|
+
thread, for a benchmark with threads of its own), and fails one that
|
|
36
|
+
allocates more than 32 bytes more an iteration. Before, JRuby had no
|
|
37
|
+
allocation count. Where each JVM's JIT does away with different
|
|
38
|
+
allocations (escape analysis), workers' counts differ, and Duel finds
|
|
39
|
+
only as many more as they can tell apart.
|
|
40
|
+
- A count is over workers too: each worker's fewest in three rounds of
|
|
41
|
+
a sample from every worker, and it fails when an interval for how many
|
|
42
|
+
more the candidate's workers count than the baseline's lies above the
|
|
43
|
+
tolerance. Before, it was each version's fewest of 3 samples, and on
|
|
44
|
+
JRuby, where one worker's JIT can do away with an allocation another's
|
|
45
|
+
doesn't, one worker's luck could fail identical code. Where every
|
|
46
|
+
worker counts the same, as on CRuby, it's as exact as before. With
|
|
47
|
+
fewer than 5 workers of each version, it's the fewest of 3 samples (10
|
|
48
|
+
on JRuby), as before.
|
|
49
|
+
- On JRuby and TruffleRuby, a worker may take 5 minutes to load or to
|
|
50
|
+
answer (`--timeout`), not 1: with `-Xbatch`, 16 JVMs starting at once
|
|
51
|
+
could take longer than a minute.
|
|
52
|
+
- Counts of 9999.5 or more an iteration show as whole numbers (25736, not
|
|
53
|
+
2.574e+04).
|
|
54
|
+
|
|
55
|
+
### Changed
|
|
56
|
+
|
|
57
|
+
- A timing's interval is over worker processes, not pairs of samples.
|
|
58
|
+
Where things land in a process's memory (or on JRuby, what its JIT
|
|
59
|
+
compiles) can make the same code faster or slower for the process's
|
|
60
|
+
whole life, so the pairs of 3 workers of each version were far from
|
|
61
|
+
independent, and with identical code on both sides, their intervals
|
|
62
|
+
left out 1 several times as often as 99% intervals should. Each
|
|
63
|
+
worker's median is now a sample, and the interval is the Mann-Whitney
|
|
64
|
+
interval for the ratio of the candidate's workers to the baseline's.
|
|
65
|
+
Duel takes rounds of a sample from every worker, up to some 120
|
|
66
|
+
pairs, and starts 12 workers of each version by default (8 on JRuby
|
|
67
|
+
and TruffleRuby). CPU-time intervals are about as wide as before, or
|
|
68
|
+
narrower, and instruction counts' (which a process repeats almost
|
|
69
|
+
exactly) a little wider. On JRuby, they're some 5% to 8% wide, as
|
|
70
|
+
wide as each JVM's JIT makes them, so Duel leaves smaller timing
|
|
71
|
+
regressions there unsettled. With fewer than 5 workers of each version
|
|
72
|
+
(`--processes`, and `Duel::Testing.compare`'s 2), and for a process
|
|
73
|
+
benchmark, the interval is over the pairs, as before, and the first
|
|
74
|
+
line of the output says which.
|
|
75
|
+
- Across the board, the interval is over the workers too: each worker's
|
|
76
|
+
score is how much faster or slower it was than its lane's other
|
|
77
|
+
workers, at the median of the benchmarks it measured. A worker whose
|
|
78
|
+
luck slowed it at every benchmark could fail identical code across
|
|
79
|
+
the board.
|
|
80
|
+
- Without `--warm`, each worker warms a benchmark up until it's steady
|
|
81
|
+
(the median of its samples over each of three half seconds in a row
|
|
82
|
+
is no more than 1% faster than over the last half second that was),
|
|
83
|
+
for up to a minute on JRuby and TruffleRuby, or 10 seconds with YJIT
|
|
84
|
+
on. Before, it warmed up for 5 seconds, or 2, every time, and on
|
|
85
|
+
JRuby, some code was still several times faster after that, so it
|
|
86
|
+
was measured while it got faster. `--warm` still warms up for the time
|
|
87
|
+
it says.
|
|
88
|
+
- On JRuby, each worker's JVM compiles in the foreground (`-Xbatch`) and
|
|
89
|
+
collects garbage with the serial collector, after any options in
|
|
90
|
+
`JRUBY_OPTS` (and without the serial collector where they choose
|
|
91
|
+
another). Each JVM's JIT compiles the same code a little differently,
|
|
92
|
+
and that made JRuby's intervals over workers twice as wide.
|
|
93
|
+
- A run measures for two hours at most by default (`--time-limit`), not
|
|
94
|
+
half an hour, as on JRuby a benchmark can take half a minute to warm
|
|
95
|
+
up.
|
|
96
|
+
- On JRuby and TruffleRuby, Duel measures one benchmark at a time for
|
|
97
|
+
every four CPUs (`--jobs`), not every two: their JIT compilers keep the
|
|
98
|
+
other CPUs busy, so more were no faster, and each worker is a JVM of
|
|
99
|
+
some 500 MB.
|
|
100
|
+
|
|
101
|
+
### Fixed
|
|
102
|
+
|
|
103
|
+
- The comment workflow fences a fork's report with more backticks than
|
|
104
|
+
any run of them in it even when the report has a NUL byte. GNU grep
|
|
105
|
+
took such a report for binary and found no backticks, so the fence
|
|
106
|
+
was three, and a line of three backticks in the report closed it. The
|
|
107
|
+
workflow now drops NUL bytes, which GitHub doesn't take in a comment,
|
|
108
|
+
and reads the report as text whatever's in it.
|
|
109
|
+
- The Mark the commit passed job runs one at a time for each branch
|
|
110
|
+
(and on GitLab CI, one at a time). Before the branch that marks the
|
|
111
|
+
last commit Duel passed existed, two pushes that passed at once could
|
|
112
|
+
both try to make it, and the second failed, saying another branch
|
|
113
|
+
kept it from being made. Where making the branch fails, the job now
|
|
114
|
+
also looks for it again before it says so. Update the job in
|
|
115
|
+
`.github/workflows/perf.yml` (or `.gitlab/duel.gitlab-ci.yml`) to pick
|
|
116
|
+
this up.
|
|
117
|
+
|
|
118
|
+
## [0.1.5] - 2026-11-08
|
|
119
|
+
|
|
120
|
+
### Changed
|
|
121
|
+
|
|
122
|
+
- `duel init` writes the Comment on the commit job only where `.duel.rb`
|
|
123
|
+
sets `c.fail_on = :pull_requests`, and the Mark the commit passed job
|
|
124
|
+
only where it sets `c.push_base = :last_passed`, as with the defaults
|
|
125
|
+
neither runs, and each asks for `contents: write`. It reads the
|
|
126
|
+
settings from `.duel.rb` where there is one, and where that doesn't
|
|
127
|
+
load (one of its plugins isn't installed, say), writes both, and says
|
|
128
|
+
so. To add a job later, set its setting, remove the workflow, and run
|
|
129
|
+
`duel init` again, or add it from the guide.
|
|
130
|
+
- `duel init` writes the Update the badge job only with `--badge`. It
|
|
131
|
+
pushes a branch, so it needs `contents: write`, and without the jobs
|
|
132
|
+
for pushes, it was the workflow's only job that writes to the
|
|
133
|
+
repository. The workflow `duel init` writes by default now only reads.
|
|
134
|
+
To keep a badge you already show, add `--badge` when you run
|
|
135
|
+
`duel init` again, or keep the job you have.
|
|
136
|
+
- The `.duel.rb` that `duel init` writes sets
|
|
137
|
+
`c.accept_label = "duel: accept"`, so a pull request that's slower on
|
|
138
|
+
purpose can pass with that label. Duel's own default is still no
|
|
139
|
+
label, so a `.duel.rb` you already have is unchanged.
|
|
140
|
+
- `require "duel/rake_task"` loads only the task, and the task loads
|
|
141
|
+
Duel as it runs, so a Rakefile with Duel's task no longer loads all of
|
|
142
|
+
Duel (and what it requires) for every other task, `rake test` included.
|
|
143
|
+
|
|
144
|
+
### Fixed
|
|
145
|
+
|
|
146
|
+
- The Perf comment workflow posted the report that a pull request's run
|
|
147
|
+
wrote as Markdown, as `github-actions[bot]`. A fork's run runs the
|
|
148
|
+
fork's code, which could write anything there, mentions, links and
|
|
149
|
+
images included. A fork's report is now shown as text, in a code
|
|
150
|
+
block, where it can't do any of that, and cut short where a comment
|
|
151
|
+
couldn't hold it. A pull request from a branch of the repository, whose
|
|
152
|
+
author can push to it anyway, still gets the Markdown. Update
|
|
153
|
+
`.github/workflows/perf-comment.yml` to pick this up.
|
|
154
|
+
- The Update the badge job downloaded the badge's image from Duel's own
|
|
155
|
+
repository, so a change there changed every project's badge, and a
|
|
156
|
+
failed download failed the job. It now writes the image itself, and
|
|
157
|
+
fetches nothing. Update it in `.github/workflows/perf.yml` to pick
|
|
158
|
+
this up.
|
|
159
|
+
- Two pushes' runs of the Update the badge job could overlap, as a
|
|
160
|
+
push's run doesn't wait for the one before it, and an older push's
|
|
161
|
+
could write its badge over a newer one's after both had checked
|
|
162
|
+
which push the badge was for. The job now runs one at a time. Update
|
|
163
|
+
it in `.github/workflows/perf.yml` to pick this up.
|
|
164
|
+
|
|
10
165
|
## [0.1.4] - 2026-11-06
|
|
11
166
|
|
|
12
167
|
### Fixed
|
|
@@ -527,7 +682,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
527
682
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
528
683
|
release.
|
|
529
684
|
|
|
530
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.
|
|
685
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.6...HEAD
|
|
686
|
+
[0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
|
|
687
|
+
[0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
|
|
531
688
|
[0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
|
|
532
689
|
[0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
|
|
533
690
|
[0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
|
data/README.md
CHANGED
|
@@ -14,7 +14,7 @@ code from becoming slower.
|
|
|
14
14
|
|
|
15
15
|
```sh
|
|
16
16
|
$ duel
|
|
17
|
-
Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02)
|
|
17
|
+
Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02, over 12 worker processes of each version)
|
|
18
18
|
parse: a small document 0.6s n=412 cpu 0.998 [0.991, 1.006] same allocations 20
|
|
19
19
|
! parse: a large document 1.1s n=9 cpu 1.081 [1.062, 1.097] slower allocations 5790
|
|
20
20
|
...
|
|
@@ -83,7 +83,10 @@ out are Duel's.
|
|
|
83
83
|
|
|
84
84
|
`duel init` doesn't overwrite a file that exists. To overwrite it, use
|
|
85
85
|
`--force`. To see what `duel init` would do without writing anything,
|
|
86
|
-
use `--dry-run`.
|
|
86
|
+
use `--dry-run`. For a badge in your README that turns red when Duel
|
|
87
|
+
fails a push, use `--badge` (see
|
|
88
|
+
[Show that a project is guarded](#show-that-a-project-is-guarded)).
|
|
89
|
+
When it's done, `duel init` lists the next steps.
|
|
87
90
|
|
|
88
91
|
For a Rails app, use
|
|
89
92
|
[duel-rails](https://github.com/sferik/duel-ruby/tree/main/plugins/rails)
|
|
@@ -190,11 +193,11 @@ and how it compares a project in a subdirectory of its repository, see
|
|
|
190
193
|
### Timeouts and exit status
|
|
191
194
|
|
|
192
195
|
A benchmark that hangs, or that waits on a server that died, doesn't
|
|
193
|
-
hang the run. If a worker takes more than a minute
|
|
194
|
-
Duel stops the worker and reports the
|
|
195
|
-
the limit, use `--timeout`.
|
|
196
|
+
hang the run. If a worker takes more than a minute (five on JRuby and
|
|
197
|
+
TruffleRuby) to load or to answer, Duel stops the worker and reports the
|
|
198
|
+
benchmark as an error. To change the limit, use `--timeout`.
|
|
196
199
|
|
|
197
|
-
A run measures for
|
|
200
|
+
A run measures for two hours at most, not counting the builds. Past
|
|
198
201
|
that, Duel starts no more measurements. A benchmark that it hasn't
|
|
199
202
|
measured yet fails as an error, and a benchmark that looked worse, but
|
|
200
203
|
that Duel hasn't measured again as often as it usually does, fails if it
|
|
@@ -335,17 +338,17 @@ says.
|
|
|
335
338
|
### Show that a project is guarded
|
|
336
339
|
|
|
337
340
|
To show that Duel guards your project, add its badge to your README,
|
|
338
|
-
next to your other badges. With the
|
|
339
|
-
the badge turns red when Duel fails a push to your default
|
|
340
|
-
Replace `OWNER/REPO` with your repository's:
|
|
341
|
+
next to your other badges. With the job that `duel init --badge` adds to
|
|
342
|
+
its workflow, the badge turns red when Duel fails a push to your default
|
|
343
|
+
branch. Replace `OWNER/REPO` with your repository's:
|
|
341
344
|
|
|
342
345
|
```markdown
|
|
343
346
|
[](https://github.com/sferik/duel-ruby)
|
|
344
347
|
```
|
|
345
348
|
|
|
346
349
|
See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
|
|
347
|
-
for how the
|
|
348
|
-
|
|
350
|
+
for how the job keeps it. Without the job, such as on GitLab CI, use
|
|
351
|
+
Duel's own badge, which is always green:
|
|
349
352
|
|
|
350
353
|
```markdown
|
|
351
354
|
[](https://github.com/sferik/duel-ruby)
|
|
@@ -388,8 +391,8 @@ plugin, see [Write a Duel plugin](docs/plugins.md).
|
|
|
388
391
|
The [Duel guide](docs/README.md) covers the rest:
|
|
389
392
|
|
|
390
393
|
- [Write benchmarks](docs/benchmarks.md): hooks, undoing each
|
|
391
|
-
iteration, sharing code,
|
|
392
|
-
benchmarking a whole process.
|
|
394
|
+
iteration, sharing code, giving both versions the same work, choosing
|
|
395
|
+
what a benchmark is judged on, and benchmarking a whole process.
|
|
393
396
|
- [Configure Duel](docs/configuration.md): `.duel.rb`, preparing each
|
|
394
397
|
worker and each version, each version's own Ruby, your own metrics,
|
|
395
398
|
and plugins.
|
data/docs/README.md
CHANGED
|
@@ -5,7 +5,7 @@ benchmark, run it, and run it in CI. This guide covers the rest.
|
|
|
5
5
|
|
|
6
6
|
| Page | What it covers |
|
|
7
7
|
|---|---|
|
|
8
|
-
| [Write benchmarks](benchmarks.md) | Hooks that run code around a benchmark, starting every sample alike, undoing each iteration, measuring work that a benchmark leaves for later, sharing code between benchmarks, choosing what a benchmark is judged on, and benchmarking a whole process. |
|
|
8
|
+
| [Write benchmarks](benchmarks.md) | Hooks that run code around a benchmark, starting every sample alike, undoing each iteration, measuring work that a benchmark leaves for later, sharing code between benchmarks, giving both versions the same work, choosing what a benchmark is judged on, and benchmarking a whole process. |
|
|
9
9
|
| [Configure Duel](configuration.md) | `.duel.rb` and its settings, preparing each worker and each version, running code in each worker, running each version on its own Ruby, adding your own metrics, and using plugins. |
|
|
10
10
|
| [Run Duel in CI](ci.md) | The GitHub Actions workflow that `duel init` writes, catching slow creep, commenting on a pull request, and installing the baseline's Ruby on CI. |
|
|
11
11
|
| [How Duel works](how-it-works.md) | How Duel tells a regression from noise, what it measures, where the extra time went, where the baseline comes from, and JIT compilers, including JRuby and TruffleRuby. |
|
data/docs/benchmarks.md
CHANGED
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
This page covers what a benchmark can do beyond timing a block: run
|
|
4
4
|
code around its samples and iterations, share code with other
|
|
5
|
-
benchmarks, choose what it's judged
|
|
6
|
-
For a first benchmark, with `once`,
|
|
7
|
-
the number of iterations, see
|
|
5
|
+
benchmarks, give both versions the same work, choose what it's judged
|
|
6
|
+
on, and benchmark a whole process. For a first benchmark, with `once`,
|
|
7
|
+
`untimed`, and a block that takes the number of iterations, see
|
|
8
8
|
[Write a benchmark](../README.md#write-a-benchmark).
|
|
9
9
|
|
|
10
10
|
## Run code around a benchmark
|
|
@@ -198,6 +198,35 @@ benchmarks: `duel`, `duel_process`, `once`, `untimed`, the hooks
|
|
|
198
198
|
Duel's method would hide a helper of the same name, so the worker
|
|
199
199
|
fails, and names the helper's file and line.
|
|
200
200
|
|
|
201
|
+
## Give both versions the same work
|
|
202
|
+
|
|
203
|
+
Each version's workers load your benchmarks with that version's code,
|
|
204
|
+
so a benchmark compares like with like only where both versions are
|
|
205
|
+
given the same work.
|
|
206
|
+
|
|
207
|
+
Build a benchmark's inputs without the code it measures. A JSON
|
|
208
|
+
parser's benchmark that builds its document with the library's own
|
|
209
|
+
`dump` parses whatever each version dumps: where a change makes `dump`
|
|
210
|
+
write something else, the two versions parse different documents, and
|
|
211
|
+
the comparison measures that, not the parser. Write the input out, read
|
|
212
|
+
it from a file, or build it with code that both versions share, such as
|
|
213
|
+
Ruby's own.
|
|
214
|
+
|
|
215
|
+
```ruby
|
|
216
|
+
# duel/parser_duel.rb
|
|
217
|
+
DOCUMENT = File.read(File.join(__dir__, "fixtures", "document.json"))
|
|
218
|
+
|
|
219
|
+
duel("parse: a document") { MyParser.parse(DOCUMENT) }
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
Every worker loads every file of benchmarks, so a setting that changes
|
|
223
|
+
the whole process, such as a library's global option or a patch that a
|
|
224
|
+
mode installs, changes every benchmark that runs after it, in every
|
|
225
|
+
file. Turn it on in the benchmark's own hooks, and put it back in
|
|
226
|
+
`teardown`, or benchmark it in a process of its own, with
|
|
227
|
+
`duel_process`. Then the benchmarks that don't ask for it measure the
|
|
228
|
+
code that they name.
|
|
229
|
+
|
|
201
230
|
## Choose what a benchmark is judged on
|
|
202
231
|
|
|
203
232
|
By default, Duel judges each benchmark on its timings and its counts:
|