duel 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENT_PROMPT.md +51 -27
- data/CHANGELOG.md +247 -1
- data/README.md +30 -14
- data/docs/benchmarks.md +28 -24
- data/docs/ci.md +328 -362
- data/docs/configuration.md +40 -37
- data/docs/how-it-works.md +23 -10
- data/docs/json.md +21 -25
- data/docs/plugins.md +40 -22
- data/docs/troubleshooting.md +3 -2
- data/lib/duel/across.rb +18 -18
- data/lib/duel/badge.rb +174 -0
- data/lib/duel/baseline.rb +38 -15
- data/lib/duel/cli.rb +175 -107
- data/lib/duel/comment.rb +80 -71
- data/lib/duel/config.rb +145 -63
- data/lib/duel/context.rb +29 -24
- data/lib/duel/forge.rb +87 -0
- data/lib/duel/init.rb +44 -87
- data/lib/duel/json_report.rb +34 -25
- data/lib/duel/lane.rb +107 -61
- data/lib/duel/lock.rb +1 -1
- data/lib/duel/mark_passed.rb +132 -0
- data/lib/duel/metric.rb +28 -27
- data/lib/duel/native.rb +7 -5
- data/lib/duel/platform.rb +4 -0
- data/lib/duel/process_sample.rb +9 -6
- data/lib/duel/rake_task.rb +4 -5
- data/lib/duel/resource.rb +10 -11
- data/lib/duel/result.rb +22 -14
- data/lib/duel/rubies.rb +6 -2
- data/lib/duel/runner.rb +27 -15
- data/lib/duel/spawn.rb +29 -4
- data/lib/duel/stats.rb +14 -12
- data/lib/duel/summary.rb +31 -29
- data/lib/duel/templates/benchmark_duel.rb.erb +2 -2
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +18 -54
- data/lib/duel/templates/duel.rb.erb +7 -7
- data/lib/duel/templates/perf-caller.yml.erb +22 -5
- data/lib/duel/templates/perf-comment.yml.erb +11 -11
- data/lib/duel/templates/perf-push-jobs.yml.erb +127 -0
- data/lib/duel/templates/perf-push.yml.erb +50 -0
- data/lib/duel/templates/perf.yml.erb +113 -335
- data/lib/duel/testing.rb +20 -25
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +16 -9
- data/lib/duel/worker_process.rb +6 -6
- data/lib/duel.rb +88 -33
- data/sig/duel/across.rbs +4 -0
- data/sig/duel/badge.rbs +51 -0
- data/sig/duel/baseline.rbs +7 -2
- data/sig/duel/cli.rbs +31 -20
- data/sig/duel/comment.rbs +14 -15
- data/sig/duel/config.rbs +71 -29
- data/sig/duel/context.rbs +4 -4
- data/sig/duel/forge.rbs +42 -0
- data/sig/duel/init.rbs +8 -26
- data/sig/duel/json_report.rbs +7 -6
- data/sig/duel/lane.rbs +29 -6
- data/sig/duel/mark_passed.rbs +39 -0
- data/sig/duel/metric.rbs +10 -10
- data/sig/duel/native.rbs +3 -0
- data/sig/duel/platform.rbs +1 -0
- data/sig/duel/process_sample.rbs +3 -0
- data/sig/duel/resource.rbs +3 -3
- data/sig/duel/result.rbs +12 -5
- data/sig/duel/rubies.rbs +3 -0
- data/sig/duel/runner.rbs +23 -16
- data/sig/duel/spawn.rbs +7 -0
- data/sig/duel/stats.rbs +4 -1
- data/sig/duel/summary.rbs +8 -0
- data/sig/duel/testing.rbs +2 -2
- data/sig/duel/worker.rbs +6 -5
- data/sig/duel/worker_process.rbs +3 -3
- data/sig/duel.rbs +50 -31
- metadata +9 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4011a6478545bb287113c617c1ffe74165fad6c74043b9208663a555bb4a30a8
|
|
4
|
+
data.tar.gz: a5e698aac1d4042e66aa0c1e5e2b948548ab2bb56b998db81973ac1fa7a21367
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b5ec57f3d533abe47b097d7ebe08594f3615340e72b2ff94ffa0db3cb98501502176555a3da673c93a7219a7ecff63b1f1972c7dc6733977792fc3a044eec102
|
|
7
|
+
data.tar.gz: 8d0b7c39d106e4e3ca32a69c9cb375a7177f0cd48d98055cfbdbf284e3cada2a75c1a489c37dc560beb440d578eafdef1290fe4ece9251d81143eeeae941feb4
|
data/AGENT_PROMPT.md
CHANGED
|
@@ -141,9 +141,10 @@ at the API.
|
|
|
141
141
|
If the project pins its Ruby (`.ruby-version`, `.tool-versions`, or
|
|
142
142
|
`mise.toml`), set `c.ruby = :version_file`, so that a branch that
|
|
143
143
|
upgrades Ruby is compared on each version's own Ruby. `duel init`
|
|
144
|
-
does this
|
|
145
|
-
|
|
146
|
-
file,
|
|
144
|
+
does this for a `.ruby-version`, and has the workflow set up the
|
|
145
|
+
baseline's Ruby too (`version-file: true`). For another version
|
|
146
|
+
file, set it yourself, and see step 6 for the workflow. Duel uses the
|
|
147
|
+
Ruby that runs it
|
|
147
148
|
when that's the one asked for, so CI needs nothing more until a
|
|
148
149
|
branch changes the Ruby.
|
|
149
150
|
|
|
@@ -191,34 +192,40 @@ at the API.
|
|
|
191
192
|
your names can be picked out. Without `--force`, a tree with
|
|
192
193
|
nothing changed since the main branch has nothing to compare.
|
|
193
194
|
|
|
194
|
-
6. **Check CI.** `duel init` wrote two GitHub Actions workflows
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
runs
|
|
200
|
-
|
|
195
|
+
6. **Check CI.** `duel init` wrote two GitHub Actions workflows, short
|
|
196
|
+
ones that call Duel's reusable workflows, at the tag of the Duel
|
|
197
|
+
release that wrote them, with what `.duel.rb` sets that they need.
|
|
198
|
+
- `.github/workflows/perf.yml` runs Duel. Its `duel` job runs the
|
|
199
|
+
branch's code, so it only reads (`contents: read`), and its `push`
|
|
200
|
+
job, which runs only for a push and none of the code, writes
|
|
201
|
+
(`contents: write`). Keep the permissions as `duel init` wrote
|
|
202
|
+
them: GitHub checks every job of Duel's workflows against them,
|
|
203
|
+
whether it runs or not.
|
|
201
204
|
- `.github/workflows/perf-comment.yml` comments on a pull request
|
|
202
205
|
with Duel's report once the first has run (`c.comment` in
|
|
203
206
|
`.duel.rb` chooses when, and Duel also adds the report to the
|
|
204
207
|
job's summary). It runs none of the pull request's code. Keep it as
|
|
205
208
|
it is.
|
|
206
|
-
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
:pull_requests` or `c.push_base = :last_passed`, and
|
|
210
|
-
job only with `--badge`. Each
|
|
211
|
-
set those settings, or
|
|
209
|
+
- The workflow's jobs for pushes (a comment on a pushed commit,
|
|
210
|
+
and a `duel/passed/` branch that marks the last commit Duel
|
|
211
|
+
passed) run only where `.duel.rb` sets `c.fail_on =
|
|
212
|
+
:pull_requests` or `c.push_base = :last_passed`, and `duel init`
|
|
213
|
+
writes the badge's job only with `--badge`. Each writes to the
|
|
214
|
+
repository, so don't set those settings, or ask for the badge,
|
|
215
|
+
unless I ask.
|
|
212
216
|
|
|
213
217
|
Check that `ruby-version` is one the project supports, unless it has
|
|
214
|
-
a `.ruby-version`.
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
the
|
|
221
|
-
the
|
|
218
|
+
a `.ruby-version`. Give the project's other CI setup that its tests
|
|
219
|
+
need (system packages, services) to the call as `setup:`, shell
|
|
220
|
+
commands that run before Ruby is set up (Duel's `docs/ci.md` lists
|
|
221
|
+
the inputs). Where that isn't enough, or with `c.ruby =
|
|
222
|
+
:version_file` and another version file than `.ruby-version`, delete
|
|
223
|
+
the two workflows and run `bundle exec duel init --standalone`,
|
|
224
|
+
which writes the whole ones (and leaves the files that are there).
|
|
225
|
+
Add the setup to its Duel job, and for the version file, the steps in
|
|
226
|
+
Duel's `docs/ci.md` that install the baseline's Ruby, reading that
|
|
227
|
+
file instead. Keep the scheduled `last-release` run only if the
|
|
228
|
+
project tags its releases (`v*` by default, or else set
|
|
222
229
|
`c.release_tags`), or leave the schedule out. Until the first
|
|
223
230
|
release is tagged, that run says there's nothing to compare against,
|
|
224
231
|
and passes. Once it's pushed, run the workflow by hand: it compares
|
|
@@ -227,10 +234,14 @@ at the API.
|
|
|
227
234
|
|
|
228
235
|
7. **Show the badge.** If the project's README already shows badges (CI
|
|
229
236
|
status, gem version), add Duel's to that row, with exactly this
|
|
230
|
-
markdown
|
|
237
|
+
markdown, `OWNER/REPO` the project's repository, so that it links to
|
|
238
|
+
the Perf workflow's runs (for a project in a subdirectory of its
|
|
239
|
+
repository, `perf-<its path>.yml`, the workflow `duel init` wrote, in
|
|
240
|
+
place of `perf.yml`; on GitLab CI, the link is to
|
|
241
|
+
`https://gitlab.com/OWNER/REPO/-/pipelines`):
|
|
231
242
|
|
|
232
243
|
```markdown
|
|
233
|
-
[](https://github.com/
|
|
244
|
+
[](https://github.com/OWNER/REPO/actions/workflows/perf.yml)
|
|
234
245
|
```
|
|
235
246
|
|
|
236
247
|
If the README has no badges, don't add a row just for Duel's. This
|
|
@@ -249,4 +260,17 @@ at the API.
|
|
|
249
260
|
(`c.accept_label`, `duel: accept` in the `.duel.rb` that
|
|
250
261
|
`duel init` writes), which the repository needs to have.
|
|
251
262
|
|
|
252
|
-
|
|
263
|
+
9. **Add the project to Duel's list.** If the project's repository is
|
|
264
|
+
public, open a pull request in Duel's repository
|
|
265
|
+
(https://github.com/sferik/duel-ruby) that adds the project to
|
|
266
|
+
`USERS.md`, the list of projects using Duel. Fork Duel's repository
|
|
267
|
+
(`gh repo fork sferik/duel-ruby --clone`), add one line for the
|
|
268
|
+
project to the list, in alphabetical order and in the format of the
|
|
269
|
+
others (a link to its repository, named by its owner and name, and
|
|
270
|
+
what the project is, in a few words), commit it on a branch of its
|
|
271
|
+
own, and open the pull request (`gh pr create --repo
|
|
272
|
+
sferik/duel-ruby`). Leave out a private repository. Tell me the pull
|
|
273
|
+
request's link.
|
|
274
|
+
|
|
275
|
+
Apart from that pull request, don't commit. Leave the changes for me to
|
|
276
|
+
review.
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,251 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.3.0] - 2026-11-20
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- `c.release_tags` and `c.benchmarks` take a list of globs too:
|
|
15
|
+
`c.release_tags = %w[v* my_gem-v*]` for a gem that renamed its tags
|
|
16
|
+
(a tag that matches any of them is a release), and `c.benchmarks =
|
|
17
|
+
%w[perf/**/*_duel.rb test/perf/*_duel.rb]` for benchmarks in two
|
|
18
|
+
places (each glob's directory is watched). One glob works as before.
|
|
19
|
+
- `Duel.find_program` finds a plugin's server program, as
|
|
20
|
+
duel-postgresql, duel-mysql, duel-redis and duel-memcached now all
|
|
21
|
+
do: a path (one with a `/` or a `\` in it, `~` expanded) as it is,
|
|
22
|
+
or a name in the first of the `PATH`'s directories that has it,
|
|
23
|
+
either one ending with `.exe` on Windows too. `Duel.path?` says which
|
|
24
|
+
it takes a server for, for the plugin's error that it isn't there.
|
|
25
|
+
- Duel's reusable Perf workflow runs on Windows (`runs-on:
|
|
26
|
+
windows-latest`): the Duel job's steps run in bash on every runner (on
|
|
27
|
+
Windows, not PowerShell), and paths it hands Ruby are Windows's there.
|
|
28
|
+
A new input, `setup-after-ruby`, runs shell commands once Ruby is set
|
|
29
|
+
up, before the bundle is installed, for what needs Ruby first: on
|
|
30
|
+
Windows, a package of Ruby's devkit, from `pacman`, that a gem's C
|
|
31
|
+
extension links. And `c.ruby = :version_file` finds a Ruby on
|
|
32
|
+
Windows, whose program is `ruby.exe` (or `jruby.exe`).
|
|
33
|
+
- Duel fails a run where a worker's environment turns on one of CRuby's
|
|
34
|
+
JITs (`RUBY_YJIT_ENABLE` or `RUBY_ZJIT_ENABLE`, as `1`, `true` or
|
|
35
|
+
`yes`) that the worker doesn't run, saying which version's workers,
|
|
36
|
+
which JIT, and on what Ruby. A Ruby without the JIT ignores its
|
|
37
|
+
variable, so `jit: zjit` on Ruby 3.4, or `jit: yjit` on JRuby,
|
|
38
|
+
measured without it, and nothing said so.
|
|
39
|
+
- ZJIT, the JIT compiler that's experimental in Ruby 4.0 and is due to
|
|
40
|
+
ship in Ruby 4.1, is taken for a JIT as YJIT is: where a worker has
|
|
41
|
+
it on (`RUBY_ZJIT_ENABLE=1`, or turned on as the app loads), Duel
|
|
42
|
+
runs each benchmark until it's steady before it measures it, for up
|
|
43
|
+
to ten seconds (or as `--warm` says). It measured from the first
|
|
44
|
+
sample before, while ZJIT was still compiling. Duel's reusable Perf
|
|
45
|
+
workflow takes the JIT to turn on as one input, `jit:` (`none`,
|
|
46
|
+
`yjit` or `zjit`, as Ruby runs one at most), in place of its `yjit:
|
|
47
|
+
true`.
|
|
48
|
+
- `duel comment --commit SHA` comments on a pushed commit, as Duel's
|
|
49
|
+
outputs say (`DUEL_COMMENT` and `DUEL_STATUS`), on GitHub or GitLab.
|
|
50
|
+
`duel mark-passed BRANCH` moves the branch that marks the last commit
|
|
51
|
+
Duel passed forward to a push, or makes it, and `duel badge BRANCH`
|
|
52
|
+
writes Duel's badge to its branch, on GitHub. The Comment on the
|
|
53
|
+
commit, Mark the commit passed and Update the badge jobs that
|
|
54
|
+
`duel init` writes (and Duel's reusable Perf workflow) run them,
|
|
55
|
+
installed from RubyGems as the comment workflow's `duel comment` is,
|
|
56
|
+
in place of the shell, `curl` and `jq` that they ran. So a fix to one
|
|
57
|
+
comes with a release of Duel, without a change to the workflow.
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
|
|
61
|
+
- A plugin's resource gives its server a port with `w.free_port`, in
|
|
62
|
+
place of `w.port`, which read as the resource's one port, though each
|
|
63
|
+
call is another: `port = w.free_port`, then that `port` for both the
|
|
64
|
+
server's option and its URL.
|
|
65
|
+
- `duel_process` takes the Ruby code it runs as `code:`, in place of
|
|
66
|
+
`ruby:`, which read as the Ruby to run it on, as `c.ruby`, `--ruby`
|
|
67
|
+
and `DUEL_RUBY` mean: `duel_process "boot: require the library",
|
|
68
|
+
code: %(require "my_gem")`.
|
|
69
|
+
- The settings that take a glob, a path, a name or a lambda
|
|
70
|
+
(`c.benchmarks`, `c.watch`, `c.release_tags`, `c.library`, `c.build`,
|
|
71
|
+
`c.main_branch` and `c.ruby`) fail as they're set, on a value they
|
|
72
|
+
don't take, saying what to give (`c.watch = :lib: give a path, or a
|
|
73
|
+
list of them`), as `c.comment` and the others do. A wrong one failed
|
|
74
|
+
deep in a run (`String#+: no implicit conversion of Array into
|
|
75
|
+
String`), or not at all. And `c.watch` takes one path too
|
|
76
|
+
(`c.watch = "app"`), as `c.benchmarks` and `c.release_tags` take one
|
|
77
|
+
glob.
|
|
78
|
+
- A metric judged like CPU time is `kind: :timed`, in place of
|
|
79
|
+
`kind: :ratio`, as `judge: :timed`, `Definition#timed?` and the
|
|
80
|
+
report's `timed` call what it measures: a metric's `measures` say
|
|
81
|
+
`{"wait" => {kind: :timed, unit: "µs"}}`. (The JSON report's `ratio`,
|
|
82
|
+
a timing's, is as it was.)
|
|
83
|
+
- A ratio metric's `sizes_samples: false` is `size_samples: false`
|
|
84
|
+
(don't size samples by it), as `calibrate: false` and `skip: true`
|
|
85
|
+
say what to do.
|
|
86
|
+
- A counted metric's slack is `per_iteration:` (how much more an
|
|
87
|
+
iteration still isn't more) and `per_sample:` (how much more a
|
|
88
|
+
sample, whatever its length), in place of `tolerance:` and
|
|
89
|
+
`allowance:`, whose names didn't say which was which: in a metric's
|
|
90
|
+
`measures`, `c.measure("retained_objects", per_sample: 32)`, and
|
|
91
|
+
`Duel::Metric.within?(more, n, per_iteration:, per_sample:)`.
|
|
92
|
+
- A plugin's resource fails with `w.fail!(message)`, in place of
|
|
93
|
+
`w.failed(message)`, which read like a question but raised
|
|
94
|
+
`Duel::Failed`: `Duel.find_program(server) || w.fail!("no #{server}")`.
|
|
95
|
+
- The RBS signatures of `c.comment=`, `c.fail_on=`, `c.push_base=` and
|
|
96
|
+
`c.accept_label=` take what each takes (`c.comment = :failure`, a
|
|
97
|
+
typo for `:failures`, fails Steep), as the other settings' do. They
|
|
98
|
+
took anything, and said nothing of what they return.
|
|
99
|
+
- The jobs of Duel's reusable Perf workflow for a push, which write to
|
|
100
|
+
the repository (Comment on the commit, Mark the commit passed and
|
|
101
|
+
Update the badge), are a reusable workflow of their own,
|
|
102
|
+
`.github/workflows/duel-push.yml`, so a Perf workflow that calls
|
|
103
|
+
Duel's grants `contents: read` to the Duel job's call, which runs the
|
|
104
|
+
branch's code, and `contents: write` to the push workflow's alone,
|
|
105
|
+
which it calls only for a push. Before, its one call granted
|
|
106
|
+
`contents: write`, for a pull request's run too, as GitHub checks
|
|
107
|
+
every job of a called workflow against what the caller grants,
|
|
108
|
+
whether it runs or not. `duel init` writes both calls, the second
|
|
109
|
+
passing on what the Duel job said (`comment`, `status`, `passed`, and
|
|
110
|
+
`result`, how its call ended). The `badge` input is the push
|
|
111
|
+
workflow's, not the Perf workflow's: a Perf workflow that passes it
|
|
112
|
+
to `duel.yml` fails, so run `duel init --force` (or
|
|
113
|
+
`duel init --check` to see the change).
|
|
114
|
+
- The jobs of Duel's workflows that write to the repository or comment
|
|
115
|
+
(Comment on the commit, Mark the commit passed, Update the badge, and
|
|
116
|
+
the Perf comment workflow's), reusable or as `duel init` writes them,
|
|
117
|
+
install exactly the release of Duel that wrote them
|
|
118
|
+
(`gem install duel --version '0.3.0'`), not the newest of its minor
|
|
119
|
+
version (`~> 0.2.0`), and every action Duel's workflows use is named
|
|
120
|
+
by its commit, with its release beside it
|
|
121
|
+
(`ruby/setup-ruby@f9c4492… # v1.342.0`), as a tag can be moved.
|
|
122
|
+
`docs/ci.md` shows how to name Duel's reusable workflows by commit
|
|
123
|
+
too.
|
|
124
|
+
- `duel init` writes short workflows that call Duel's reusable ones, at
|
|
125
|
+
the tag of the release that writes them, as `--reusable` did, which
|
|
126
|
+
is gone. `--standalone` writes the whole workflows, as `duel init`
|
|
127
|
+
did, to change as you like. `--check` compares the ones it would
|
|
128
|
+
write with the same options, so `duel init --check --standalone`
|
|
129
|
+
checks whole workflows that an earlier `duel init` wrote.
|
|
130
|
+
- The badge links to the project's Perf workflow's runs
|
|
131
|
+
(`https://github.com/OWNER/REPO/actions/workflows/perf.yml`, or for a
|
|
132
|
+
project in a subdirectory, `perf-<its path>.yml`), which say why it's
|
|
133
|
+
green or red, not to the repository itself: in the README, the guide,
|
|
134
|
+
AGENT_PROMPT.md's badge step, and the workflows that `duel init` and
|
|
135
|
+
`rails generate duel:install` write, whose comments give the badge's
|
|
136
|
+
Markdown. USERS.md, in Duel's repository, lists the projects using
|
|
137
|
+
Duel: a pull request adds one, and AGENT_PROMPT.md's last step has the
|
|
138
|
+
agent open one for the project it set Duel up in, if that's public.
|
|
139
|
+
- `Duel::Testing.measure`'s block is one iteration, which it runs `n:`
|
|
140
|
+
times, as a worker runs a benchmark's block that takes no `n`. It ran
|
|
141
|
+
the block once, and only told the metrics `n`, unless there was an
|
|
142
|
+
`around_iteration` hook (the configuration's, or its own), so a
|
|
143
|
+
plugin that added one changed how many times every test's block ran.
|
|
144
|
+
A test that did its `n` iterations in the block does one.
|
|
145
|
+
- A `Duel::RakeTask` exits with `duel`'s own status. It exited 1 for
|
|
146
|
+
status 2 (couldn't compare), and passed for status 3 (slower, but only
|
|
147
|
+
warned, which GitLab's `allow_failure: exit_codes: [3]` relies on), so
|
|
148
|
+
`rake duel` hid a slowdown that `duel` reported.
|
|
149
|
+
- Each of a run's settings can be set the same three ways: an option, a
|
|
150
|
+
variable, or `.duel.rb`, in that order of precedence. `c.jobs` sets
|
|
151
|
+
`--jobs`, and each option `.duel.rb` can set, and `--ruby`, has a
|
|
152
|
+
variable, `DUEL_` and its name (`DUEL_LIMIT`, `DUEL_TIME_LIMIT`,
|
|
153
|
+
`DUEL_RUBY`...), as `--base`, `--filter` and `--force` had.
|
|
154
|
+
`--ruby` takes `version_file` too.
|
|
155
|
+
- `c.library` takes a lambda, given the root of a version's tree, that
|
|
156
|
+
loads the version instead, which `c.load { |root| ... }` did:
|
|
157
|
+
`c.library = ->(root) { require File.join(root, "config", "boot") }`.
|
|
158
|
+
`c.load`, which hid `Kernel#load`, is gone.
|
|
159
|
+
- What a benchmark is judged on is one option, `judge:`, one or a list of
|
|
160
|
+
`:timed`, `:counted` and `:peak_memory`, named as a result's measures
|
|
161
|
+
are, and `[:timed, :counted]` by default. It replaces `only: :count`
|
|
162
|
+
(`judge: :counted`), `only: :timing` (`judge: :timed`) and
|
|
163
|
+
`peak_memory: true` (`:peak_memory` too, which needs `:timed`: with
|
|
164
|
+
counts alone, it was quietly ignored), so `only: :timing, peak_memory:
|
|
165
|
+
true` reads as `judge: %i[timed peak_memory]`. A metric sees it as
|
|
166
|
+
`sample.benchmark.judge`, with `timed?`, `counted?` and
|
|
167
|
+
`peak_memory?`, in place of `only` and `peak_memory`, and
|
|
168
|
+
`Duel::Testing.measure` and `Duel::Testing.sample` take `judge:` too.
|
|
169
|
+
- The JSON report's schema is 2:
|
|
170
|
+
- `status` is what the comparison found, and `exit` is `duel`'s exit
|
|
171
|
+
status, with `passed_anyway` saying why a run that failed passed
|
|
172
|
+
anyway (`"accept_label"`, or `"fail_on"`), which nothing said
|
|
173
|
+
before: a regression a label accepted was `"passed"`.
|
|
174
|
+
- A timing too close to its limit to settle is `"unsettled"`, as its
|
|
175
|
+
benchmark's verdict is, not `"undecided"`, in the JSON report, the
|
|
176
|
+
Markdown report and the terminal's.
|
|
177
|
+
- A count, and peak memory, has a `verdict` (`"more"`, `"fewer"` or
|
|
178
|
+
`"same"`) in place of `worse`.
|
|
179
|
+
- `unsettled` (a benchmark's `verdict` says so), `errors` (each
|
|
180
|
+
benchmark's `error` says so) and `comment` (the workflow's step
|
|
181
|
+
output says so) are gone.
|
|
182
|
+
- On CI, Duel finds the baseline itself, from the run's event: for a
|
|
183
|
+
push, the commit before it (on GitHub Actions, the event's `before`,
|
|
184
|
+
and on GitLab CI, `CI_COMMIT_BEFORE_SHA`), for a merge queue, its
|
|
185
|
+
base, and on a schedule, the last release. The workflows `duel init`
|
|
186
|
+
writes gave it as `DUEL_BASE`, which now only names another, as
|
|
187
|
+
`--base` does. (A `DUEL_BASE` that's the event's own is still taken
|
|
188
|
+
for it, as the workflows earlier versions wrote give it.) And `duel
|
|
189
|
+
base` prints the commit a run compares with, which the workflow's
|
|
190
|
+
step that installs the baseline's Ruby reads, once the project's Ruby
|
|
191
|
+
and bundle are set up, so it no longer finds the main branch and the
|
|
192
|
+
last release in shell of its own, which `c.main_branch` and
|
|
193
|
+
`c.release_tags` had to be copied into.
|
|
194
|
+
- The Perf workflow follows `.duel.rb`, rather than copying it. Its
|
|
195
|
+
jobs for pushes, the comment on a pushed commit and marking a push
|
|
196
|
+
passed, are always there (`duel init` wrote them only where `.duel.rb`
|
|
197
|
+
asked, and said to write the workflow again if that changed), and run
|
|
198
|
+
where Duel's outputs say to: its `passed` output is the branch that
|
|
199
|
+
marks the push passed (`duel/passed/main`), not `true`, so the job
|
|
200
|
+
moves the one Duel reads. Duel's reusable Perf workflow takes none of
|
|
201
|
+
`.duel.rb`'s settings: `accept-label`, `release-tags`,
|
|
202
|
+
`comment-on-pushes`, `mark-passed`, `passed-branch`, `badge-branch`
|
|
203
|
+
and `marker` are gone, as Duel reads `.duel.rb`, and the names it
|
|
204
|
+
needs come from `working-directory`. And on a schedule, Duel finds
|
|
205
|
+
the last change a label accepted since the last release itself, with
|
|
206
|
+
`gh` (`DUEL_ACCEPTED` still names one), which the workflow's labels
|
|
207
|
+
step did, with `c.accept_label` and `c.release_tags` copied into it.
|
|
208
|
+
- `--comment` and `--fail-on` are gone: when the workflow comments, and
|
|
209
|
+
which runs a regression fails, are `.duel.rb`'s to say (`c.comment`,
|
|
210
|
+
`c.fail_on`), as the workflow passes neither, and outside CI, they
|
|
211
|
+
changed nothing.
|
|
212
|
+
- `duel` fails on an argument that isn't an option (`duel v1.2.0`, say,
|
|
213
|
+
for `duel --base v1.2.0`), which it ignored.
|
|
214
|
+
- `duel clean` removes `tmp/duel` and the baseline's git worktree, as
|
|
215
|
+
`duel --clean` did, which took (and ignored) every other option, and
|
|
216
|
+
is gone. It takes `--config`.
|
|
217
|
+
- `duel comment` comments on a GitLab merge request where `GITLAB_CI`
|
|
218
|
+
says it runs on GitLab CI, without `--gitlab`, which the GitLab CI
|
|
219
|
+
jobs `duel init --gitlab` writes no longer pass. (It's still taken,
|
|
220
|
+
and does nothing, for the jobs that earlier versions wrote.)
|
|
221
|
+
- `--candidate-dir` is `--candidate`, as `--base` names the baseline,
|
|
222
|
+
and `DUEL_CANDIDATE` sets it.
|
|
223
|
+
- `DUEL_FORCE` is `1` or `true` (or `0` or `false`, for off), and
|
|
224
|
+
anything else fails. Any value forced before, `DUEL_FORCE=0` too.
|
|
225
|
+
- A `setup`, `teardown`, `before_sample` or `after_iterations` hook
|
|
226
|
+
takes no parameter, at any level, and one that takes one fails before
|
|
227
|
+
anything runs. The first three were given the benchmark if they took
|
|
228
|
+
a parameter, which nothing needed. Only `around_iteration` takes one:
|
|
229
|
+
its iteration.
|
|
230
|
+
- `c.accept_label` is `"duel: accept"` by default, the label that the
|
|
231
|
+
`.duel.rb` `duel init` writes named, so a project set up by hand
|
|
232
|
+
accepts a regression as one set up by `duel init` does. Set it to
|
|
233
|
+
`nil` for no such label. That `.duel.rb` shows the label, and the
|
|
234
|
+
build of a project with `ext/` (`bundle exec rake compile`, Duel's
|
|
235
|
+
default there already), commented out, as the defaults they are.
|
|
236
|
+
|
|
237
|
+
### Fixed
|
|
238
|
+
|
|
239
|
+
- A process benchmark whose Ruby process of its own ended while it ran
|
|
240
|
+
a command (killed, say) leaves that command running no longer: what's
|
|
241
|
+
left of the Ruby process's group is killed as it's stopped, where the
|
|
242
|
+
command went on, orphaned, for as long as it ran (for ever, for one
|
|
243
|
+
that hung).
|
|
244
|
+
- The Perf workflow that `duel init --reusable` wrote granted
|
|
245
|
+
`contents: read` unless it turned on a job that writes (marking a
|
|
246
|
+
push passed, commenting on one, or the badge), but GitHub checks every
|
|
247
|
+
job of a workflow it calls against what the caller grants, whether
|
|
248
|
+
the job runs or not, so the run failed before it started. The jobs
|
|
249
|
+
that write are now Duel's push workflow's, which the Perf workflow
|
|
250
|
+
that `duel init` writes calls apart, with `contents: write`, and only
|
|
251
|
+
for a push, while the Duel job's call still only reads (see Changed).
|
|
252
|
+
Run `duel init --force` for a workflow that calls both
|
|
253
|
+
(`duel init --check` shows the change).
|
|
254
|
+
|
|
10
255
|
## [0.2.0] - 2026-11-16
|
|
11
256
|
|
|
12
257
|
### Added
|
|
@@ -906,7 +1151,8 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
906
1151
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
907
1152
|
release.
|
|
908
1153
|
|
|
909
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.
|
|
1154
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.3.0...HEAD
|
|
1155
|
+
[0.3.0]: https://github.com/sferik/duel-ruby/compare/v0.2.0...v0.3.0
|
|
910
1156
|
[0.2.0]: https://github.com/sferik/duel-ruby/compare/v0.1.8...v0.2.0
|
|
911
1157
|
[0.1.8]: https://github.com/sferik/duel-ruby/compare/v0.1.7...v0.1.8
|
|
912
1158
|
[0.1.7]: https://github.com/sferik/duel-ruby/compare/v0.1.6...v0.1.7
|
data/README.md
CHANGED
|
@@ -109,7 +109,7 @@ one on Linux or macOS.
|
|
|
109
109
|
|---|---|
|
|
110
110
|
| `.duel.rb` | Duel's configuration. It builds a project that has an `ext/` directory with `rake compile`, and runs each version on its own Ruby when the project has a `.ruby-version`. Without a gemspec to name the library, it names the only file in `lib/`, or leaves the library for you to set. See [Configure Duel](docs/configuration.md). |
|
|
111
111
|
| `duel/<library>_duel.rb` | An example benchmark to replace with your own, with advice on writing benchmarks that tell a regression from noise. See [Write a benchmark](#write-a-benchmark). |
|
|
112
|
-
| `.github/workflows/perf.yml` |
|
|
112
|
+
| `.github/workflows/perf.yml` | A short GitHub Actions workflow that calls Duel's reusable one (see [Call Duel's reusable workflows](docs/ci.md#call-duels-reusable-workflows)), with `perf-comment.yml`, which comments on a pull request. With `--standalone`, they're the whole workflows in [Run Duel in CI](docs/ci.md#catch-slow-creep). For a project in a subdirectory of its repository, the workflow is at the repository's root, named after the subdirectory, and runs in it. |
|
|
113
113
|
| `Rakefile` | If there is one, `duel init` adds Duel's Rake task to it, as `rake duel`. See [Run Duel in CI](#run-duel-in-ci). |
|
|
114
114
|
| `Gemfile` | Unless it mentions StackProf already, `duel init` adds `gem "stackprof", platforms: :mri` to its `development` group (or, without one, to its end), so Duel can show where a regression spent its extra time. StackProf runs on CRuby only, so the gem isn't installed on JRuby or TruffleRuby. |
|
|
115
115
|
|
|
@@ -118,9 +118,9 @@ one on Linux or macOS.
|
|
|
118
118
|
use `--dry-run`. For a badge in your README that turns red when Duel
|
|
119
119
|
fails a push, use `--badge` (see
|
|
120
120
|
[Show that a project is guarded](#show-that-a-project-is-guarded)).
|
|
121
|
-
To write
|
|
122
|
-
|
|
123
|
-
[
|
|
121
|
+
To write the whole workflows, to change as you like, rather than short
|
|
122
|
+
ones that call Duel's reusable workflows, use `--standalone` (see
|
|
123
|
+
[Run Duel in CI](docs/ci.md)).
|
|
124
124
|
When it's done, `duel init` lists the next steps.
|
|
125
125
|
|
|
126
126
|
For a Rails app, use
|
|
@@ -195,6 +195,7 @@ duel --base v1.4.0 # against any git ref
|
|
|
195
195
|
duel --base last-release # against the newest release tag before HEAD
|
|
196
196
|
duel --filter "^parse:" # only benchmarks whose names match (a regular expression)
|
|
197
197
|
duel --base ../old # against a tree on disk, already built
|
|
198
|
+
duel --candidate ../new # a tree on disk, already built, against the baseline
|
|
198
199
|
```
|
|
199
200
|
|
|
200
201
|
To see every option, such as `--processes`, `--timeout`, and `--limit`,
|
|
@@ -210,16 +211,23 @@ second run fails and names the process that's running the first. A run
|
|
|
210
211
|
that crashes leaves nothing behind that blocks the next run.
|
|
211
212
|
|
|
212
213
|
Where it's hard to pass options, such as behind a Rake task or in a CI
|
|
213
|
-
expression, set
|
|
214
|
+
expression, set environment variables instead. Each is `DUEL_` and the
|
|
215
|
+
option's name, for `--base`, `--candidate`, `--filter`, `--ruby`, and
|
|
216
|
+
each option that `.duel.rb` can set too (see
|
|
217
|
+
[Set a run's settings](docs/configuration.md#set-a-runs-settings)):
|
|
214
218
|
|
|
215
219
|
| Variable | Option |
|
|
216
220
|
|---|---|
|
|
217
221
|
| `DUEL_BASE` | `--base` |
|
|
222
|
+
| `DUEL_CANDIDATE` | `--candidate` |
|
|
218
223
|
| `DUEL_FILTER` | `--filter` |
|
|
219
|
-
| `
|
|
224
|
+
| `DUEL_RUBY` | `--ruby` |
|
|
225
|
+
| `DUEL_LIMIT`, `DUEL_CONFIDENCE`, `DUEL_JOBS`, `DUEL_PROCESSES`, `DUEL_TIMEOUT`, `DUEL_TIME_LIMIT`, `DUEL_WARM` | `--limit`, `--confidence`, `--jobs`, `--processes`, `--timeout`, `--time-limit`, `--warm` |
|
|
226
|
+
| `DUEL_FORCE=1` (or `true`) | `--force` (`0` or `false` is off, and anything else fails) |
|
|
220
227
|
|
|
221
|
-
An option on the command line takes precedence over its variable,
|
|
222
|
-
an empty variable counts as
|
|
228
|
+
An option on the command line takes precedence over its variable,
|
|
229
|
+
which takes precedence over `.duel.rb`, and an empty variable counts as
|
|
230
|
+
unset.
|
|
223
231
|
|
|
224
232
|
For where the baseline comes from, how Duel checks it out and builds it,
|
|
225
233
|
and how it compares a project in a subdirectory of its repository, see
|
|
@@ -269,7 +277,8 @@ merge request with the branch it targets and comment on it. See
|
|
|
269
277
|
[Run Duel on GitLab CI](docs/ci.md#run-duel-on-gitlab-ci).
|
|
270
278
|
|
|
271
279
|
To run Duel from Rake, define a task with `Duel::RakeTask`. Any
|
|
272
|
-
arguments after the task's name are passed to `duel
|
|
280
|
+
arguments after the task's name are passed to `duel`, and the task
|
|
281
|
+
exits with `duel`'s status:
|
|
273
282
|
|
|
274
283
|
```ruby
|
|
275
284
|
require "duel/rake_task"
|
|
@@ -380,20 +389,27 @@ says.
|
|
|
380
389
|
To show that Duel guards your project, add its badge to your README,
|
|
381
390
|
next to your other badges. With the job that `duel init --badge` adds to
|
|
382
391
|
its workflow, the badge turns red when Duel fails a push to your default
|
|
383
|
-
branch. Replace `OWNER/REPO`
|
|
392
|
+
branch, and links to your Perf workflow's runs. Replace `OWNER/REPO`
|
|
393
|
+
with your repository's:
|
|
384
394
|
|
|
385
395
|
```markdown
|
|
386
|
-
[](https://github.com/
|
|
396
|
+
[](https://github.com/OWNER/REPO/actions/workflows/perf.yml)
|
|
387
397
|
```
|
|
388
398
|
|
|
389
399
|
See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
|
|
390
|
-
for how the job keeps it. Without the job,
|
|
391
|
-
|
|
400
|
+
for how the job keeps it. Without the job, use Duel's own badge, which
|
|
401
|
+
is always green, linked to your Perf workflow's runs too:
|
|
392
402
|
|
|
393
403
|
```markdown
|
|
394
|
-
[](https://github.com/
|
|
404
|
+
[](https://github.com/OWNER/REPO/actions/workflows/perf.yml)
|
|
395
405
|
```
|
|
396
406
|
|
|
407
|
+
For a project in a subdirectory of its repository, the workflow is
|
|
408
|
+
`perf-<its path>.yml` (`perf-gems-parser.yml`, say). On GitLab CI, link
|
|
409
|
+
to its pipelines (`https://gitlab.com/OWNER/REPO/-/pipelines`). Projects
|
|
410
|
+
using Duel are listed in [USERS.md](USERS.md): add yours with a pull
|
|
411
|
+
request.
|
|
412
|
+
|
|
397
413
|
## Plugins
|
|
398
414
|
|
|
399
415
|
A plugin packages what a server or a library needs, such as a server for
|
data/docs/benchmarks.md
CHANGED
|
@@ -47,9 +47,9 @@ benchmark's.
|
|
|
47
47
|
|
|
48
48
|
A file's hooks and a benchmark's own run in the benchmark's context, so
|
|
49
49
|
they share its instance variables. Hooks in `.duel.rb` run where they're
|
|
50
|
-
written, which suits a plugin's hooks.
|
|
51
|
-
|
|
52
|
-
|
|
50
|
+
written, which suits a plugin's hooks. Only `around_iteration` takes
|
|
51
|
+
a parameter, its iteration: a hook of any other kind that takes one
|
|
52
|
+
fails, before anything runs.
|
|
53
53
|
|
|
54
54
|
Inside a benchmark's own block, `once { ... }` runs its block the first
|
|
55
55
|
time that the benchmark runs, and isn't timed, as in
|
|
@@ -82,7 +82,7 @@ a database's tables, give the hook in `.duel.rb`:
|
|
|
82
82
|
|
|
83
83
|
```ruby
|
|
84
84
|
Duel.configure do |c|
|
|
85
|
-
c.before_sample {
|
|
85
|
+
c.before_sample { MyDatabase.truncate_all } # not measured
|
|
86
86
|
end
|
|
87
87
|
```
|
|
88
88
|
|
|
@@ -235,24 +235,28 @@ By default, Duel judges each benchmark on its timings and its counts:
|
|
|
235
235
|
own.
|
|
236
236
|
- Counts: allocations, and any other exact counts.
|
|
237
237
|
|
|
238
|
-
To change that,
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
benchmark thread's.
|
|
247
|
-
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
238
|
+
To change that, give `duel` what to judge, `judge:`, one of these or a
|
|
239
|
+
list of them (`judge: %i[timed counted]` by default):
|
|
240
|
+
|
|
241
|
+
- `:counted`: its counts. Alone, for work that's mostly waiting, too
|
|
242
|
+
slow and light to time, Duel judges the benchmark on its counts over a
|
|
243
|
+
few iterations.
|
|
244
|
+
- `:timed`: its timings. Alone, for work done by threads that the
|
|
245
|
+
benchmark starts, which make its counts vary from run to run, Duel
|
|
246
|
+
takes the process's CPU time instead of the benchmark thread's.
|
|
247
|
+
- `:peak_memory`: its peak memory too, with its timings: the most
|
|
248
|
+
resident memory that its measured work reached, above where it
|
|
249
|
+
started, or on JRuby, the most of the JVM's heap that it had in use.
|
|
250
|
+
Duel takes each version's lowest peak from three fresh workers. Like
|
|
251
|
+
every other measure, peak memory leaves out what `once` and `untimed`
|
|
252
|
+
blocks leave behind.
|
|
253
253
|
|
|
254
254
|
```ruby
|
|
255
|
-
duel "
|
|
255
|
+
duel "redis: PING", judge: :counted do
|
|
256
|
+
# ...
|
|
257
|
+
end
|
|
258
|
+
|
|
259
|
+
duel "server: 10 workers", judge: %i[timed peak_memory] do |n|
|
|
256
260
|
# ...
|
|
257
261
|
end
|
|
258
262
|
```
|
|
@@ -263,12 +267,12 @@ To guard how long your app takes to boot, or your library to load, use
|
|
|
263
267
|
`duel_process`. Each sample starts fresh processes for each version,
|
|
264
268
|
in that version's directory and with its environment, and the two
|
|
265
269
|
versions take turns, as they do for every other benchmark. Give
|
|
266
|
-
`duel_process` a command, or Ruby code to run with `
|
|
270
|
+
`duel_process` a command, or Ruby code to run with `code:`:
|
|
267
271
|
|
|
268
272
|
```ruby
|
|
269
273
|
# duel/boot_duel.rb
|
|
270
274
|
duel_process "boot: the app", "bin/rails", "runner", "1"
|
|
271
|
-
duel_process "boot: require the library",
|
|
275
|
+
duel_process "boot: require the library", code: %(require "my_gem")
|
|
272
276
|
```
|
|
273
277
|
|
|
274
278
|
Duel runs a command the way that `Process.spawn` does, so a single
|
|
@@ -294,7 +298,7 @@ the most, gets 3, and an app that takes a second or more to boot gets
|
|
|
294
298
|
yourself, pass `runs:`:
|
|
295
299
|
|
|
296
300
|
```ruby
|
|
297
|
-
duel_process "boot: require the library",
|
|
301
|
+
duel_process "boot: require the library", code: %(require "my_gem"), runs: 5
|
|
298
302
|
```
|
|
299
303
|
|
|
300
304
|
Duel starts and measures the processes from a small Ruby process of its
|
|
@@ -315,5 +319,5 @@ makes the same code look slower.
|
|
|
315
319
|
A process benchmark doesn't run in a worker, so nothing that runs there
|
|
316
320
|
applies to it: hooks, from `.duel.rb` or its file, and the metrics that
|
|
317
321
|
plugins add. `duel_process` doesn't take the options of `duel`
|
|
318
|
-
(`
|
|
322
|
+
(`judge:`, and the hooks, such as `around_iteration:`),
|
|
319
323
|
and raises an `ArgumentError` if it's given one.
|