duel 0.1.3 → 0.1.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3827f9485e70bb95c50b32ff9438042bd60cb26885d92f01fe6c1dc6de33ee72
4
- data.tar.gz: 53e410c89687f6bcdec3c46be4d370409110ac134d90ef74ca8aaaad62394a7e
3
+ metadata.gz: 2528f9b36dd187ea88916fa507ab544b243a58b6c74cfe9c75083c127d9c3bbb
4
+ data.tar.gz: ca3838705ef0db2b367c115e96e555fb346173be3aa7a1a1f1740e7a5c895a40
5
5
  SHA512:
6
- metadata.gz: 7710b0ab9cc84821d6e66af1d66e1005b98aa0e97a1f047e2a76d6c5d3a83defb413895e11476d067265b97d589dcbc2fb5bdf78550616da74cf92df0da2bd19
7
- data.tar.gz: 6d9d0df5accaf420f0a4e554f21eb22e79e77f8865986af856759272c5358b6cacd22af27800827d6faf7360daeded8d58481a6ef3c22669279d13a85fe4ff83
6
+ metadata.gz: 0663ad25b83ffa57ff1fdedb64c462c8fea600967d26ec49831071bf69fa4ebfbbf2a9a17f8ed1bdfbb3c8bb7a55757cea1fd9a6ad3937bb435ed849dda4ab55
7
+ data.tar.gz: d2b0f54723cfe98dbffa56103b5699bc532b9d305c2d75f5304d2749de0f1c455ab12c6250afbf8932537b7a883a6101bf61f242a0e7c5f24f65f866d60e6fb5
data/AGENT_PROMPT.md CHANGED
@@ -17,10 +17,11 @@ at the API.
17
17
  development group (unless the Gemfile has it already), and run
18
18
  `bundle install`. Then, unless the project is a Rails app (see step
19
19
  4), run `bundle exec duel init`, and read what it wrote: `.duel.rb`,
20
- an example benchmark in `duel/`, the GitHub Actions workflow
21
- (`.github/workflows/perf.yml`), Duel's Rake task, if there's a
22
- Rakefile, and `gem "stackprof", platforms: :mri` in the Gemfile, for
23
- profiles of regressions. It lists what it did and what's next: run
20
+ an example benchmark in `duel/`, two GitHub Actions workflows
21
+ (`.github/workflows/perf.yml`, and `perf-comment.yml`, which comments
22
+ on pull requests), Duel's Rake task, if there's a Rakefile, and
23
+ `gem "stackprof", platforms: :mri` in the Gemfile, for profiles of
24
+ regressions. It lists what it did and what's next: run
24
25
  `bundle install` again for StackProf.
25
26
 
26
27
  2. **Find what's worth guarding.** Read the code and its tests to find
@@ -58,6 +59,22 @@ at the API.
58
59
  includes what differs between the two versions' workers, like a
59
60
  server's socket path or a worker's directory: don't time work on
60
61
  it.
62
+ - Build inputs without the code you're measuring. Each version's
63
+ workers run the benchmarks with that version's code, so an input
64
+ made with the project's own encoder (its `dump`, `to_json` or
65
+ `serialize`) differs wherever the two versions' output does, and
66
+ the benchmark then compares different work. Write the input out as
67
+ a fixture, or build it from literals or Ruby's standard library.
68
+ See "Give both versions the same work" in Duel's
69
+ `docs/benchmarks.md`.
70
+ - Leave process-wide settings alone. Every worker loads every
71
+ benchmark file, so a mode, patch or global option that changes the
72
+ whole process (an `optimize` call, a changed default) changes every
73
+ benchmark that runs after it, and the default path goes unmeasured.
74
+ Never turn one on in `duel_helper.rb` or at a file's top level.
75
+ Turn it on only in the benchmarks that are about it, and put it
76
+ back in `teardown`, or measure it in a process of its own with
77
+ `duel_process`.
61
78
  - Use the sizes the code really sees. Read it for limits and
62
79
  constants (a cap on how many items it ever handles, say) rather
63
80
  than guessing.
@@ -135,9 +152,12 @@ at the API.
135
152
  there. Run `duel init` there too: it puts the workflow at the
136
153
  repository's root, and runs it in the subdirectory.
137
154
 
138
- Make sure the linter checks `.duel.rb`. It's a dotfile, so RuboCop
139
- leaves it out unless its config includes it (`AllCops: Include`,
140
- merged with the defaults by `inherit_mode`).
155
+ Don't change the project's own configuration beyond what `duel init`
156
+ did: its linter's config, its README (but for step 7), its other CI
157
+ workflows. Those are the maintainers' choices. If the linter flags
158
+ the benchmarks, fix the benchmarks. If it can't be satisfied without
159
+ a config change (it leaves `.duel.rb` out, say, as RuboCop does with
160
+ dotfiles), say so in your report instead.
141
161
 
142
162
  5. **Check that the guards hold.** While you're still writing
143
163
  benchmarks, run a few at a time (`bundle exec duel --force --filter
@@ -171,29 +191,39 @@ at the API.
171
191
  your names can be picked out. Without `--force`, a tree with
172
192
  nothing changed since the main branch has nothing to compare.
173
193
 
174
- 6. **Check CI.** `duel init` wrote Duel's GitHub Actions
175
- workflow (`.github/workflows/perf.yml`). Keep `fetch-depth: 0`:
176
- Duel's baseline comes from git history. Keep its check for a push
177
- with no commit before it in the history (a branch's first push, or a
178
- force push), which would otherwise fail the job. Keep its Comment
179
- job, which comments on a pull request with Duel's report when a
180
- benchmark fails (Duel also adds the report to the job's summary, and
181
- `c.comment` in `.duel.rb` chooses when it comments), and keep
182
- `pull-requests: write` on that job alone: the Duel job runs the
183
- branch's code, so it only reads. Check that `ruby-version` is one the project supports, unless
184
- it has a `.ruby-version`. With `c.ruby = :version_file` and another
185
- version file than `.ruby-version`, add the steps in Duel's
186
- `docs/ci.md` that install the baseline's Ruby, reading that file
187
- instead. If `.duel.rb` sets `c.main_branch` or `c.release_tags`, set
188
- those steps' `MAIN_BRANCH` or `RELEASE_TAGS` to match. Add the
189
- project's other CI setup that its tests need (system packages,
190
- services) to the Duel job as well. Keep the scheduled `last-release`
191
- run only if the project tags its releases (`v*` by default, or else
192
- set `c.release_tags`), or leave the schedule out. Until the first release is tagged, that run says
193
- there's nothing to compare against, and passes. Once it's pushed,
194
- run the workflow by hand: it compares the code with itself there,
195
- which shows that the benchmarks run on CI's machines (with their
196
- packages and services).
194
+ 6. **Check CI.** `duel init` wrote two GitHub Actions workflows.
195
+ - `.github/workflows/perf.yml` runs Duel. Keep `fetch-depth: 0`:
196
+ Duel's baseline comes from git history. Keep its check for a push
197
+ with no commit before it in the history (a branch's first push, or
198
+ a force push), which would otherwise fail the job. The Duel job
199
+ runs the branch's code, so it only reads: don't give it, or the
200
+ workflow, any write permission.
201
+ - `.github/workflows/perf-comment.yml` comments on a pull request
202
+ with Duel's report once the first has run (`c.comment` in
203
+ `.duel.rb` chooses when, and Duel also adds the report to the
204
+ job's summary). It runs none of the pull request's code, and shows
205
+ a fork's report as text. Keep it as it is.
206
+ - `duel init` writes the jobs for pushes (a comment on a pushed
207
+ commit, and a `duel/passed/` branch that marks the last commit
208
+ Duel passed) only where `.duel.rb` sets `c.fail_on =
209
+ :pull_requests` or `c.push_base = :last_passed`, and the badge's
210
+ job only with `--badge`. Each needs `contents: write`, so don't
211
+ set those settings, or add those jobs, unless I ask.
212
+
213
+ Check that `ruby-version` is one the project supports, unless it has
214
+ a `.ruby-version`. With `c.ruby = :version_file` and another version
215
+ file than `.ruby-version`, add the steps in Duel's `docs/ci.md` that
216
+ install the baseline's Ruby, reading that file instead. If
217
+ `.duel.rb` sets `c.main_branch` or `c.release_tags`, set those
218
+ steps' `MAIN_BRANCH` or `RELEASE_TAGS` to match. Add the project's
219
+ other CI setup that its tests need (system packages, services) to
220
+ the Duel job as well. Keep the scheduled `last-release` run only if
221
+ the project tags its releases (`v*` by default, or else set
222
+ `c.release_tags`), or leave the schedule out. Until the first
223
+ release is tagged, that run says there's nothing to compare against,
224
+ and passes. Once it's pushed, run the workflow by hand: it compares
225
+ the code with itself there, which shows that the benchmarks run on
226
+ CI's machines (with their packages and services).
197
227
 
198
228
  7. **Show the badge.** If the project's README already shows badges (CI
199
229
  status, gem version), add Duel's to that row, with exactly this
@@ -203,7 +233,10 @@ at the API.
203
233
  [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
204
234
  ```
205
235
 
206
- If the README has no badges, don't add a row just for Duel's.
236
+ If the README has no badges, don't add a row just for Duel's. This
237
+ badge is always green: one that turns red when Duel fails a push
238
+ needs the badge's job (`duel init --badge`), so add that only if I
239
+ ask.
207
240
 
208
241
  8. **Report back** with:
209
242
  - The benchmarks you wrote and what each one guards.
@@ -211,5 +244,9 @@ at the API.
211
244
  - The output of the self-comparison and of the deliberate-regression
212
245
  check.
213
246
  - Anything you couldn't guard reliably.
247
+ - Every file you changed outside `duel/` and `.duel.rb`, and why.
248
+ - The label that lets a pull request that's slower on purpose pass
249
+ (`c.accept_label`, `duel: accept` in the `.duel.rb` that
250
+ `duel init` writes), which the repository needs to have.
214
251
 
215
252
  Don't commit. Leave the changes for me to review.
data/CHANGELOG.md CHANGED
@@ -7,6 +7,92 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.5] - 2026-11-08
11
+
12
+ ### Changed
13
+
14
+ - `duel init` writes the Comment on the commit job only where `.duel.rb`
15
+ sets `c.fail_on = :pull_requests`, and the Mark the commit passed job
16
+ only where it sets `c.push_base = :last_passed`, as with the defaults
17
+ neither runs, and each asks for `contents: write`. It reads the
18
+ settings from `.duel.rb` where there is one, and where that doesn't
19
+ load (one of its plugins isn't installed, say), writes both, and says
20
+ so. To add a job later, set its setting, remove the workflow, and run
21
+ `duel init` again, or add it from the guide.
22
+ - `duel init` writes the Update the badge job only with `--badge`. It
23
+ pushes a branch, so it needs `contents: write`, and without the jobs
24
+ for pushes, it was the workflow's only job that writes to the
25
+ repository. The workflow `duel init` writes by default now only reads.
26
+ To keep a badge you already show, add `--badge` when you run
27
+ `duel init` again, or keep the job you have.
28
+ - The `.duel.rb` that `duel init` writes sets
29
+ `c.accept_label = "duel: accept"`, so a pull request that's slower on
30
+ purpose can pass with that label. Duel's own default is still no
31
+ label, so a `.duel.rb` you already have is unchanged.
32
+ - `require "duel/rake_task"` loads only the task, and the task loads
33
+ Duel as it runs, so a Rakefile with Duel's task no longer loads all of
34
+ Duel (and what it requires) for every other task, `rake test` included.
35
+
36
+ ### Fixed
37
+
38
+ - The Perf comment workflow posted the report that a pull request's run
39
+ wrote as Markdown, as `github-actions[bot]`. A fork's run runs the
40
+ fork's code, which could write anything there, mentions, links and
41
+ images included. A fork's report is now shown as text, in a code
42
+ block, where it can't do any of that, and cut short where a comment
43
+ couldn't hold it. A pull request from a branch of the repository, whose
44
+ author can push to it anyway, still gets the Markdown. Update
45
+ `.github/workflows/perf-comment.yml` to pick this up.
46
+ - The Update the badge job downloaded the badge's image from Duel's own
47
+ repository, so a change there changed every project's badge, and a
48
+ failed download failed the job. It now writes the image itself, and
49
+ fetches nothing. Update it in `.github/workflows/perf.yml` to pick
50
+ this up.
51
+ - Two pushes' runs of the Update the badge job could overlap, as a
52
+ push's run doesn't wait for the one before it, and an older push's
53
+ could write its badge over a newer one's after both had checked
54
+ which push the badge was for. The job now runs one at a time. Update
55
+ it in `.github/workflows/perf.yml` to pick this up.
56
+
57
+ ## [0.1.4] - 2026-11-06
58
+
59
+ ### Fixed
60
+
61
+ - In json's own repository, every baseline worker loaded the
62
+ candidate's json, extensions and all, before the baseline's, as the
63
+ worker loaded json for its own messages to the runner before it
64
+ loaded the version. A worker and the runner now send each other
65
+ their messages marshaled, so a worker loads nothing a version could
66
+ be before it loads the version.
67
+ - A worker loaded etc, a gem, before it loaded the version it
68
+ measures, for the page size that `/proc/self/statm` counts resident
69
+ memory in. In etc's own repository, a worker would load the
70
+ bundle's etc before the version's own. A worker now reads resident
71
+ memory, in kB, from `/proc/self/status` (VmRSS), and loads no etc.
72
+ - A worker loaded fiddle, a gem, as Duel loaded, before it loaded the
73
+ version it measures, so in fiddle's own repository, a worker would
74
+ load the bundle's fiddle before the version's own. Duel now loads
75
+ fiddle as it first calls into the operating system, once the
76
+ version has loaded (and on JRuby, which Duel makes no such calls
77
+ on, never).
78
+ - A worker that loaded a native extension twice, the candidate's that
79
+ something required before the worker loaded the version and then the
80
+ version's own over it, went unnoticed, as loading the version took
81
+ the candidate's off `$LOADED_FEATURES`. Duel now fails, and names the
82
+ extension and both of its files.
83
+ - On JRuby, peak memory came out short, by tens of megabytes at times,
84
+ and a benchmark that needed more memory could pass. A peak is
85
+ measured above what's in use after the garbage is collected, but
86
+ JRuby's `GC.start` collects nothing, so the garbage left from before
87
+ stayed in use until the benchmark's own objects made room. Duel now
88
+ collects it with the JVM's own call (`java.lang.System.gc`).
89
+ - Where a repository's history was replaced, so the commit that
90
+ `duel/passed/BRANCH` marks has no history in common with a push, the
91
+ Mark the commit passed job failed, as GitHub can't compare the two.
92
+ It now moves the branch to the push, as it does where history was
93
+ rewritten. Update the job in `.github/workflows/perf.yml` to pick
94
+ this up.
95
+
10
96
  ## [0.1.3] - 2026-11-04
11
97
 
12
98
  ### Added
@@ -488,7 +574,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
488
574
  Anything marked `@api private`, in lib/ and sig/, may change in any
489
575
  release.
490
576
 
491
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.3...HEAD
577
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.5...HEAD
578
+ [0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
579
+ [0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
492
580
  [0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
493
581
  [0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
494
582
  [0.1.1]: https://github.com/sferik/duel-ruby/compare/v0.1.0...v0.1.1
data/README.md CHANGED
@@ -83,7 +83,10 @@ out are Duel's.
83
83
 
84
84
  `duel init` doesn't overwrite a file that exists. To overwrite it, use
85
85
  `--force`. To see what `duel init` would do without writing anything,
86
- use `--dry-run`. When it's done, `duel init` lists the next steps.
86
+ use `--dry-run`. For a badge in your README that turns red when Duel
87
+ fails a push, use `--badge` (see
88
+ [Show that a project is guarded](#show-that-a-project-is-guarded)).
89
+ When it's done, `duel init` lists the next steps.
87
90
 
88
91
  For a Rails app, use
89
92
  [duel-rails](https://github.com/sferik/duel-ruby/tree/main/plugins/rails)
@@ -335,17 +338,17 @@ says.
335
338
  ### Show that a project is guarded
336
339
 
337
340
  To show that Duel guards your project, add its badge to your README,
338
- next to your other badges. With the workflow that `duel init` writes,
339
- the badge turns red when Duel fails a push to your default branch.
340
- Replace `OWNER/REPO` with your repository's:
341
+ next to your other badges. With the job that `duel init --badge` adds to
342
+ its workflow, the badge turns red when Duel fails a push to your default
343
+ branch. Replace `OWNER/REPO` with your repository's:
341
344
 
342
345
  ```markdown
343
346
  [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
344
347
  ```
345
348
 
346
349
  See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
347
- for how the workflow keeps it. Without the workflow, such as on GitLab
348
- CI, use Duel's own badge, which is always green:
350
+ for how the job keeps it. Without the job, such as on GitLab CI, use
351
+ Duel's own badge, which is always green:
349
352
 
350
353
  ```markdown
351
354
  [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
@@ -388,8 +391,8 @@ plugin, see [Write a Duel plugin](docs/plugins.md).
388
391
  The [Duel guide](docs/README.md) covers the rest:
389
392
 
390
393
  - [Write benchmarks](docs/benchmarks.md): hooks, undoing each
391
- iteration, sharing code, choosing what a benchmark is judged on, and
392
- benchmarking a whole process.
394
+ iteration, sharing code, giving both versions the same work, choosing
395
+ what a benchmark is judged on, and benchmarking a whole process.
393
396
  - [Configure Duel](docs/configuration.md): `.duel.rb`, preparing each
394
397
  worker and each version, each version's own Ruby, your own metrics,
395
398
  and plugins.
data/docs/README.md CHANGED
@@ -5,7 +5,7 @@ benchmark, run it, and run it in CI. This guide covers the rest.
5
5
 
6
6
  | Page | What it covers |
7
7
  |---|---|
8
- | [Write benchmarks](benchmarks.md) | Hooks that run code around a benchmark, starting every sample alike, undoing each iteration, measuring work that a benchmark leaves for later, sharing code between benchmarks, choosing what a benchmark is judged on, and benchmarking a whole process. |
8
+ | [Write benchmarks](benchmarks.md) | Hooks that run code around a benchmark, starting every sample alike, undoing each iteration, measuring work that a benchmark leaves for later, sharing code between benchmarks, giving both versions the same work, choosing what a benchmark is judged on, and benchmarking a whole process. |
9
9
  | [Configure Duel](configuration.md) | `.duel.rb` and its settings, preparing each worker and each version, running code in each worker, running each version on its own Ruby, adding your own metrics, and using plugins. |
10
10
  | [Run Duel in CI](ci.md) | The GitHub Actions workflow that `duel init` writes, catching slow creep, commenting on a pull request, and installing the baseline's Ruby on CI. |
11
11
  | [How Duel works](how-it-works.md) | How Duel tells a regression from noise, what it measures, where the extra time went, where the baseline comes from, and JIT compilers, including JRuby and TruffleRuby. |
data/docs/benchmarks.md CHANGED
@@ -2,9 +2,9 @@
2
2
 
3
3
  This page covers what a benchmark can do beyond timing a block: run
4
4
  code around its samples and iterations, share code with other
5
- benchmarks, choose what it's judged on, and benchmark a whole process.
6
- For a first benchmark, with `once`, `untimed`, and a block that takes
7
- the number of iterations, see
5
+ benchmarks, give both versions the same work, choose what it's judged
6
+ on, and benchmark a whole process. For a first benchmark, with `once`,
7
+ `untimed`, and a block that takes the number of iterations, see
8
8
  [Write a benchmark](../README.md#write-a-benchmark).
9
9
 
10
10
  ## Run code around a benchmark
@@ -198,6 +198,35 @@ benchmarks: `duel`, `duel_process`, `once`, `untimed`, the hooks
198
198
  Duel's method would hide a helper of the same name, so the worker
199
199
  fails, and names the helper's file and line.
200
200
 
201
+ ## Give both versions the same work
202
+
203
+ Each version's workers load your benchmarks with that version's code,
204
+ so a benchmark compares like with like only where both versions are
205
+ given the same work.
206
+
207
+ Build a benchmark's inputs without the code it measures. A JSON
208
+ parser's benchmark that builds its document with the library's own
209
+ `dump` parses whatever each version dumps: where a change makes `dump`
210
+ write something else, the two versions parse different documents, and
211
+ the comparison measures that, not the parser. Write the input out, read
212
+ it from a file, or build it with code that both versions share, such as
213
+ Ruby's own.
214
+
215
+ ```ruby
216
+ # duel/parser_duel.rb
217
+ DOCUMENT = File.read(File.join(__dir__, "fixtures", "document.json"))
218
+
219
+ duel("parse: a document") { MyParser.parse(DOCUMENT) }
220
+ ```
221
+
222
+ Every worker loads every file of benchmarks, so a setting that changes
223
+ the whole process, such as a library's global option or a patch that a
224
+ mode installs, changes every benchmark that runs after it, in every
225
+ file. Turn it on in the benchmark's own hooks, and put it back in
226
+ `teardown`, or benchmark it in a process of its own, with
227
+ `duel_process`. Then the benchmarks that don't ask for it measure the
228
+ code that they name.
229
+
201
230
  ## Choose what a benchmark is judged on
202
231
 
203
232
  By default, Duel judges each benchmark on its timings and its counts: