duel 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 24277600f238129c16a52fb4115ccb6a6bd340602e7a38187622bfe1550adc1d
4
- data.tar.gz: 489836bc05d935ef59e369149df8ec47e06279005691fd084ea9b587c489f175
3
+ metadata.gz: 4cc965d0ce98a08843bc8c1e1c2029045f029a9a4ae3cb05c6fa44c4a1dd7cc4
4
+ data.tar.gz: 54a78bbafc884061376d691fe80f6db3e034d837dcf3d2a8448d50522758f669
5
5
  SHA512:
6
- metadata.gz: 2ce8583f3c82f785cd25b95339c9fd7a751e42b7221d0a8b9af793351560945b8ca4d4001c7725db93c3a4d77506baf3cf5de37b5decd4c055f1ca17313de997
7
- data.tar.gz: 81e190402ee6b2eb3705b41fece6ee45e12bc3a6f71645ac1467d7bbe5ed015195d9291fb4b651888b69b759fa4f109f88029e7b7c0272260c2326af11692b65
6
+ metadata.gz: 28a9a52155e5aa377428a359b2fb0c5c17b6518f9c2af07c473afb66b772c5e46825bb5f712df21590d60d65e35f2446e9412e93013cca2332bcc5bb99f94da0
7
+ data.tar.gz: be20c6a67c205344149983ae8e01d49570eb2406d4ff2a4dcf828ea3ab03a260faa7ce782fec119f8257b9138b15bc877b2747f4f7fb1c9c4b3ae2a8c60a4a07
data/AGENT_PROMPT.md CHANGED
@@ -151,9 +151,9 @@ at the API.
151
151
  candidate's).
152
152
 
153
153
  Then do all three:
154
- - Run `bundle exec duel --force`. With `lib/` unchanged, it compares
155
- the code with itself and must pass. Run it three times: a noisy
156
- benchmark can pass twice by luck. A benchmark that fails or
154
+ - Run `bundle exec duel --force`. With the code unchanged, it
155
+ compares the code with itself and must pass. Run it three times: a
156
+ noisy benchmark can pass twice by luck. A benchmark that fails or
157
157
  errors is broken. One reported unsettled in more than one run is
158
158
  noisy: fix it or drop it. (Across many benchmarks, one unsettled
159
159
  once is normal: it doesn't fail the run.) Benchmarks that run a
@@ -168,24 +168,28 @@ at the API.
168
168
  and rebuild the extension if you changed one.
169
169
  - Run `bundle exec duel --force --filter <pattern>` for a single
170
170
  benchmark, and for one area (`--filter "^parse:"`), to confirm
171
- your names can be picked out. Without `--force`, an unchanged
172
- `lib/` has nothing to compare.
171
+ your names can be picked out. Without `--force`, a tree with
172
+ nothing changed since the main branch has nothing to compare.
173
173
 
174
174
  6. **Check CI.** `duel init` wrote Duel's GitHub Actions
175
175
  workflow (`.github/workflows/perf.yml`). Keep `fetch-depth: 0`:
176
176
  Duel's baseline comes from git history. Keep its check for a push
177
177
  with no commit before it in the history (a branch's first push, or a
178
- force push), which would otherwise fail the job. Keep its last step,
179
- which comments on a pull request with Duel's report (Duel also adds
180
- the report to the job's summary). Check that `ruby-version` is one
181
- the project supports, unless it has a `.ruby-version`. With
182
- `c.ruby = :version_file` and another version file than
183
- `.ruby-version`, add the steps in Duel's `docs/ci.md` that install the
184
- baseline's Ruby for a pull request. Add the project's other CI setup that its
185
- tests need (system packages, services) to the workflow as well. Keep
186
- the scheduled `last-release` run only if the project tags its
187
- releases (`v*` by default, or else set `c.release_tags`), or leave
188
- the schedule out. Until the first release is tagged, that run says
178
+ force push), which would otherwise fail the job. Keep its Comment
179
+ job, which comments on a pull request with Duel's report when a
180
+ benchmark fails (Duel also adds the report to the job's summary, and
181
+ `c.comment` in `.duel.rb` chooses when it comments), and keep
182
+ `pull-requests: write` on that job alone: the Duel job runs the
183
+ branch's code, so it only reads. Check that `ruby-version` is one the project supports, unless
184
+ it has a `.ruby-version`. With `c.ruby = :version_file` and another
185
+ version file than `.ruby-version`, add the steps in Duel's
186
+ `docs/ci.md` that install the baseline's Ruby, reading that file
187
+ instead. If `.duel.rb` sets `c.main_branch` or `c.release_tags`, set
188
+ those steps' `MAIN_BRANCH` or `RELEASE_TAGS` to match. Add the
189
+ project's other CI setup that its tests need (system packages,
190
+ services) to the Duel job as well. Keep the scheduled `last-release`
191
+ run only if the project tags its releases (`v*` by default, or else
192
+ set `c.release_tags`), or leave the schedule out. Until the first release is tagged, that run says
189
193
  there's nothing to compare against, and passes. Once it's pushed,
190
194
  run the workflow by hand: it compares the code with itself there,
191
195
  which shows that the benchmarks run on CI's machines (with their
data/CHANGELOG.md CHANGED
@@ -7,6 +7,151 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.2] - 2026-11-02
11
+
12
+ ### Added
13
+
14
+ - `c.fail_on = :pull_requests` (or `duel --fail-on pull_requests`) fails
15
+ a regression only in a pull request (or a merge queue), and outside
16
+ GitHub Actions. On GitHub Actions, a regression in a push, the
17
+ schedule, or a run by hand passes, with a warning, and the workflow
18
+ that `duel init` writes comments on a pushed commit with the report,
19
+ as `c.comment` says, which notifies the commit's author. The default,
20
+ `:every_run`, fails every run, as before, and comments on no commit.
21
+ - `c.push_base = :last_passed` compares a push with the last commit
22
+ that Duel passed, which a branch marks (`duel/passed/main` for
23
+ `main`, and each branch the workflow guards has its own), instead of
24
+ the commit before the push, so slowdowns too small for any one push to
25
+ fail add up until they fail one. The workflow that `duel init` writes
26
+ moves the branch when a push passes. The default, `:before`, compares
27
+ a push with the commit before it, as before.
28
+ - `c.accept_label` names a pull request label that accepts a regression:
29
+ with it, the regression passes, with a warning, and the report says
30
+ so. The workflow that `duel init` writes passes the pull request's
31
+ labels to `duel` (`DUEL_LABELS`), or for a push, the labels of the
32
+ pull request that merged it. By default, there's no such label.
33
+ - `duel init --gitlab` writes GitLab CI jobs to include from
34
+ `.gitlab-ci.yml`, instead of GitHub Actions workflows. They compare a
35
+ merge request with the branch it targets, comment on it, and with
36
+ project access tokens, comment on a pushed commit and mark a push
37
+ passed. On GitLab CI, `duel` reads a merge request's target and labels,
38
+ writes its decisions to `DUEL_OUTPUT` as variables, and where only a
39
+ merge request fails (`c.fail_on`), exits with status 3 for a
40
+ regression outside one, which the job lets fail, as a warning.
41
+ - The workflow that `duel init` writes runs for a merge queue
42
+ (`merge_group`), and compares its group with the queue's base, so the
43
+ Duel check can be required with a merge queue.
44
+
45
+ ### Changed
46
+
47
+ - To upgrade a project that ran `duel init` before 0.1.2, delete
48
+ `.github/workflows/perf.yml` (or `perf-<its path>.yml`, for a project
49
+ in a subdirectory) and run `duel init` again, then carry over any
50
+ edits of your own. It writes both workflows, and skips the files that
51
+ exist, such as `.duel.rb`. Kept, the old workflow comments on a pull
52
+ request itself, and with the new one, every comment comes twice.
53
+ - A pull request gets its comment from a second workflow that `duel init`
54
+ writes, `.github/workflows/perf-comment.yml`, which runs once the Perf
55
+ workflow has (`workflow_run`), with a token that can write, so a pull
56
+ request from a fork gets a comment too. It runs none of the pull
57
+ request's code, and trusts only what Duel could have said in the
58
+ report's artifact. The Perf workflow no longer has a Comment job, and
59
+ for a project in a subdirectory, it's named for the directory, as
60
+ `Perf (gems/parser)`.
61
+ - The workflow that `duel init` writes cancels a pull request's run
62
+ when a newer push to the pull request starts one. Every other run,
63
+ such as a push's to `main`, still runs to the end.
64
+ - A push whose commit before it isn't in the history now reports that
65
+ there's no commit to compare against, and passes, from `duel` itself.
66
+ The workflow no longer checks this in its Compare step.
67
+
68
+ ### Fixed
69
+
70
+ - On GitHub Actions, a pull request is compared with the branch that it
71
+ targets (`GITHUB_BASE_REF`), not the main branch. A pull request into
72
+ a stable branch, or into another pull request's branch, was compared
73
+ with `main`.
74
+
75
+ ## [0.1.1] - 2026-10-26
76
+
77
+ ### Added
78
+
79
+ - `c.comment` chooses when the workflow that `duel init` writes
80
+ comments on a pull request with the report: `:always`, `:results`
81
+ (when a run passed or failed, but not with nothing to compare),
82
+ `:failures` (when a benchmark failed or Duel couldn't compare, the
83
+ default), or `:never`. `duel --comment MODE` overrides it, and any
84
+ other value fails with the values it takes. On GitHub Actions, `duel`
85
+ sets the step outputs `comment` (`post`, `change`, or `none`) and
86
+ `status` (`passed`, `failed`, or `neutral`), whatever the outcome, and
87
+ the workflow's Comment job acts on them, since it runs none of the
88
+ branch's code and can't read `.duel.rb`. The JSON report's `comment`
89
+ key has the same decision.
90
+
91
+ ### Changed
92
+
93
+ - The workflow that `duel init` writes no longer comments on every pull
94
+ request. By default, it comments only when a benchmark fails or Duel
95
+ couldn't compare. Set `c.comment = :always` for the old behavior.
96
+ - The workflow never edits a comment. It used to update one comment on
97
+ every push, and now posts a new one each time it comments, so the pull
98
+ request keeps every report. Each comment's hidden marker records the
99
+ run's status (`<!-- duel status=failed -->`, say). When a run wouldn't
100
+ get a comment of its own, the workflow still posts one if its last
101
+ comment on the pull request had another status, so after a failure the
102
+ first run that passes says so, and the last comment is never stale. A
103
+ comment from an earlier version, whose marker has no status, counts
104
+ as one with another status.
105
+
106
+ - A change to only the benchmarks or the configuration is compared. Duel
107
+ watches what decides how the code is measured, as well as the code
108
+ that `c.watch` names: the benchmarks' directory (`c.benchmarks` up to
109
+ its first wildcard, `duel/` by default), the configuration's file
110
+ (`.duel.rb`), and, with `c.ruby = :version_file`, the version files at
111
+ the project's root. Each version runs the candidate's benchmarks and
112
+ configuration, so such a change compares the code with itself, which
113
+ checks that they run and pass, where before Duel passed saying there
114
+ was nothing to compare. "Nothing to compare" messages, and the
115
+ candidate's `uncommitted_changes` in JSON, cover them too.
116
+ - The Markdown report (`--markdown`, the job's summary and the pull
117
+ request comment) is shorter. Its heading starts with 🟢 when every
118
+ benchmark passed (`🟢 Duel: 100% of benchmarks passed`), 🔻 when one
119
+ failed or Duel couldn't compare (`🔻 Duel: 3 benchmarks failed`), and
120
+ ⬜ when there's nothing to compare. One line names the two commits,
121
+ bare, so GitHub links them, and how the baseline was chosen. Only the
122
+ benchmarks that failed are in view, in a table of what got worse and
123
+ by how much, or the error, and where the most extra CPU time went.
124
+ Everything else is collapsed, and the report no longer names the
125
+ plugins or explains how Duel decides outside the collapsed details.
126
+ The terminal and JSON reports are unchanged.
127
+
128
+ ### Security
129
+
130
+ - The workflow that `duel init` writes no longer gives the job that runs
131
+ a pull request's code the `pull-requests: write` permission. The Duel
132
+ job only reads (`contents: read`, and its checkout keeps no
133
+ credentials), and keeps the report as an artifact for a day. A
134
+ Comment job of its own, which checks nothing out and runs none of the
135
+ pull request's code, downloads the report and comments with it, for a
136
+ pull request from the same repository.
137
+
138
+ ### Fixed
139
+
140
+ - With `c.ruby = :version_file`, the workflow installs the bundle for
141
+ the baseline's Ruby too, when it differs from the candidate's. The
142
+ baseline's workers load that bundle, and gems installed for one Ruby
143
+ version don't load on another, so a branch that changed the Ruby's
144
+ minor version failed before it compared anything.
145
+
146
+ - With `c.ruby = :version_file`, the workflow that `duel init` writes
147
+ installs the baseline's Ruby for every event that compares, not only
148
+ for a pull request. It finds the baseline that the Compare step uses
149
+ (where HEAD left the main branch for a pull request and a manual run,
150
+ the commit before a push, and the last release on a schedule, with
151
+ the release tags' glob in `RELEASE_TAGS`), and installs its Ruby only
152
+ when its `.ruby-version` names another Ruby than HEAD's. Its scripts
153
+ take the workflow's values from environment variables, not `${{ }}`.
154
+
10
155
  ## [0.1.0] - 2026-10-24
11
156
 
12
157
  The first release. Duel is open source under the MIT license, and
@@ -277,5 +422,7 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
277
422
  Anything marked `@api private`, in lib/ and sig/, may change in any
278
423
  release.
279
424
 
280
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.0...HEAD
425
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.2...HEAD
426
+ [0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
427
+ [0.1.1]: https://github.com/sferik/duel-ruby/compare/v0.1.0...v0.1.1
281
428
  [0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
data/README.md CHANGED
@@ -201,6 +201,7 @@ the limit, use `--timeout`.
201
201
  | `0` | Nothing got slower, or there was nothing to compare. |
202
202
  | `1` | Something got slower, or a benchmark failed. |
203
203
  | `2` | Duel couldn't compare. |
204
+ | `3` | Something got slower outside a merge request on GitLab CI, where only a merge request fails ([`c.fail_on`](docs/configuration.md#fail-only-a-pull-request)). The job that `duel init --gitlab` writes lets it fail, as a warning. |
204
205
 
205
206
  ## Run Duel in CI
206
207
 
@@ -215,6 +216,10 @@ duel:
215
216
  GIT_DEPTH: 0 # the baseline comes from history
216
217
  ```
217
218
 
219
+ `duel init --gitlab` writes GitLab CI jobs for you, which compare a
220
+ merge request with the branch it targets and comment on it. See
221
+ [Run Duel on GitLab CI](docs/ci.md#run-duel-on-gitlab-ci).
222
+
218
223
  To run Duel from Rake, define a task with `Duel::RakeTask`. Any
219
224
  arguments after the task's name are passed to `duel`:
220
225
 
@@ -225,11 +230,14 @@ Duel::RakeTask.new(:perf_since_release, "--base", "last-release")
225
230
  ```
226
231
 
227
232
  `duel init` writes a GitHub Actions workflow,
228
- `.github/workflows/perf.yml`. It compares each pull request with the
229
- main branch, each push to `main` with the commit before it, and, on a
230
- schedule, your last release, to catch slow creep. It also comments on
231
- each pull request with Duel's report. For the workflow, and for a branch
232
- that upgrades Ruby, see [Run Duel in CI](docs/ci.md).
233
+ `.github/workflows/perf.yml`, and a second one,
234
+ `.github/workflows/perf-comment.yml`, that comments on a pull request,
235
+ a fork's too. It compares each pull request with the
236
+ branch that it targets, each push to `main` with the commit before it, and, on a
237
+ schedule, your last release, to catch slow creep. It also comments on a
238
+ pull request with Duel's report when a benchmark fails. For the
239
+ workflow, and for a branch that upgrades Ruby, see
240
+ [Run Duel in CI](docs/ci.md).
233
241
 
234
242
  ### Report on a pull request
235
243
 
@@ -242,28 +250,27 @@ comment, use `--markdown`:
242
250
  duel --markdown tmp/duel.md
243
251
  ```
244
252
 
245
- The report starts with the verdict and the two versions that Duel
246
- compared, with each version's Ruby when they differ. A table lists the
247
- benchmarks that changed, regressions first, with each timing's ratio and
248
- its interval, and the counts that changed. Each regression is explained
249
- as it is in the terminal, with what its metrics said and where the extra
250
- CPU time went. Then come the benchmarks whose timings were too close to
251
- call, and the benchmarks that failed. The benchmarks that didn't change
252
- are collapsed, so the report stays short with hundreds of benchmarks:
253
+ The report is short. In view, it has the verdict, the two commits that
254
+ Duel compared (bare, so GitHub links them), and how the baseline was
255
+ chosen. When a benchmark fails, a table shows what got worse and by how
256
+ much, or the error, and the frame where the most extra CPU time went.
257
+ Everything else is collapsed: each regression explained as it is in the
258
+ terminal, every benchmark's timings, the timings too close to call, the
259
+ verdicts across the board, each version's Ruby when they differ, and
260
+ how Duel decides. So the report stays short with hundreds of
261
+ benchmarks:
253
262
 
254
263
  ````markdown
255
- ### Duel failed: 1 regression
256
-
257
- - **Baseline:** `3f9c2a1`, where HEAD left `origin/main`
258
- - **Candidate:** `8e4d0b7`
264
+ ### 🔻 Duel: 1 benchmark failed
259
265
 
260
- 12 benchmarks in 14s. A timing is slower when a 99.0% confidence interval for its ratio, the candidate's to the baseline's, is above 1.02.
266
+ Compared 8e4d0b7 with 3f9c2a1 (where HEAD left `origin/main`).
261
267
 
262
- | Benchmark | Verdict | Timings (ratio [interval]) | Counts an iteration |
263
- |---|---|---|---|
264
- | `parse: a large document` | **regressed** | instructions 1.071 [1.069, 1.073] slower<br>cpu 1.081 [1.062, 1.097] slower | allocations 5790 → 5791 |
268
+ | Benchmark | Failure | Most extra CPU in |
269
+ |---|---|---|
270
+ | `parse: a large document` | instructions +7.1%, cpu +8.1%, allocations 5790 → 5791 | `MyParser::Lexer#next_token` +6.8% |
265
271
 
266
- #### Regressions
272
+ <details>
273
+ <summary>Details: 12 benchmarks in 14s</summary>
267
274
 
268
275
  **`parse: a large document`**: worse instructions, cpu, allocations in every measurement
269
276
 
@@ -276,19 +283,45 @@ where the extra CPU went (as a share of the baseline's: in each frame itself, an
276
283
  +0.4% +0.4% String#byteslice
277
284
  ```
278
285
 
279
- <details>
280
- <summary>11 benchmarks unchanged</summary>
286
+ | Benchmark | Verdict | Timings (ratio [interval]) | Counts an iteration |
287
+ |---|---|---|---|
288
+ | `parse: a large document` | **regressed** | instructions 1.071 [1.069, 1.073] slower<br>cpu 1.081 [1.062, 1.097] slower | allocations 5790 → 5791 |
289
+ | `parse: a short line` | same | instructions 1.000 [1.000, 1.000]<br>cpu 1.002 [0.992, 1.017] | |
281
290
 
282
291
  ...
283
292
 
284
293
  </details>
285
294
  ````
286
295
 
296
+ When every benchmark passes, the heading says `🟢 Duel: 100% of
297
+ benchmarks passed`, and only the commits are in view. A timing too
298
+ close to the limit to call doesn't fail the run, and is in the details.
299
+ A benchmark that the baseline can't run (a new one, say) is skipped, and
300
+ the heading counts it. A run that has nothing to compare, or no release
301
+ to compare against, says so under ⬜, and one that couldn't compare
302
+ says why under 🔻.
303
+
287
304
  The workflow that `duel init` writes posts this report as a comment on
288
- the pull request, and updates the same comment on every push. For more
289
- information, see
305
+ the pull request when a benchmark fails, or when Duel couldn't compare.
306
+ After such a comment, the first run that passes posts one more, so the
307
+ last comment is never stale. It never edits a comment, so the pull
308
+ request keeps every report. To comment on every run, on every result
309
+ but nothing to compare, or never, set
310
+ [`c.comment`](docs/configuration.md#choose-when-to-comment-on-a-pull-request)
311
+ in `.duel.rb`. For more information, see
290
312
  [Comment on a pull request](docs/ci.md#comment-on-a-pull-request).
291
313
 
314
+ In a project where people push to `main` directly, set
315
+ [`c.fail_on = :pull_requests`](docs/configuration.md#fail-only-a-pull-request)
316
+ to pass a regression in a push, with a warning, and comment on the
317
+ pushed commit with the report instead. To compare each push with the
318
+ last commit that passed, instead of the commit before it, so that small
319
+ slowdowns add up until they fail a push, set
320
+ [`c.push_base = :last_passed`](docs/configuration.md#compare-a-push-with-the-last-commit-that-passed).
321
+ To accept a regression that's on purpose, label the pull request, as
322
+ [`c.accept_label`](docs/configuration.md#accept-a-regression-with-a-label)
323
+ says.
324
+
292
325
  ### Show that a project is guarded
293
326
 
294
327
  To show that Duel guards your project, add its badge to your README,