duel 0.1.6 → 0.1.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a3bd63fc0a6b4eb33d7fe48f733446e7b35c3e961dfdc42e85531fdc680ca53c
4
- data.tar.gz: d44709c7629208381a9e24235f2401dc53c69c11fd540c8b8ab5c4e02710bf7a
3
+ metadata.gz: e153495685151ec755386e3fb34aeb21d3f9e3cb2015d4900b6d09f611939a25
4
+ data.tar.gz: 30c26bc5ca7ac5d36c1cb438ccfb4886fa4e9b4fb0637188b9a656dd15d9b231
5
5
  SHA512:
6
- metadata.gz: 44fd8dbe608296d4af112bd27525f38d544662e858e39abfc4913583fa1a0d5d74cd9c20c3745dcc226ef1c6f89c6deb9f2f6780bb457627da18e0b7bf82373e
7
- data.tar.gz: 6984d61ce88c4042e8447a9ec03886bb68fa39d4e2d5bdef448a6b4dd1cd7ef6d258a940aac56b3bd08892907fc4d626cd4f91dcbbcbca634b29c78227c7f860
6
+ metadata.gz: 3b902a527e27cce62d6ac90883410da63a3842c727b81656983031235baf44b27e7e28fe3b3de69ec9f7554215b94d74643af41d5f569c0567ef78e423625d70
7
+ data.tar.gz: 3e0255a630de657157d36087f12144d6dd2e0b556a06afd87d86e7ac0cd43fc7047522ec83431dd33369a96d083d546e2fa5608fb247970d5993cf0a2288174f
data/AGENT_PROMPT.md CHANGED
@@ -201,8 +201,8 @@ at the API.
201
201
  - `.github/workflows/perf-comment.yml` comments on a pull request
202
202
  with Duel's report once the first has run (`c.comment` in
203
203
  `.duel.rb` chooses when, and Duel also adds the report to the
204
- job's summary). It runs none of the pull request's code, and shows
205
- a fork's report as text. Keep it as it is.
204
+ job's summary). It runs none of the pull request's code. Keep it as
205
+ it is.
206
206
  - `duel init` writes the jobs for pushes (a comment on a pushed
207
207
  commit, and a `duel/passed/` branch that marks the last commit
208
208
  Duel passed) only where `.duel.rb` sets `c.fail_on =
data/CHANGELOG.md CHANGED
@@ -7,6 +7,86 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.8] - 2026-11-12
11
+
12
+ ### Added
13
+
14
+ - `duel comment`, which the Perf comment workflow now runs instead of
15
+ a shell script: it installs Duel from RubyGems (the newest release of
16
+ the same minor version) and comments with it, so a fix to how it
17
+ comments comes with a release of Duel, without a change to the
18
+ workflow. `duel init --check` shows how a project's workflows differ
19
+ from what the installed Duel writes.
20
+ - Once the Perf comment workflow posts a comment, it hides Duel's
21
+ earlier ones on the pull request as outdated (those not hidden
22
+ already), which GitHub collapses, so the newest is the one in view.
23
+ - `duel init` offers to create the label that `c.accept_label` names,
24
+ with `gh`, where it can (at a terminal, in a GitHub repository that
25
+ hasn't the label). Where it doesn't, its next steps say to.
26
+ - A plugin can say how much memory what it starts for each worker
27
+ takes at most (`def self.worker_memory`), and Duel counts at least
28
+ that much for each worker as it fits them in the memory available. A
29
+ server takes more as it's used than as it starts, when Duel measures
30
+ it.
31
+ - `duel comment --gitlab`, which the GitLab CI jobs' comment job now
32
+ runs instead of a shell script, as the GitHub workflow runs `duel
33
+ comment`: it trusts only what Duel said, comments only if the merge
34
+ request's head is still the run's, and comes from RubyGems, so a fix
35
+ comes with a release. In a merged results pipeline, the report names
36
+ the merge request's own commit, as on GitHub, and `duel init --gitlab`
37
+ offers to create the accept label with `glab`.
38
+
39
+ ### Changed
40
+
41
+ - On a pull request's run on GitHub Actions, the report names the pull
42
+ request's own commit, "merged with `main`", instead of the commit
43
+ that merges it, which GitHub makes for the run and isn't in the pull
44
+ request.
45
+ - A run whose failures `c.accept_label` accepted says so in its
46
+ heading, under 🟡 instead of 🔻: "1 benchmark failed, accepted".
47
+ - The Perf comment workflow posts a report from a fork's pull request
48
+ as Markdown, like any other, instead of as text in a code block.
49
+ - A regression that `c.accept_label` accepts is commented on as a run
50
+ that passed would be. With the default `c.comment = :failures`, Duel
51
+ no longer posts the report again when the label is added, since the
52
+ last comment already showed it failing.
53
+
54
+ ### Fixed
55
+
56
+ - On TruffleRuby, Duel fits the workers in the memory available as
57
+ they'll be once warm, not as they start: its workers grow some 2.9
58
+ times as its JIT compiles, so where 8 of each version seemed to fit,
59
+ they took 24 GB, more than a 16 GB CI runner has, and the runner
60
+ stopped answering.
61
+
62
+ To pick up the Perf comment workflow's changes, run `duel init --check`
63
+ (or see duel-rails's generator's template), and copy them into
64
+ `.github/workflows/perf-comment.yml`.
65
+
66
+ ## [0.1.7] - 2026-11-10
67
+
68
+ ### Fixed
69
+
70
+ - Duel measures fewer benchmarks at once, and where even one
71
+ benchmark's workers wouldn't fit, starts fewer workers, so that they
72
+ fit in the memory available. Since 0.1.6 starts twelve workers of
73
+ each version for each benchmark it measures at once, a plugin whose
74
+ workers each start a server could run a machine out of memory: on a
75
+ GitHub runner, duel-mysql's 48 MySQL servers took more than its
76
+ 16 GB, and the runner stopped answering until its job was lost. Duel
77
+ starts one worker of each version first to find what a pair takes
78
+ (with its resources), and allows each pair a quarter more, and 1 GB
79
+ for everything else. It reads what's available from Linux's
80
+ MemAvailable (or a container's cgroup limit, if lower) or macOS's
81
+ `vm_stat`, and its first line says when it lowered either. A
82
+ `--jobs` or `--processes` you give is kept as it is.
83
+ - The badge (in the workflow that `duel init --badge` writes) says
84
+ "not measured", in grey, where the Duel job failed
85
+ before Duel judged the push (its runner lost, say, or a step before
86
+ Duel's failed), instead of "perf regressed". It's red only where
87
+ Duel failed the push or found a regression, and a job that fails
88
+ after Duel passed the push leaves it green.
89
+
10
90
  ## [0.1.6] - 2026-11-10
11
91
 
12
92
  ### Added
@@ -682,7 +762,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
682
762
  Anything marked `@api private`, in lib/ and sig/, may change in any
683
763
  release.
684
764
 
685
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.6...HEAD
765
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.8...HEAD
766
+ [0.1.8]: https://github.com/sferik/duel-ruby/compare/v0.1.7...v0.1.8
767
+ [0.1.7]: https://github.com/sferik/duel-ruby/compare/v0.1.6...v0.1.7
686
768
  [0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
687
769
  [0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
688
770
  [0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
data/README.md CHANGED
@@ -312,7 +312,12 @@ close to the limit to call doesn't fail the run, and is in the details.
312
312
  A benchmark that the baseline can't run (a new one, say) is skipped, and
313
313
  the heading counts it. A run that has nothing to compare, or no release
314
314
  to compare against, says so under ⬜, and one that couldn't compare
315
- says why under 🔻.
315
+ says why under 🔻. One whose failures the pull request's label accepted
316
+ (see [`c.accept_label`](docs/configuration.md#accept-a-regression-with-a-label))
317
+ says so under 🟡. On a pull request, GitHub Actions checks out a commit
318
+ that merges it with the branch it targets, which isn't in the pull
319
+ request, so the report names the pull request's own commit, "merged
320
+ with `main`".
316
321
 
317
322
  The workflow that `duel init` writes posts this report as a comment on
318
323
  the pull request when a benchmark fails, or when Duel couldn't compare.
data/docs/ci.md CHANGED
@@ -193,7 +193,9 @@ check, because `duel` exits with a nonzero status, and the job's
193
193
  summary has the report.
194
194
 
195
195
  The workflow never edits or deletes a comment, so the pull request
196
- keeps every report that it posted. Each comment starts with a hidden
196
+ keeps every report that it posted. Once it posts one, it hides its
197
+ earlier ones as outdated, which GitHub collapses, so the newest is the
198
+ one in view. Each comment starts with a hidden
197
199
  marker that records the run's status: `<!-- duel status=passed -->`,
198
200
  `status=failed` (a run that couldn't compare too), or `status=neutral`
199
201
  (nothing to compare). When a run's outcome wouldn't get a comment of
@@ -228,16 +230,14 @@ the pull request's number, `comment`, and `status`. A second workflow,
228
230
  the Perf workflow completes (`workflow_run`), with your repository's
229
231
  own copy of the workflow and a token that can write, so a pull request
230
232
  from a fork gets a comment too. It checks nothing out and runs none of
231
- the pull request's code. The pull request's code ran in the Duel job,
232
- and could have written anything into the artifact, so the workflow
233
- trusts only what Duel could have said there: a number that names the
234
- pull request whose head the run compared, and a `comment` and a
235
- `status` that Duel uses. For a pull request from a branch of your
236
- repository, whose author can push to it anyway, the report's Markdown
237
- is the comment's body. For one from a fork, the report is shown as text,
238
- in a code block, since the fork's code wrote it: there, posted by
239
- `github-actions[bot]`, it can't mention anyone, link anywhere or show an
240
- image. A report too long for a comment is cut short. When the Duel job fails before it compares, there's
233
+ the pull request's code. It installs Duel from RubyGems (the newest
234
+ 0.1 release, 0.1.8 or later) and runs `duel comment`, so a fix to how it comments comes
235
+ with a release of Duel, without a change to the workflow. The pull
236
+ request's code ran in the Duel job, and could have written anything
237
+ into the artifact, so `duel comment` trusts only what Duel could have
238
+ said there: a number that names the pull request whose head the run
239
+ compared, and a `comment` and a `status` that Duel uses. The report's
240
+ Markdown, a fork's too, is the comment's body. When the Duel job fails before it compares, there's
241
241
  no report, and no comment. GitHub runs a `workflow_run` workflow only
242
242
  from your default branch, so the pull request that adds the workflow
243
243
  gets no comment.
@@ -249,14 +249,15 @@ name: Perf comment
249
249
  # Duel's report, as a new comment on the pull request, when Duel says to:
250
250
  # always (post), or only when its last comment's status differs from the
251
251
  # run's (change), so the latest comment is never stale. It never edits or
252
- # deletes a comment. It runs once the Perf workflow has, with this
253
- # repository's own workflow and a token that can write, so a pull request
254
- # from a fork gets a comment too. It checks nothing out and runs none of
255
- # the pull request's code, and of what the Duel job kept (where that code
256
- # ran), it trusts only what that job could have said: the pull request's
257
- # number, if it names the pull request the run compared, and the comment
258
- # and the status, if they're ones Duel says. A fork's report, which the
259
- # fork's code wrote, is shown as text.
252
+ # deletes a comment, but hides Duel's earlier ones as outdated, so the
253
+ # newest is the one in view. It runs once the Perf workflow has, with
254
+ # this repository's own workflow and a token that can write, so a pull
255
+ # request from a fork gets a comment too. It checks nothing out and runs
256
+ # none of the pull request's code, only `duel comment` (see Duel's
257
+ # docs/ci.md), which trusts only what the Duel job could have said in
258
+ # what it kept: the pull request's number, if it names the pull request
259
+ # the run compared, and the comment and the status, if they're ones Duel
260
+ # says. Its report, a fork's too, is the comment, as Markdown.
260
261
  #
261
262
  # And when the label that accepts a regression (c.accept_label in
262
263
  # .duel.rb) is added to a pull request, or taken off, it runs the Duel
@@ -294,46 +295,20 @@ jobs:
294
295
  path: ${{ runner.temp }}/report
295
296
  run-id: ${{ github.event.workflow_run.id }}
296
297
  github-token: ${{ github.token }}
298
+ - uses: ruby/setup-ruby@v1
299
+ if: steps.report.outcome == 'success'
300
+ with:
301
+ ruby-version: ruby
302
+ # Duel's own code, from RubyGems (none of the pull request's), so a
303
+ # fix to how it comments comes with a release of Duel.
297
304
  - name: Comment on the pull request
298
305
  if: steps.report.outcome == 'success'
299
306
  env:
300
307
  GH_TOKEN: ${{ github.token }}
301
308
  HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
302
309
  run: |
303
- said() { grep -xE "$1=($2)" "$RUNNER_TEMP/report/comment.env" | tail -n 1 | cut -d= -f2; }
304
- PR=$(said pr '[0-9]+')
305
- COMMENT=$(said comment 'post|change')
306
- STATUS=$(said status 'passed|failed|neutral')
307
- [ -n "$PR" ] && [ -n "$COMMENT" ] && [ -n "$STATUS" ] || exit 0
308
- # (The pull request's head commit, and the repository it's from.)
309
- head=$(gh api "repos/$GITHUB_REPOSITORY/pulls/$PR" --jq '"\(.head.sha) \(.head.repo.full_name)"')
310
- [ "${head%% *}" = "$HEAD_SHA" ] || exit 0
311
- # The report, as Markdown for a branch of this repository (whose
312
- # author can push here anyway), but for a fork's, as text: its run
313
- # ran the fork's code, which could have written anything in it. In
314
- # a code block, fenced with more backticks than any run of them in
315
- # the report, it can't mention anyone, link anywhere or show an
316
- # image. It's cut short where a comment couldn't hold it, and loses
317
- # any NUL bytes, which GitHub wouldn't take, and grep reads it as
318
- # text whatever's in it, so it finds every run of backticks.
319
- report() {
320
- if [ "${head#* }" = "$GITHUB_REPOSITORY" ]; then cat "$RUNNER_TEMP/report/duel.md" && return; fi
321
- text="$RUNNER_TEMP/report.txt"
322
- LC_ALL=C tr -d '\000' < "$RUNNER_TEMP/report/duel.md" | head -c 60000 > "$text"
323
- fence=$(LC_ALL=C grep -ao '``*' "$text" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
324
- printf 'From a fork, so shown as text:\n\n%s\n' "$fence"
325
- cat "$text"
326
- printf '\n%s\n' "$fence"
327
- }
328
- marker="<!-- duel"
329
- # The first line of the last comment Duel posted here: its marker,
330
- # with its status (none in Duel 0.1.0's, which counts as another).
331
- last=$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues/$PR/comments" \
332
- --jq ".[] | select(.user.login == \"github-actions[bot]\") | .body | split(\"\n\")[0] | rtrimstr(\"\r\") | select(startswith(\"$marker status=\") or . == \"$marker -->\")" | tail -n 1)
333
- if [ "$COMMENT" = post ] || { [ -n "$last" ] && [ "$last" != "$marker status=$STATUS -->" ]; }; then
334
- { echo "$marker status=$STATUS -->"; report; } > "$RUNNER_TEMP/comment.md"
335
- gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
336
- fi
310
+ gem install duel --version '~> 0.1.8' --no-document
311
+ duel comment --report "$RUNNER_TEMP/report" --head "$HEAD_SHA" --marker 'duel'
337
312
 
338
313
  again:
339
314
  name: Run Duel again
@@ -442,7 +417,10 @@ label is added to a pull request or taken off. For the label `.duel.rb`
442
417
  names, it waits for the Perf run of the pull request's head commit to
443
418
  finish. Then it runs the failed jobs again where the label was added
444
419
  to a run that failed, or the whole run again where the label was
445
- taken off one that passed. It checks nothing out and runs none of the
420
+ taken off one that passed. The run the label accepts is commented on
421
+ as one that passed would be: with the default `c.comment = :failures`,
422
+ only where the last comment's status isn't its own, so the report that
423
+ failed isn't posted again. It checks nothing out and runs none of the
446
424
  pull request's code, and its only permission is `actions: write`. The
447
425
  workflows name the label as `.duel.rb` did when `duel init` wrote them,
448
426
  so if you rename it, rename it in both workflows too.
@@ -584,16 +562,19 @@ your README can show it:
584
562
  ```
585
563
 
586
564
  The badge says "perf guarded", in green, when Duel passed the push. It
587
- says "perf regressed", in red, when the Duel job failed or found a
565
+ says "perf regressed", in red, when Duel failed the push or found a
588
566
  regression, even one that passed, with
589
567
  [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
590
- label accepted. A push with nothing to compare leaves the badge as it
591
- was, and so does a run that finishes after a newer push's, so the
592
- badge is always the newest push's that Duel compared. The branch holds
593
- one commit, with the badge alone, whose message names that push. The
594
- job checks nothing out, and it gets the `contents: write` permission
595
- that moving a branch takes. For a project in a subdirectory of its
596
- repository, the branch names the directory, such as
568
+ label accepted. It says "not measured", in grey, when the Duel job
569
+ failed before Duel could judge the push, as when its runner ran out of
570
+ memory or a step before Duel's failed. A job that fails after Duel
571
+ passed the push leaves it green. A push with nothing to compare leaves
572
+ the badge as it was, and so does a run that finishes after a newer
573
+ push's, so the badge is always the newest push's that Duel compared.
574
+ The branch holds one commit, with the badge alone, whose message names
575
+ that push. The job checks nothing out, and it gets the `contents: write`
576
+ permission that moving a branch takes. For a project in a subdirectory
577
+ of its repository, the branch names the directory, such as
597
578
  `duel/badge-gems-parser`.
598
579
 
599
580
  Until the first push to your default branch after you add the job, the
@@ -607,13 +588,15 @@ keeps `.duel.rb`), or add the job yourself:
607
588
 
608
589
  ```yaml
609
590
  # .github/workflows/perf.yml, in jobs: (with duel init --badge)
610
- # Duel's badge, for the README: green (perf guarded) or red (perf
611
- # regressed), as the last push to the default branch that Duel compared
612
- # left it. It's red where the Duel job failed or found a regression,
613
- # even one that passed (with c.fail_on = :pull_requests in .duel.rb). A
614
- # push with nothing to compare leaves it as it was, and so does a run
615
- # for an older push than the badge's. This job writes the badge, alone,
616
- # to a branch of its own, which the README shows:
591
+ # Duel's badge, for the README: green (perf guarded), red (perf
592
+ # regressed) or grey (not measured), as the last push to the default
593
+ # branch that Duel compared left it. It's red where Duel failed the push
594
+ # or found a regression, even one that passed (with
595
+ # c.fail_on = :pull_requests in .duel.rb), and grey where the Duel job
596
+ # failed before Duel could judge the push (its runner ran out of memory,
597
+ # say). A push with nothing to compare leaves it as it was, and so does
598
+ # a run for an older push than the badge's. This job writes the badge,
599
+ # alone, to a branch of its own, which the README shows:
617
600
  #
618
601
  # [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
619
602
  #
@@ -659,10 +642,13 @@ keeps `.duel.rb`), or add the job yourself:
659
642
  last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
660
643
  if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
661
644
  # (Its image, written here rather than fetched from anywhere: green,
662
- # or red where the Duel job failed or found a regression.)
645
+ # red where Duel failed the push or found a regression, or grey
646
+ # where the Duel job failed before Duel said either.)
663
647
  state="perf guarded" width=133 top="#34D058" bottom="#28A745" right="76.939C78.629 0 80 1.343 80"
664
- if [ "$RESULT" = failure ] || [ "$STATUS" = failed ]; then
648
+ if [ "$STATUS" = failed ]; then
665
649
  state="perf regressed" width=142 top="#EA4A5A" bottom="#D73A49" right="85.939C87.629 0 89 1.343 89"
650
+ elif [ "$RESULT" = failure ] && [ "$STATUS" != passed ]; then
651
+ state="not measured" width=138 top="#959DA5" bottom="#6A737D" right="81.939C83.629 0 85 1.343 85"
666
652
  fi
667
653
  cat > "$RUNNER_TEMP/badge.svg" <<SVG
668
654
  <svg xmlns="http://www.w3.org/2000/svg" width="$width" height="20">
@@ -732,6 +718,27 @@ that branch, which a branch of its own marks: `duel/passed/2-x-stable`.
732
718
  A pull request into the branch is compared with the branch, whatever
733
719
  the workflow guards.
734
720
 
721
+ ## Keep the workflows up to date
722
+
723
+ A release of Duel can change the workflows that `duel init` writes. To
724
+ see how yours differ from what the installed Duel writes, run it with
725
+ `--check`, and the options you ran it with (such as `--badge`):
726
+
727
+ ```sh
728
+ bundle exec duel init --check
729
+ ```
730
+
731
+ It says whether each workflow is the same, differs (and shows how, as
732
+ `diff -u` does), or is missing, and writes nothing. It exits with 1
733
+ where one isn't the same. Copy in the changes you want, so that steps
734
+ you added to the Duel job (to install a database server, say) stay.
735
+ With `--gitlab`, it checks the GitLab CI jobs. In a Rails app, run
736
+ duel-rails's generator with `--check` instead:
737
+
738
+ ```sh
739
+ bin/rails generate duel:install --check
740
+ ```
741
+
735
742
  ## Run Duel on GitLab CI
736
743
 
737
744
  `duel init --gitlab` writes GitLab CI jobs, in
@@ -776,9 +783,22 @@ log and its `duel.md` artifact have the report.
776
783
  protected, so only your protected branches' pipelines have it, to
777
784
  comment on a pushed commit and to mark a push passed.
778
785
 
779
- The comment on a pushed commit looks only at the comments on the commit
780
- before it for its last status. A merge request from a fork runs its
781
- pipeline in the fork, without your variables, so it gets no comment.
786
+ The comment on a merge request comes from `duel comment --gitlab`,
787
+ installed from RubyGems in a `ruby:slim` image, as on GitHub, so a fix
788
+ to it comes with a release of Duel. It trusts only a `DUEL_COMMENT` and
789
+ a `DUEL_STATUS` that Duel says, and comments only if the merge
790
+ request's head is still the one the pipeline compared. In a merged
791
+ results pipeline, the report names the merge request's own commit,
792
+ "merged with `main`", as it does on GitHub.
793
+
794
+ Three things GitLab does differently. Duel can't collapse its earlier
795
+ comments there, because only the Developer role can resolve a thread,
796
+ and the merge request's token has the Reporter role. Adding the label
797
+ that accepts a regression doesn't run the pipeline again, as GitLab has
798
+ no pipeline for a label, so run it again yourself (or push). And a
799
+ merge request from a fork runs its pipeline in the fork, without your
800
+ variables, so it gets no comment. The comment on a pushed commit looks
801
+ only at the comments on the commit before it for its last status.
782
802
 
783
803
  ```yaml
784
804
  # .gitlab/duel.gitlab-ci.yml
@@ -845,9 +865,11 @@ duel:
845
865
  # The report, as a new comment on the merge request, when Duel says to:
846
866
  # always (post), or only when its last comment's status differs from this
847
867
  # run's (change), so the latest comment is never stale. It never edits or
848
- # deletes a comment.
868
+ # deletes a comment. It runs `duel comment`, from RubyGems (none of the
869
+ # merge request's code), so a fix to how it comments comes with a release
870
+ # of Duel.
849
871
  duel:comment:
850
- image: alpine:3
872
+ image: ruby:slim
851
873
  stage: .post
852
874
  needs: [duel]
853
875
  rules:
@@ -856,23 +878,10 @@ duel:comment:
856
878
  variables:
857
879
  GIT_STRATEGY: none # it runs none of the code
858
880
  script:
859
- - apk add --no-cache --quiet curl jq
860
- - |
861
- case "$DUEL_COMMENT" in post|change) ;; *) exit 0 ;; esac
862
- report=duel.md
863
- [ -f "$report" ] || exit 0
864
- api="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
865
- token="PRIVATE-TOKEN: $DUEL_COMMENT_TOKEN"
866
- marker="<!-- duel"
867
- me=$(curl -sf -H "$token" "$CI_API_V4_URL/user" | jq .id)
868
- # The first line of the last comment Duel posted here: its marker,
869
- # with its status.
870
- last=$(curl -sf -H "$token" "$api/merge_requests/$CI_MERGE_REQUEST_IID/notes?sort=desc&order_by=created_at&per_page=100" |
871
- jq -r --argjson me "$me" --arg marker "$marker status=" '.[] | select(.author.id == $me) | .body | split("\n")[0] | rtrimstr("\r") | select(startswith($marker))' | head -n 1)
872
- if [ "$DUEL_COMMENT" = post ] || { [ -n "$last" ] && [ "$last" != "$marker status=$DUEL_STATUS -->" ]; }; then
873
- { echo "$marker status=$DUEL_STATUS -->"; cat "$report"; } | jq -Rs '{body: .}' |
874
- curl -sf -H "$token" -H "Content-Type: application/json" --data @- "$api/merge_requests/$CI_MERGE_REQUEST_IID/notes" > /dev/null
875
- fi
881
+ - gem install duel --version '~> 0.1.8' --no-document
882
+ # (The merge request's head: in a merged results pipeline, not the
883
+ # commit that merges it.)
884
+ - duel comment --gitlab --report . --head "${CI_MERGE_REQUEST_SOURCE_BRANCH_SHA:-$CI_COMMIT_SHA}" --marker 'duel'
876
885
 
877
886
  # The report, as a new comment on a pushed commit, when Duel says to, as
878
887
  # for a merge request: only where a regression in a push passes (see
@@ -139,7 +139,9 @@ end
139
139
  On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
140
140
  that `duel init` writes passes Duel the pull request's labels. With
141
141
  that label, a regression passes, with a warning, and the report says
142
- that the label accepted it. The labels are read when the workflow runs,
142
+ that the label accepted it. It comments on that run as it would on one
143
+ that passed, so by default, after a comment on the run that failed, it
144
+ doesn't comment again. The labels are read when the workflow runs,
143
145
  so the comment workflow runs the Duel job again when you add the label
144
146
  to a pull request whose run failed, or take it off one whose run
145
147
  passed. For a push, Duel reads the labels of the pull request that
data/docs/how-it-works.md CHANGED
@@ -51,6 +51,23 @@ never on noise, so that it can gate every pull request:
51
51
  independent already. It can also stop after any pair once the
52
52
  interval shows that it isn't slower, which only ever ends a
53
53
  measurement before it could find a slowdown.
54
+
55
+ Duel measures one benchmark at a time for every two CPUs
56
+ (`--jobs`), each with workers of its own, so a run can start many
57
+ workers, and each one's resources (a database server, say) with it.
58
+ Where they wouldn't fit in the memory available (Linux's
59
+ MemAvailable, or a container's cgroup limit, or macOS's free and
60
+ inactive pages), Duel measures fewer benchmarks at once, and where
61
+ even one benchmark's workers wouldn't fit, it starts fewer workers.
62
+ Running out would make the machine swap, or stop a CI runner from
63
+ answering, so its job is lost. To find what they take, Duel starts
64
+ one worker of each version first, and allows each pair a quarter
65
+ more than that (on TruffleRuby, whose workers grow as its JIT warms
66
+ up, 2.5 times that), or what its plugins say each worker's resources
67
+ take at most, if that's more (a server takes more as it's used), and
68
+ 1 GB for everything else. Its first line says
69
+ when it lowered either. A `--jobs` or `--processes` you give is
70
+ kept as it is.
54
71
  - **Measured again before failing.** Duel measures whatever looks
55
72
  slower again from scratch, up to three times, in fresh processes. It
56
73
  fails only if every measurement finds the same metric worse. A
data/docs/json.md CHANGED
@@ -243,8 +243,8 @@ was slower across the board and confirmed (`"confirmed": true`).
243
243
  | `confidence` | The confidence of each interval, such as `0.99`. |
244
244
  | `limit` | The ratio above which a timing is slower (`--limit`). A metric can have its own limit, which its timings show. |
245
245
  | `filter` | The regular expression that picked the benchmarks (`--filter`), as source. `""` matches every one. |
246
- | `jobs` | Benchmarks measured at once (`--jobs`). |
247
- | `processes` | Worker processes for each version (`--processes`). |
246
+ | `jobs` | Benchmarks measured at once (`--jobs`), or fewer, where that many wouldn't fit in the memory available. |
247
+ | `processes` | Worker processes for each version (`--processes`), or fewer, where even one benchmark's wouldn't fit in the memory available. |
248
248
  | `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
249
249
  | `time_limit` | The seconds that a run measures for, at most (`--time-limit`). |
250
250
  | `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default: until steady, for up to as long as the Ruby takes. |
data/docs/plugins.md CHANGED
@@ -77,6 +77,13 @@ When you write a plugin, follow these guidelines:
77
77
  - Set up what a whole version needs once, such as its bundle, with
78
78
  `c.prepare(:version)`, not in each worker's `c.prepare(:worker)`. Run
79
79
  its commands on the version's Ruby: `v.env` puts it first on `PATH`.
80
+ - If each worker's `c.prepare(:worker)` starts something that takes a
81
+ lot of memory once it's used, such as a database server, say how much
82
+ at most, in bytes, as the module's `worker_memory`, for example
83
+ `def self.worker_memory = 450 << 20`. Duel fits its workers in the
84
+ memory available by what one of each version takes as it starts, and
85
+ a server takes more as it's used, so Duel counts at least this much
86
+ for each worker. A version of Duel before 0.1.8 ignores it.
80
87
 
81
88
  > [!IMPORTANT]
82
89
  > A helper that's meant for setup, such as one that runs a query, must