duel 0.1.6 → 0.1.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENT_PROMPT.md +2 -2
- data/CHANGELOG.md +83 -1
- data/README.md +6 -1
- data/docs/ci.md +102 -93
- data/docs/configuration.md +3 -1
- data/docs/how-it-works.md +17 -0
- data/docs/json.md +2 -2
- data/docs/plugins.md +7 -0
- data/lib/duel/cli.rb +60 -11
- data/lib/duel/comment.rb +235 -0
- data/lib/duel/config.rb +11 -2
- data/lib/duel/init.rb +116 -5
- data/lib/duel/memory.rb +54 -0
- data/lib/duel/runner.rb +63 -2
- data/lib/duel/summary.rb +11 -5
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +8 -19
- data/lib/duel/templates/perf-comment.yml.erb +17 -42
- data/lib/duel/templates/perf.yml.erb +14 -9
- data/lib/duel/version.rb +1 -1
- data/lib/duel.rb +8 -0
- data/sig/duel/cli.rbs +11 -0
- data/sig/duel/comment.rbs +82 -0
- data/sig/duel/config.rbs +5 -0
- data/sig/duel/init.rbs +39 -1
- data/sig/duel/memory.rbs +30 -0
- data/sig/duel/runner.rbs +22 -2
- data/sig/duel/summary.rbs +3 -1
- metadata +5 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e153495685151ec755386e3fb34aeb21d3f9e3cb2015d4900b6d09f611939a25
|
|
4
|
+
data.tar.gz: 30c26bc5ca7ac5d36c1cb438ccfb4886fa4e9b4fb0637188b9a656dd15d9b231
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3b902a527e27cce62d6ac90883410da63a3842c727b81656983031235baf44b27e7e28fe3b3de69ec9f7554215b94d74643af41d5f569c0567ef78e423625d70
|
|
7
|
+
data.tar.gz: 3e0255a630de657157d36087f12144d6dd2e0b556a06afd87d86e7ac0cd43fc7047522ec83431dd33369a96d083d546e2fa5608fb247970d5993cf0a2288174f
|
data/AGENT_PROMPT.md
CHANGED
|
@@ -201,8 +201,8 @@ at the API.
|
|
|
201
201
|
- `.github/workflows/perf-comment.yml` comments on a pull request
|
|
202
202
|
with Duel's report once the first has run (`c.comment` in
|
|
203
203
|
`.duel.rb` chooses when, and Duel also adds the report to the
|
|
204
|
-
job's summary). It runs none of the pull request's code
|
|
205
|
-
|
|
204
|
+
job's summary). It runs none of the pull request's code. Keep it as
|
|
205
|
+
it is.
|
|
206
206
|
- `duel init` writes the jobs for pushes (a comment on a pushed
|
|
207
207
|
commit, and a `duel/passed/` branch that marks the last commit
|
|
208
208
|
Duel passed) only where `.duel.rb` sets `c.fail_on =
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,86 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.8] - 2026-11-12
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- `duel comment`, which the Perf comment workflow now runs instead of
|
|
15
|
+
a shell script: it installs Duel from RubyGems (the newest release of
|
|
16
|
+
the same minor version) and comments with it, so a fix to how it
|
|
17
|
+
comments comes with a release of Duel, without a change to the
|
|
18
|
+
workflow. `duel init --check` shows how a project's workflows differ
|
|
19
|
+
from what the installed Duel writes.
|
|
20
|
+
- Once the Perf comment workflow posts a comment, it hides Duel's
|
|
21
|
+
earlier ones on the pull request as outdated (those not hidden
|
|
22
|
+
already), which GitHub collapses, so the newest is the one in view.
|
|
23
|
+
- `duel init` offers to create the label that `c.accept_label` names,
|
|
24
|
+
with `gh`, where it can (at a terminal, in a GitHub repository that
|
|
25
|
+
hasn't the label). Where it doesn't, its next steps say to.
|
|
26
|
+
- A plugin can say how much memory what it starts for each worker
|
|
27
|
+
takes at most (`def self.worker_memory`), and Duel counts at least
|
|
28
|
+
that much for each worker as it fits them in the memory available. A
|
|
29
|
+
server takes more as it's used than as it starts, when Duel measures
|
|
30
|
+
it.
|
|
31
|
+
- `duel comment --gitlab`, which the GitLab CI jobs' comment job now
|
|
32
|
+
runs instead of a shell script, as the GitHub workflow runs `duel
|
|
33
|
+
comment`: it trusts only what Duel said, comments only if the merge
|
|
34
|
+
request's head is still the run's, and comes from RubyGems, so a fix
|
|
35
|
+
comes with a release. In a merged results pipeline, the report names
|
|
36
|
+
the merge request's own commit, as on GitHub, and `duel init --gitlab`
|
|
37
|
+
offers to create the accept label with `glab`.
|
|
38
|
+
|
|
39
|
+
### Changed
|
|
40
|
+
|
|
41
|
+
- On a pull request's run on GitHub Actions, the report names the pull
|
|
42
|
+
request's own commit, "merged with `main`", instead of the commit
|
|
43
|
+
that merges it, which GitHub makes for the run and isn't in the pull
|
|
44
|
+
request.
|
|
45
|
+
- A run whose failures `c.accept_label` accepted says so in its
|
|
46
|
+
heading, under 🟡 instead of 🔻: "1 benchmark failed, accepted".
|
|
47
|
+
- The Perf comment workflow posts a report from a fork's pull request
|
|
48
|
+
as Markdown, like any other, instead of as text in a code block.
|
|
49
|
+
- A regression that `c.accept_label` accepts is commented on as a run
|
|
50
|
+
that passed would be. With the default `c.comment = :failures`, Duel
|
|
51
|
+
no longer posts the report again when the label is added, since the
|
|
52
|
+
last comment already showed it failing.
|
|
53
|
+
|
|
54
|
+
### Fixed
|
|
55
|
+
|
|
56
|
+
- On TruffleRuby, Duel fits the workers in the memory available as
|
|
57
|
+
they'll be once warm, not as they start: its workers grow some 2.9
|
|
58
|
+
times as its JIT compiles, so where 8 of each version seemed to fit,
|
|
59
|
+
they took 24 GB, more than a 16 GB CI runner has, and the runner
|
|
60
|
+
stopped answering.
|
|
61
|
+
|
|
62
|
+
To pick up the Perf comment workflow's changes, run `duel init --check`
|
|
63
|
+
(or see duel-rails's generator's template), and copy them into
|
|
64
|
+
`.github/workflows/perf-comment.yml`.
|
|
65
|
+
|
|
66
|
+
## [0.1.7] - 2026-11-10
|
|
67
|
+
|
|
68
|
+
### Fixed
|
|
69
|
+
|
|
70
|
+
- Duel measures fewer benchmarks at once, and where even one
|
|
71
|
+
benchmark's workers wouldn't fit, starts fewer workers, so that they
|
|
72
|
+
fit in the memory available. Since 0.1.6 starts twelve workers of
|
|
73
|
+
each version for each benchmark it measures at once, a plugin whose
|
|
74
|
+
workers each start a server could run a machine out of memory: on a
|
|
75
|
+
GitHub runner, duel-mysql's 48 MySQL servers took more than its
|
|
76
|
+
16 GB, and the runner stopped answering until its job was lost. Duel
|
|
77
|
+
starts one worker of each version first to find what a pair takes
|
|
78
|
+
(with its resources), and allows each pair a quarter more, and 1 GB
|
|
79
|
+
for everything else. It reads what's available from Linux's
|
|
80
|
+
MemAvailable (or a container's cgroup limit, if lower) or macOS's
|
|
81
|
+
`vm_stat`, and its first line says when it lowered either. A
|
|
82
|
+
`--jobs` or `--processes` you give is kept as it is.
|
|
83
|
+
- The badge (in the workflow that `duel init --badge` writes) says
|
|
84
|
+
"not measured", in grey, where the Duel job failed
|
|
85
|
+
before Duel judged the push (its runner lost, say, or a step before
|
|
86
|
+
Duel's failed), instead of "perf regressed". It's red only where
|
|
87
|
+
Duel failed the push or found a regression, and a job that fails
|
|
88
|
+
after Duel passed the push leaves it green.
|
|
89
|
+
|
|
10
90
|
## [0.1.6] - 2026-11-10
|
|
11
91
|
|
|
12
92
|
### Added
|
|
@@ -682,7 +762,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
682
762
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
683
763
|
release.
|
|
684
764
|
|
|
685
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.
|
|
765
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.8...HEAD
|
|
766
|
+
[0.1.8]: https://github.com/sferik/duel-ruby/compare/v0.1.7...v0.1.8
|
|
767
|
+
[0.1.7]: https://github.com/sferik/duel-ruby/compare/v0.1.6...v0.1.7
|
|
686
768
|
[0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
|
|
687
769
|
[0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
|
|
688
770
|
[0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
|
data/README.md
CHANGED
|
@@ -312,7 +312,12 @@ close to the limit to call doesn't fail the run, and is in the details.
|
|
|
312
312
|
A benchmark that the baseline can't run (a new one, say) is skipped, and
|
|
313
313
|
the heading counts it. A run that has nothing to compare, or no release
|
|
314
314
|
to compare against, says so under ⬜, and one that couldn't compare
|
|
315
|
-
says why under 🔻.
|
|
315
|
+
says why under 🔻. One whose failures the pull request's label accepted
|
|
316
|
+
(see [`c.accept_label`](docs/configuration.md#accept-a-regression-with-a-label))
|
|
317
|
+
says so under 🟡. On a pull request, GitHub Actions checks out a commit
|
|
318
|
+
that merges it with the branch it targets, which isn't in the pull
|
|
319
|
+
request, so the report names the pull request's own commit, "merged
|
|
320
|
+
with `main`".
|
|
316
321
|
|
|
317
322
|
The workflow that `duel init` writes posts this report as a comment on
|
|
318
323
|
the pull request when a benchmark fails, or when Duel couldn't compare.
|
data/docs/ci.md
CHANGED
|
@@ -193,7 +193,9 @@ check, because `duel` exits with a nonzero status, and the job's
|
|
|
193
193
|
summary has the report.
|
|
194
194
|
|
|
195
195
|
The workflow never edits or deletes a comment, so the pull request
|
|
196
|
-
keeps every report that it posted.
|
|
196
|
+
keeps every report that it posted. Once it posts one, it hides its
|
|
197
|
+
earlier ones as outdated, which GitHub collapses, so the newest is the
|
|
198
|
+
one in view. Each comment starts with a hidden
|
|
197
199
|
marker that records the run's status: `<!-- duel status=passed -->`,
|
|
198
200
|
`status=failed` (a run that couldn't compare too), or `status=neutral`
|
|
199
201
|
(nothing to compare). When a run's outcome wouldn't get a comment of
|
|
@@ -228,16 +230,14 @@ the pull request's number, `comment`, and `status`. A second workflow,
|
|
|
228
230
|
the Perf workflow completes (`workflow_run`), with your repository's
|
|
229
231
|
own copy of the workflow and a token that can write, so a pull request
|
|
230
232
|
from a fork gets a comment too. It checks nothing out and runs none of
|
|
231
|
-
the pull request's code.
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
`github-actions[bot]`, it can't mention anyone, link anywhere or show an
|
|
240
|
-
image. A report too long for a comment is cut short. When the Duel job fails before it compares, there's
|
|
233
|
+
the pull request's code. It installs Duel from RubyGems (the newest
|
|
234
|
+
0.1 release, 0.1.8 or later) and runs `duel comment`, so a fix to how it comments comes
|
|
235
|
+
with a release of Duel, without a change to the workflow. The pull
|
|
236
|
+
request's code ran in the Duel job, and could have written anything
|
|
237
|
+
into the artifact, so `duel comment` trusts only what Duel could have
|
|
238
|
+
said there: a number that names the pull request whose head the run
|
|
239
|
+
compared, and a `comment` and a `status` that Duel uses. The report's
|
|
240
|
+
Markdown, a fork's too, is the comment's body. When the Duel job fails before it compares, there's
|
|
241
241
|
no report, and no comment. GitHub runs a `workflow_run` workflow only
|
|
242
242
|
from your default branch, so the pull request that adds the workflow
|
|
243
243
|
gets no comment.
|
|
@@ -249,14 +249,15 @@ name: Perf comment
|
|
|
249
249
|
# Duel's report, as a new comment on the pull request, when Duel says to:
|
|
250
250
|
# always (post), or only when its last comment's status differs from the
|
|
251
251
|
# run's (change), so the latest comment is never stale. It never edits or
|
|
252
|
-
# deletes a comment
|
|
253
|
-
#
|
|
254
|
-
#
|
|
255
|
-
#
|
|
256
|
-
#
|
|
257
|
-
#
|
|
258
|
-
#
|
|
259
|
-
#
|
|
252
|
+
# deletes a comment, but hides Duel's earlier ones as outdated, so the
|
|
253
|
+
# newest is the one in view. It runs once the Perf workflow has, with
|
|
254
|
+
# this repository's own workflow and a token that can write, so a pull
|
|
255
|
+
# request from a fork gets a comment too. It checks nothing out and runs
|
|
256
|
+
# none of the pull request's code, only `duel comment` (see Duel's
|
|
257
|
+
# docs/ci.md), which trusts only what the Duel job could have said in
|
|
258
|
+
# what it kept: the pull request's number, if it names the pull request
|
|
259
|
+
# the run compared, and the comment and the status, if they're ones Duel
|
|
260
|
+
# says. Its report, a fork's too, is the comment, as Markdown.
|
|
260
261
|
#
|
|
261
262
|
# And when the label that accepts a regression (c.accept_label in
|
|
262
263
|
# .duel.rb) is added to a pull request, or taken off, it runs the Duel
|
|
@@ -294,46 +295,20 @@ jobs:
|
|
|
294
295
|
path: ${{ runner.temp }}/report
|
|
295
296
|
run-id: ${{ github.event.workflow_run.id }}
|
|
296
297
|
github-token: ${{ github.token }}
|
|
298
|
+
- uses: ruby/setup-ruby@v1
|
|
299
|
+
if: steps.report.outcome == 'success'
|
|
300
|
+
with:
|
|
301
|
+
ruby-version: ruby
|
|
302
|
+
# Duel's own code, from RubyGems (none of the pull request's), so a
|
|
303
|
+
# fix to how it comments comes with a release of Duel.
|
|
297
304
|
- name: Comment on the pull request
|
|
298
305
|
if: steps.report.outcome == 'success'
|
|
299
306
|
env:
|
|
300
307
|
GH_TOKEN: ${{ github.token }}
|
|
301
308
|
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
|
|
302
309
|
run: |
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
COMMENT=$(said comment 'post|change')
|
|
306
|
-
STATUS=$(said status 'passed|failed|neutral')
|
|
307
|
-
[ -n "$PR" ] && [ -n "$COMMENT" ] && [ -n "$STATUS" ] || exit 0
|
|
308
|
-
# (The pull request's head commit, and the repository it's from.)
|
|
309
|
-
head=$(gh api "repos/$GITHUB_REPOSITORY/pulls/$PR" --jq '"\(.head.sha) \(.head.repo.full_name)"')
|
|
310
|
-
[ "${head%% *}" = "$HEAD_SHA" ] || exit 0
|
|
311
|
-
# The report, as Markdown for a branch of this repository (whose
|
|
312
|
-
# author can push here anyway), but for a fork's, as text: its run
|
|
313
|
-
# ran the fork's code, which could have written anything in it. In
|
|
314
|
-
# a code block, fenced with more backticks than any run of them in
|
|
315
|
-
# the report, it can't mention anyone, link anywhere or show an
|
|
316
|
-
# image. It's cut short where a comment couldn't hold it, and loses
|
|
317
|
-
# any NUL bytes, which GitHub wouldn't take, and grep reads it as
|
|
318
|
-
# text whatever's in it, so it finds every run of backticks.
|
|
319
|
-
report() {
|
|
320
|
-
if [ "${head#* }" = "$GITHUB_REPOSITORY" ]; then cat "$RUNNER_TEMP/report/duel.md" && return; fi
|
|
321
|
-
text="$RUNNER_TEMP/report.txt"
|
|
322
|
-
LC_ALL=C tr -d '\000' < "$RUNNER_TEMP/report/duel.md" | head -c 60000 > "$text"
|
|
323
|
-
fence=$(LC_ALL=C grep -ao '``*' "$text" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
|
|
324
|
-
printf 'From a fork, so shown as text:\n\n%s\n' "$fence"
|
|
325
|
-
cat "$text"
|
|
326
|
-
printf '\n%s\n' "$fence"
|
|
327
|
-
}
|
|
328
|
-
marker="<!-- duel"
|
|
329
|
-
# The first line of the last comment Duel posted here: its marker,
|
|
330
|
-
# with its status (none in Duel 0.1.0's, which counts as another).
|
|
331
|
-
last=$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues/$PR/comments" \
|
|
332
|
-
--jq ".[] | select(.user.login == \"github-actions[bot]\") | .body | split(\"\n\")[0] | rtrimstr(\"\r\") | select(startswith(\"$marker status=\") or . == \"$marker -->\")" | tail -n 1)
|
|
333
|
-
if [ "$COMMENT" = post ] || { [ -n "$last" ] && [ "$last" != "$marker status=$STATUS -->" ]; }; then
|
|
334
|
-
{ echo "$marker status=$STATUS -->"; report; } > "$RUNNER_TEMP/comment.md"
|
|
335
|
-
gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
|
|
336
|
-
fi
|
|
310
|
+
gem install duel --version '~> 0.1.8' --no-document
|
|
311
|
+
duel comment --report "$RUNNER_TEMP/report" --head "$HEAD_SHA" --marker 'duel'
|
|
337
312
|
|
|
338
313
|
again:
|
|
339
314
|
name: Run Duel again
|
|
@@ -442,7 +417,10 @@ label is added to a pull request or taken off. For the label `.duel.rb`
|
|
|
442
417
|
names, it waits for the Perf run of the pull request's head commit to
|
|
443
418
|
finish. Then it runs the failed jobs again where the label was added
|
|
444
419
|
to a run that failed, or the whole run again where the label was
|
|
445
|
-
taken off one that passed.
|
|
420
|
+
taken off one that passed. The run the label accepts is commented on
|
|
421
|
+
as one that passed would be: with the default `c.comment = :failures`,
|
|
422
|
+
only where the last comment's status isn't its own, so the report that
|
|
423
|
+
failed isn't posted again. It checks nothing out and runs none of the
|
|
446
424
|
pull request's code, and its only permission is `actions: write`. The
|
|
447
425
|
workflows name the label as `.duel.rb` did when `duel init` wrote them,
|
|
448
426
|
so if you rename it, rename it in both workflows too.
|
|
@@ -584,16 +562,19 @@ your README can show it:
|
|
|
584
562
|
```
|
|
585
563
|
|
|
586
564
|
The badge says "perf guarded", in green, when Duel passed the push. It
|
|
587
|
-
says "perf regressed", in red, when
|
|
565
|
+
says "perf regressed", in red, when Duel failed the push or found a
|
|
588
566
|
regression, even one that passed, with
|
|
589
567
|
[`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
|
|
590
|
-
label accepted.
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
568
|
+
label accepted. It says "not measured", in grey, when the Duel job
|
|
569
|
+
failed before Duel could judge the push, as when its runner ran out of
|
|
570
|
+
memory or a step before Duel's failed. A job that fails after Duel
|
|
571
|
+
passed the push leaves it green. A push with nothing to compare leaves
|
|
572
|
+
the badge as it was, and so does a run that finishes after a newer
|
|
573
|
+
push's, so the badge is always the newest push's that Duel compared.
|
|
574
|
+
The branch holds one commit, with the badge alone, whose message names
|
|
575
|
+
that push. The job checks nothing out, and it gets the `contents: write`
|
|
576
|
+
permission that moving a branch takes. For a project in a subdirectory
|
|
577
|
+
of its repository, the branch names the directory, such as
|
|
597
578
|
`duel/badge-gems-parser`.
|
|
598
579
|
|
|
599
580
|
Until the first push to your default branch after you add the job, the
|
|
@@ -607,13 +588,15 @@ keeps `.duel.rb`), or add the job yourself:
|
|
|
607
588
|
|
|
608
589
|
```yaml
|
|
609
590
|
# .github/workflows/perf.yml, in jobs: (with duel init --badge)
|
|
610
|
-
# Duel's badge, for the README: green (perf guarded)
|
|
611
|
-
# regressed), as the last push to the default
|
|
612
|
-
# left it. It's red where
|
|
613
|
-
# even one that passed (with
|
|
614
|
-
#
|
|
615
|
-
#
|
|
616
|
-
# to
|
|
591
|
+
# Duel's badge, for the README: green (perf guarded), red (perf
|
|
592
|
+
# regressed) or grey (not measured), as the last push to the default
|
|
593
|
+
# branch that Duel compared left it. It's red where Duel failed the push
|
|
594
|
+
# or found a regression, even one that passed (with
|
|
595
|
+
# c.fail_on = :pull_requests in .duel.rb), and grey where the Duel job
|
|
596
|
+
# failed before Duel could judge the push (its runner ran out of memory,
|
|
597
|
+
# say). A push with nothing to compare leaves it as it was, and so does
|
|
598
|
+
# a run for an older push than the badge's. This job writes the badge,
|
|
599
|
+
# alone, to a branch of its own, which the README shows:
|
|
617
600
|
#
|
|
618
601
|
# [](https://github.com/sferik/duel-ruby)
|
|
619
602
|
#
|
|
@@ -659,10 +642,13 @@ keeps `.duel.rb`), or add the job yourself:
|
|
|
659
642
|
last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
|
|
660
643
|
if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
|
|
661
644
|
# (Its image, written here rather than fetched from anywhere: green,
|
|
662
|
-
#
|
|
645
|
+
# red where Duel failed the push or found a regression, or grey
|
|
646
|
+
# where the Duel job failed before Duel said either.)
|
|
663
647
|
state="perf guarded" width=133 top="#34D058" bottom="#28A745" right="76.939C78.629 0 80 1.343 80"
|
|
664
|
-
if [ "$
|
|
648
|
+
if [ "$STATUS" = failed ]; then
|
|
665
649
|
state="perf regressed" width=142 top="#EA4A5A" bottom="#D73A49" right="85.939C87.629 0 89 1.343 89"
|
|
650
|
+
elif [ "$RESULT" = failure ] && [ "$STATUS" != passed ]; then
|
|
651
|
+
state="not measured" width=138 top="#959DA5" bottom="#6A737D" right="81.939C83.629 0 85 1.343 85"
|
|
666
652
|
fi
|
|
667
653
|
cat > "$RUNNER_TEMP/badge.svg" <<SVG
|
|
668
654
|
<svg xmlns="http://www.w3.org/2000/svg" width="$width" height="20">
|
|
@@ -732,6 +718,27 @@ that branch, which a branch of its own marks: `duel/passed/2-x-stable`.
|
|
|
732
718
|
A pull request into the branch is compared with the branch, whatever
|
|
733
719
|
the workflow guards.
|
|
734
720
|
|
|
721
|
+
## Keep the workflows up to date
|
|
722
|
+
|
|
723
|
+
A release of Duel can change the workflows that `duel init` writes. To
|
|
724
|
+
see how yours differ from what the installed Duel writes, run it with
|
|
725
|
+
`--check`, and the options you ran it with (such as `--badge`):
|
|
726
|
+
|
|
727
|
+
```sh
|
|
728
|
+
bundle exec duel init --check
|
|
729
|
+
```
|
|
730
|
+
|
|
731
|
+
It says whether each workflow is the same, differs (and shows how, as
|
|
732
|
+
`diff -u` does), or is missing, and writes nothing. It exits with 1
|
|
733
|
+
where one isn't the same. Copy in the changes you want, so that steps
|
|
734
|
+
you added to the Duel job (to install a database server, say) stay.
|
|
735
|
+
With `--gitlab`, it checks the GitLab CI jobs. In a Rails app, run
|
|
736
|
+
duel-rails's generator with `--check` instead:
|
|
737
|
+
|
|
738
|
+
```sh
|
|
739
|
+
bin/rails generate duel:install --check
|
|
740
|
+
```
|
|
741
|
+
|
|
735
742
|
## Run Duel on GitLab CI
|
|
736
743
|
|
|
737
744
|
`duel init --gitlab` writes GitLab CI jobs, in
|
|
@@ -776,9 +783,22 @@ log and its `duel.md` artifact have the report.
|
|
|
776
783
|
protected, so only your protected branches' pipelines have it, to
|
|
777
784
|
comment on a pushed commit and to mark a push passed.
|
|
778
785
|
|
|
779
|
-
The comment on a
|
|
780
|
-
|
|
781
|
-
|
|
786
|
+
The comment on a merge request comes from `duel comment --gitlab`,
|
|
787
|
+
installed from RubyGems in a `ruby:slim` image, as on GitHub, so a fix
|
|
788
|
+
to it comes with a release of Duel. It trusts only a `DUEL_COMMENT` and
|
|
789
|
+
a `DUEL_STATUS` that Duel says, and comments only if the merge
|
|
790
|
+
request's head is still the one the pipeline compared. In a merged
|
|
791
|
+
results pipeline, the report names the merge request's own commit,
|
|
792
|
+
"merged with `main`", as it does on GitHub.
|
|
793
|
+
|
|
794
|
+
Three things GitLab does differently. Duel can't collapse its earlier
|
|
795
|
+
comments there, because only the Developer role can resolve a thread,
|
|
796
|
+
and the merge request's token has the Reporter role. Adding the label
|
|
797
|
+
that accepts a regression doesn't run the pipeline again, as GitLab has
|
|
798
|
+
no pipeline for a label, so run it again yourself (or push). And a
|
|
799
|
+
merge request from a fork runs its pipeline in the fork, without your
|
|
800
|
+
variables, so it gets no comment. The comment on a pushed commit looks
|
|
801
|
+
only at the comments on the commit before it for its last status.
|
|
782
802
|
|
|
783
803
|
```yaml
|
|
784
804
|
# .gitlab/duel.gitlab-ci.yml
|
|
@@ -845,9 +865,11 @@ duel:
|
|
|
845
865
|
# The report, as a new comment on the merge request, when Duel says to:
|
|
846
866
|
# always (post), or only when its last comment's status differs from this
|
|
847
867
|
# run's (change), so the latest comment is never stale. It never edits or
|
|
848
|
-
# deletes a comment.
|
|
868
|
+
# deletes a comment. It runs `duel comment`, from RubyGems (none of the
|
|
869
|
+
# merge request's code), so a fix to how it comments comes with a release
|
|
870
|
+
# of Duel.
|
|
849
871
|
duel:comment:
|
|
850
|
-
image:
|
|
872
|
+
image: ruby:slim
|
|
851
873
|
stage: .post
|
|
852
874
|
needs: [duel]
|
|
853
875
|
rules:
|
|
@@ -856,23 +878,10 @@ duel:comment:
|
|
|
856
878
|
variables:
|
|
857
879
|
GIT_STRATEGY: none # it runs none of the code
|
|
858
880
|
script:
|
|
859
|
-
-
|
|
860
|
-
|
|
861
|
-
|
|
862
|
-
|
|
863
|
-
[ -f "$report" ] || exit 0
|
|
864
|
-
api="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
|
|
865
|
-
token="PRIVATE-TOKEN: $DUEL_COMMENT_TOKEN"
|
|
866
|
-
marker="<!-- duel"
|
|
867
|
-
me=$(curl -sf -H "$token" "$CI_API_V4_URL/user" | jq .id)
|
|
868
|
-
# The first line of the last comment Duel posted here: its marker,
|
|
869
|
-
# with its status.
|
|
870
|
-
last=$(curl -sf -H "$token" "$api/merge_requests/$CI_MERGE_REQUEST_IID/notes?sort=desc&order_by=created_at&per_page=100" |
|
|
871
|
-
jq -r --argjson me "$me" --arg marker "$marker status=" '.[] | select(.author.id == $me) | .body | split("\n")[0] | rtrimstr("\r") | select(startswith($marker))' | head -n 1)
|
|
872
|
-
if [ "$DUEL_COMMENT" = post ] || { [ -n "$last" ] && [ "$last" != "$marker status=$DUEL_STATUS -->" ]; }; then
|
|
873
|
-
{ echo "$marker status=$DUEL_STATUS -->"; cat "$report"; } | jq -Rs '{body: .}' |
|
|
874
|
-
curl -sf -H "$token" -H "Content-Type: application/json" --data @- "$api/merge_requests/$CI_MERGE_REQUEST_IID/notes" > /dev/null
|
|
875
|
-
fi
|
|
881
|
+
- gem install duel --version '~> 0.1.8' --no-document
|
|
882
|
+
# (The merge request's head: in a merged results pipeline, not the
|
|
883
|
+
# commit that merges it.)
|
|
884
|
+
- duel comment --gitlab --report . --head "${CI_MERGE_REQUEST_SOURCE_BRANCH_SHA:-$CI_COMMIT_SHA}" --marker 'duel'
|
|
876
885
|
|
|
877
886
|
# The report, as a new comment on a pushed commit, when Duel says to, as
|
|
878
887
|
# for a merge request: only where a regression in a push passes (see
|
data/docs/configuration.md
CHANGED
|
@@ -139,7 +139,9 @@ end
|
|
|
139
139
|
On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
|
|
140
140
|
that `duel init` writes passes Duel the pull request's labels. With
|
|
141
141
|
that label, a regression passes, with a warning, and the report says
|
|
142
|
-
that the label accepted it.
|
|
142
|
+
that the label accepted it. It comments on that run as it would on one
|
|
143
|
+
that passed, so by default, after a comment on the run that failed, it
|
|
144
|
+
doesn't comment again. The labels are read when the workflow runs,
|
|
143
145
|
so the comment workflow runs the Duel job again when you add the label
|
|
144
146
|
to a pull request whose run failed, or take it off one whose run
|
|
145
147
|
passed. For a push, Duel reads the labels of the pull request that
|
data/docs/how-it-works.md
CHANGED
|
@@ -51,6 +51,23 @@ never on noise, so that it can gate every pull request:
|
|
|
51
51
|
independent already. It can also stop after any pair once the
|
|
52
52
|
interval shows that it isn't slower, which only ever ends a
|
|
53
53
|
measurement before it could find a slowdown.
|
|
54
|
+
|
|
55
|
+
Duel measures one benchmark at a time for every two CPUs
|
|
56
|
+
(`--jobs`), each with workers of its own, so a run can start many
|
|
57
|
+
workers, and each one's resources (a database server, say) with it.
|
|
58
|
+
Where they wouldn't fit in the memory available (Linux's
|
|
59
|
+
MemAvailable, or a container's cgroup limit, or macOS's free and
|
|
60
|
+
inactive pages), Duel measures fewer benchmarks at once, and where
|
|
61
|
+
even one benchmark's workers wouldn't fit, it starts fewer workers.
|
|
62
|
+
Running out would make the machine swap, or stop a CI runner from
|
|
63
|
+
answering, so its job is lost. To find what they take, Duel starts
|
|
64
|
+
one worker of each version first, and allows each pair a quarter
|
|
65
|
+
more than that (on TruffleRuby, whose workers grow as its JIT warms
|
|
66
|
+
up, 2.5 times that), or what its plugins say each worker's resources
|
|
67
|
+
take at most, if that's more (a server takes more as it's used), and
|
|
68
|
+
1 GB for everything else. Its first line says
|
|
69
|
+
when it lowered either. A `--jobs` or `--processes` you give is
|
|
70
|
+
kept as it is.
|
|
54
71
|
- **Measured again before failing.** Duel measures whatever looks
|
|
55
72
|
slower again from scratch, up to three times, in fresh processes. It
|
|
56
73
|
fails only if every measurement finds the same metric worse. A
|
data/docs/json.md
CHANGED
|
@@ -243,8 +243,8 @@ was slower across the board and confirmed (`"confirmed": true`).
|
|
|
243
243
|
| `confidence` | The confidence of each interval, such as `0.99`. |
|
|
244
244
|
| `limit` | The ratio above which a timing is slower (`--limit`). A metric can have its own limit, which its timings show. |
|
|
245
245
|
| `filter` | The regular expression that picked the benchmarks (`--filter`), as source. `""` matches every one. |
|
|
246
|
-
| `jobs` | Benchmarks measured at once (`--jobs`). |
|
|
247
|
-
| `processes` | Worker processes for each version (`--processes`). |
|
|
246
|
+
| `jobs` | Benchmarks measured at once (`--jobs`), or fewer, where that many wouldn't fit in the memory available. |
|
|
247
|
+
| `processes` | Worker processes for each version (`--processes`), or fewer, where even one benchmark's wouldn't fit in the memory available. |
|
|
248
248
|
| `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
|
|
249
249
|
| `time_limit` | The seconds that a run measures for, at most (`--time-limit`). |
|
|
250
250
|
| `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default: until steady, for up to as long as the Ruby takes. |
|
data/docs/plugins.md
CHANGED
|
@@ -77,6 +77,13 @@ When you write a plugin, follow these guidelines:
|
|
|
77
77
|
- Set up what a whole version needs once, such as its bundle, with
|
|
78
78
|
`c.prepare(:version)`, not in each worker's `c.prepare(:worker)`. Run
|
|
79
79
|
its commands on the version's Ruby: `v.env` puts it first on `PATH`.
|
|
80
|
+
- If each worker's `c.prepare(:worker)` starts something that takes a
|
|
81
|
+
lot of memory once it's used, such as a database server, say how much
|
|
82
|
+
at most, in bytes, as the module's `worker_memory`, for example
|
|
83
|
+
`def self.worker_memory = 450 << 20`. Duel fits its workers in the
|
|
84
|
+
memory available by what one of each version takes as it starts, and
|
|
85
|
+
a server takes more as it's used, so Duel counts at least this much
|
|
86
|
+
for each worker. A version of Duel before 0.1.8 ignores it.
|
|
80
87
|
|
|
81
88
|
> [!IMPORTANT]
|
|
82
89
|
> A helper that's meant for setup, such as one that runs a query, must
|