duel 0.1.2 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +68 -1
- data/README.md +22 -5
- data/docs/benchmarks.md +2 -1
- data/docs/ci.md +147 -12
- data/docs/configuration.md +11 -4
- data/docs/how-it-works.md +30 -4
- data/docs/json.md +2 -0
- data/docs/plugins.md +2 -2
- data/docs/troubleshooting.md +14 -0
- data/lib/duel/baseline.rb +4 -4
- data/lib/duel/cli.rb +4 -1
- data/lib/duel/config.rb +13 -6
- data/lib/duel/context.rb +6 -5
- data/lib/duel/init.rb +4 -0
- data/lib/duel/json_report.rb +4 -3
- data/lib/duel/lock.rb +52 -8
- data/lib/duel/native.rb +20 -0
- data/lib/duel/resource.rb +37 -2
- data/lib/duel/runner.rb +98 -9
- data/lib/duel/summary.rb +4 -1
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +23 -4
- data/lib/duel/templates/perf.yml.erb +83 -3
- data/lib/duel/testing.rb +2 -2
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +1 -1
- data/sig/duel/config.rbs +3 -0
- data/sig/duel/init.rbs +3 -0
- data/sig/duel/lock.rbs +11 -0
- data/sig/duel/native.rbs +2 -0
- data/sig/duel/resource.rbs +10 -1
- data/sig/duel/runner.rbs +37 -2
- data/sig/duel/summary.rbs +3 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3827f9485e70bb95c50b32ff9438042bd60cb26885d92f01fe6c1dc6de33ee72
|
|
4
|
+
data.tar.gz: 53e410c89687f6bcdec3c46be4d370409110ac134d90ef74ca8aaaad62394a7e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7710b0ab9cc84821d6e66af1d66e1005b98aa0e97a1f047e2a76d6c5d3a83defb413895e11476d067265b97d589dcbc2fb5bdf78550616da74cf92df0da2bd19
|
|
7
|
+
data.tar.gz: 6d9d0df5accaf420f0a4e554f21eb22e79e77f8865986af856759272c5358b6cacd22af27800827d6faf7360daeded8d58481a6ef3c22669279d13a85fe4ff83
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.3] - 2026-11-04
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- The workflow that `duel init` writes keeps a badge for the README
|
|
15
|
+
that turns red when Duel fails a push to the default branch. Its
|
|
16
|
+
Update the badge job writes `badge.svg` to a branch of its own,
|
|
17
|
+
`duel/badge`, green ("perf guarded") when Duel passed the push, and
|
|
18
|
+
red ("perf regressed") when the Duel job failed or found a
|
|
19
|
+
regression, even one that passed.
|
|
20
|
+
- A run measures for 30 minutes at most (`--time-limit SECONDS`), not
|
|
21
|
+
counting the builds. Past that, it starts no more measurements: a
|
|
22
|
+
benchmark it hasn't measured fails as an error, and one that looked
|
|
23
|
+
worse is judged on the measurements it has, so a run cut short never
|
|
24
|
+
passes a regression. The report (and the JSON report's `cut_short`)
|
|
25
|
+
says what it left undone. A benchmark that starts processes of its own
|
|
26
|
+
and sat near its limit could be measured again for hours before.
|
|
27
|
+
|
|
28
|
+
### Changed
|
|
29
|
+
|
|
30
|
+
- The workers, their directories and sockets, and the resources that
|
|
31
|
+
prepare blocks get are named `base` and `head` (`base-1`, `head-1`),
|
|
32
|
+
not `baseline` and `candidate`: `Resource#name` (`w.name`, `v.name`)
|
|
33
|
+
says so, and `Duel::Testing.prepare` names its worker `head-1` by
|
|
34
|
+
default. The report still says baseline and candidate.
|
|
35
|
+
- A run removes its lock (`tmp/duel/lock`) when it ends, and
|
|
36
|
+
`tmp/duel` (and `tmp`, if Duel made it) when that leaves them empty, so
|
|
37
|
+
a project whose plugins build nothing to keep has nothing left there.
|
|
38
|
+
- When a run ends, the builds that prepare blocks made once
|
|
39
|
+
(`Resource#cached`) and the run used replace the others of their name:
|
|
40
|
+
a key's name is what's before its version and the hash it ends with
|
|
41
|
+
(`mysql`, of `mysql-0.1.2-3f9c2a1b7e5d4c6a`). A database server or a
|
|
42
|
+
plugin upgraded left its old build in `tmp/duel/cache` before, tens
|
|
43
|
+
of megabytes (82 MB for MySQL's) each time.
|
|
44
|
+
|
|
45
|
+
### Fixed
|
|
46
|
+
|
|
47
|
+
- The candidate's workers could measure slower in every pair, with the
|
|
48
|
+
same instruction counts, on code that hadn't changed. Their names
|
|
49
|
+
(`candidate-1`, against `baseline-1`) were a byte longer, and so were
|
|
50
|
+
the paths that a plugin's servers put in their environment
|
|
51
|
+
(duel-postgresql's `DATABASE_URL`, `PGHOST` and
|
|
52
|
+
`DUEL_POSTGRESQL_URL`, say). Where a process's stack starts depends on
|
|
53
|
+
how long its arguments and environment are, and at some lengths, a
|
|
54
|
+
tight loop runs the same instructions in more cycles: on macOS,
|
|
55
|
+
server-free benchmarks of pg came out 1.2 to 1.55 times slower on the
|
|
56
|
+
candidate with the plugin, and the same as the baseline without it.
|
|
57
|
+
The two versions' names are now as long as each other.
|
|
58
|
+
|
|
59
|
+
- The workflow that `duel init` writes (and its GitLab CI jobs) no
|
|
60
|
+
longer moves the branch that marks the last commit Duel passed
|
|
61
|
+
backward. When two pushes ran at once and the older one finished
|
|
62
|
+
last, the branch went back to the older push, so the next push was
|
|
63
|
+
compared with it, and a regression already accepted counted again.
|
|
64
|
+
Now the branch only moves forward, or after a force push rewrites
|
|
65
|
+
its history.
|
|
66
|
+
- Where a branch named `duel` (or `duel/passed`) keeps the workflow that
|
|
67
|
+
`duel init` writes from making `duel/passed/main` or `duel/badge`, as
|
|
68
|
+
Git can't have both, the job's error now names that branch and says to
|
|
69
|
+
delete it, instead of GitHub's bare `Reference update failed`.
|
|
70
|
+
- On JRuby, peak memory (`peak_memory: true`) is the most of the JVM's
|
|
71
|
+
heap that a benchmark had in use, not resident memory, which barely
|
|
72
|
+
moves there, as the JVM takes memory for its heap ahead of time: a
|
|
73
|
+
benchmark that needed 24 MB more could pass. JRuby now has peak
|
|
74
|
+
memory on macOS too.
|
|
75
|
+
|
|
10
76
|
## [0.1.2] - 2026-11-02
|
|
11
77
|
|
|
12
78
|
### Added
|
|
@@ -422,7 +488,8 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
422
488
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
423
489
|
release.
|
|
424
490
|
|
|
425
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.
|
|
491
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.3...HEAD
|
|
492
|
+
[0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
|
|
426
493
|
[0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
|
|
427
494
|
[0.1.1]: https://github.com/sferik/duel-ruby/compare/v0.1.0...v0.1.1
|
|
428
495
|
[0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# ⚔️ Duel: Performance regression guards for Ruby
|
|
2
2
|
|
|
3
|
-
[](https://github.com/sferik/duel-ruby/actions/workflows/perf.yml)
|
|
4
4
|
[](https://github.com/sferik/duel-ruby/actions/workflows/test.yml)
|
|
5
5
|
[](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml)
|
|
6
6
|
[](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml)
|
|
@@ -194,6 +194,16 @@ hang the run. If a worker takes more than a minute to load or to answer,
|
|
|
194
194
|
Duel stops the worker and reports the benchmark as an error. To change
|
|
195
195
|
the limit, use `--timeout`.
|
|
196
196
|
|
|
197
|
+
A run measures for 30 minutes at most, not counting the builds. Past
|
|
198
|
+
that, Duel starts no more measurements. A benchmark that it hasn't
|
|
199
|
+
measured yet fails as an error, and a benchmark that looked worse, but
|
|
200
|
+
that Duel hasn't measured again as often as it usually does, fails if it
|
|
201
|
+
was worse in every measurement so far. So a run that's cut short never
|
|
202
|
+
passes a regression. The report says what was left undone. To change
|
|
203
|
+
the limit, use `--time-limit SECONDS`. A run usually takes this long only
|
|
204
|
+
when some benchmark is slow to settle, such as one that starts processes
|
|
205
|
+
of its own, so making it quicker is the better fix.
|
|
206
|
+
|
|
197
207
|
`duel` exits with one of the following statuses:
|
|
198
208
|
|
|
199
209
|
| Status | Meaning |
|
|
@@ -325,14 +335,21 @@ says.
|
|
|
325
335
|
### Show that a project is guarded
|
|
326
336
|
|
|
327
337
|
To show that Duel guards your project, add its badge to your README,
|
|
328
|
-
next to your other badges
|
|
338
|
+
next to your other badges. With the workflow that `duel init` writes,
|
|
339
|
+
the badge turns red when Duel fails a push to your default branch.
|
|
340
|
+
Replace `OWNER/REPO` with your repository's:
|
|
329
341
|
|
|
330
342
|
```markdown
|
|
331
|
-
[](https://github.com/sferik/duel-ruby)
|
|
332
344
|
```
|
|
333
345
|
|
|
334
|
-
|
|
335
|
-
|
|
346
|
+
See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
|
|
347
|
+
for how the workflow keeps it. Without the workflow, such as on GitLab
|
|
348
|
+
CI, use Duel's own badge, which is always green:
|
|
349
|
+
|
|
350
|
+
```markdown
|
|
351
|
+
[](https://github.com/sferik/duel-ruby)
|
|
352
|
+
```
|
|
336
353
|
|
|
337
354
|
## Plugins
|
|
338
355
|
|
data/docs/benchmarks.md
CHANGED
|
@@ -217,7 +217,8 @@ To change that, pass these options to `duel`:
|
|
|
217
217
|
benchmark thread's.
|
|
218
218
|
- `peak_memory: true`: also judges the benchmark's peak memory, which is
|
|
219
219
|
the most resident memory that its measured work reached, above where
|
|
220
|
-
it started
|
|
220
|
+
it started, or on JRuby, the most of the JVM's heap that it had in
|
|
221
|
+
use. Duel takes each version's lowest peak from three fresh
|
|
221
222
|
workers. Like every other measure, peak memory leaves out what
|
|
222
223
|
`once` and `untimed` blocks leave behind.
|
|
223
224
|
|
data/docs/ci.md
CHANGED
|
@@ -167,7 +167,8 @@ jobs:
|
|
|
167
167
|
# the last commit Duel passed, which a branch marks (each branch pushed
|
|
168
168
|
# has its own), so slowdowns too small for any one push to fail add up
|
|
169
169
|
# until they fail one. This job moves the branch to a push that passed
|
|
170
|
-
# (or makes it)
|
|
170
|
+
# (or makes it), but never back: where a newer push's run moved it
|
|
171
|
+
# first, it stays. It checks nothing out and runs none of the code. To
|
|
171
172
|
# accept a regression, move the branch yourself (for main, git push
|
|
172
173
|
# origin main:duel/passed/main).
|
|
173
174
|
passed:
|
|
@@ -183,8 +184,87 @@ jobs:
|
|
|
183
184
|
GH_TOKEN: ${{ github.token }}
|
|
184
185
|
BRANCH: duel/passed/${{ github.ref_name }}
|
|
185
186
|
run: |
|
|
186
|
-
|
|
187
|
-
|
|
187
|
+
# (Git can't make a branch where another's name is a folder of its
|
|
188
|
+
# name: a branch named duel blocks duel/passed/main. If one does,
|
|
189
|
+
# this names it.)
|
|
190
|
+
blocked() {
|
|
191
|
+
prefix=$BRANCH
|
|
192
|
+
while [ "${prefix%/*}" != "$prefix" ]; do
|
|
193
|
+
prefix=${prefix%/*}
|
|
194
|
+
if gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$prefix" > /dev/null 2>&1; then
|
|
195
|
+
echo "::error::The branch $prefix keeps Duel from making $BRANCH, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
|
|
196
|
+
fi
|
|
197
|
+
done
|
|
198
|
+
exit 1
|
|
199
|
+
}
|
|
200
|
+
# (Forward: GitHub moves it only where it's behind the push.)
|
|
201
|
+
gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
|
|
202
|
+
at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
|
|
203
|
+
if [ -z "$at" ]; then
|
|
204
|
+
gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null || blocked
|
|
205
|
+
exit 0
|
|
206
|
+
fi
|
|
207
|
+
compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status)
|
|
208
|
+
case "$compared" in
|
|
209
|
+
# (A newer push's run moved it, or this one's did, run again.)
|
|
210
|
+
ahead | identical) ;;
|
|
211
|
+
# (History rewritten, a force push's, say: moved anyway.)
|
|
212
|
+
diverged | behind) gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" -F force=true > /dev/null ;;
|
|
213
|
+
*) echo "::error::Couldn't compare $BRANCH with this push: $compared" && exit 1 ;;
|
|
214
|
+
esac
|
|
215
|
+
|
|
216
|
+
# Duel's badge, for the README: green (perf guarded) or red (perf
|
|
217
|
+
# regressed), as the last push to the default branch that Duel compared
|
|
218
|
+
# left it. It's red where the Duel job failed or found a regression,
|
|
219
|
+
# even one that passed (with c.fail_on = :pull_requests in .duel.rb). A
|
|
220
|
+
# push with nothing to compare leaves it as it was, and so does a run
|
|
221
|
+
# for an older push than the badge's. This job writes the badge, alone,
|
|
222
|
+
# to a branch of its own, which the README shows:
|
|
223
|
+
#
|
|
224
|
+
# [](https://github.com/sferik/duel-ruby)
|
|
225
|
+
#
|
|
226
|
+
# It checks nothing out and runs none of the code.
|
|
227
|
+
badge:
|
|
228
|
+
name: Update the badge
|
|
229
|
+
needs: duel
|
|
230
|
+
if: ${{ !cancelled() && github.event_name == 'push' && github.ref_name == github.event.repository.default_branch && (needs.duel.result == 'failure' || needs.duel.outputs.status == 'passed' || needs.duel.outputs.status == 'failed') }}
|
|
231
|
+
runs-on: ubuntu-latest
|
|
232
|
+
permissions:
|
|
233
|
+
contents: write # to move the badge's branch
|
|
234
|
+
steps:
|
|
235
|
+
- name: Update the badge
|
|
236
|
+
env:
|
|
237
|
+
GH_TOKEN: ${{ github.token }}
|
|
238
|
+
BRANCH: duel/badge
|
|
239
|
+
RESULT: ${{ needs.duel.result }}
|
|
240
|
+
STATUS: ${{ needs.duel.outputs.status }}
|
|
241
|
+
BADGES: https://raw.githubusercontent.com/sferik/duel-ruby/main/assets
|
|
242
|
+
run: |
|
|
243
|
+
# (Git can't make a branch where another's name is a folder of its
|
|
244
|
+
# name: a branch named duel blocks duel/passed/main. If one does,
|
|
245
|
+
# this names it.)
|
|
246
|
+
blocked() {
|
|
247
|
+
prefix=$BRANCH
|
|
248
|
+
while [ "${prefix%/*}" != "$prefix" ]; do
|
|
249
|
+
prefix=${prefix%/*}
|
|
250
|
+
if gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$prefix" > /dev/null 2>&1; then
|
|
251
|
+
echo "::error::The branch $prefix keeps Duel from making $BRANCH, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
|
|
252
|
+
fi
|
|
253
|
+
done
|
|
254
|
+
exit 1
|
|
255
|
+
}
|
|
256
|
+
# (Where the badge is a newer push's, that push's run has the say.
|
|
257
|
+
# Its commit names the push.)
|
|
258
|
+
last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
|
|
259
|
+
if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
|
|
260
|
+
badge="badge.svg"
|
|
261
|
+
if [ "$RESULT" = failure ] || [ "$STATUS" = failed ]; then badge="badge-regressed.svg"; fi
|
|
262
|
+
curl -fsSL "$BADGES/$badge" -o "$RUNNER_TEMP/badge.svg"
|
|
263
|
+
tree=$(jq -n --rawfile svg "$RUNNER_TEMP/badge.svg" '{tree: [{path: "badge.svg", mode: "100644", type: "blob", content: $svg}]}' |
|
|
264
|
+
gh api --method POST "repos/$GITHUB_REPOSITORY/git/trees" --input - --jq .sha)
|
|
265
|
+
commit=$(gh api --method POST "repos/$GITHUB_REPOSITORY/git/commits" -f message="Duel's badge, for $GITHUB_SHA" -f tree="$tree" --jq .sha)
|
|
266
|
+
gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$commit" -F force=true > /dev/null 2>&1 ||
|
|
267
|
+
gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$commit" > /dev/null || blocked
|
|
188
268
|
```
|
|
189
269
|
|
|
190
270
|
A newer push to a pull request cancels the pull request's run, if it's
|
|
@@ -383,13 +463,23 @@ branch that the workflow guards has its own (see
|
|
|
383
463
|
[Guard other branches](#guard-other-branches)). When a push passes, or has
|
|
384
464
|
nothing to compare, Duel sets the `passed` output to `true`, and the
|
|
385
465
|
Mark the commit passed job moves the branch to the push, or creates the
|
|
386
|
-
branch.
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
466
|
+
branch. It never moves the branch back: when two pushes run at once and
|
|
467
|
+
the older one finishes last, the branch stays at the newer push. After
|
|
468
|
+
a force push rewrites the branch's history, it moves the branch anyway.
|
|
469
|
+
That job checks nothing out either, and it gets the `contents: write`
|
|
470
|
+
permission that moving a branch takes. Until the branch exists, and
|
|
471
|
+
when the push's history doesn't have its commit, Duel compares the push
|
|
472
|
+
with the commit before it. For a project in a subdirectory of its
|
|
473
|
+
repository, the branch names the directory, such as
|
|
391
474
|
`duel/passed-gems-parser/main`.
|
|
392
475
|
|
|
476
|
+
Git can't have a branch named `duel` and a branch named
|
|
477
|
+
`duel/passed/main` at once, because a branch's name is a path, and
|
|
478
|
+
`duel` can't be both a file and a folder. If your repository has a
|
|
479
|
+
branch named `duel`, say the one that added Duel, the job can't make
|
|
480
|
+
its branch, and its error says so. Delete that branch
|
|
481
|
+
(`git push origin --delete duel`), or rename it.
|
|
482
|
+
|
|
393
483
|
After a regression, the branch stays where it is, so every push fails
|
|
394
484
|
(or with [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit),
|
|
395
485
|
is reported) until one is no slower. To accept a regression, move the
|
|
@@ -403,6 +493,32 @@ If you set `c.push_base` back to `:before`, delete the branches too
|
|
|
403
493
|
(`git push origin --delete duel/passed/main`), because the step that
|
|
404
494
|
finds the baseline's Ruby uses one whenever it exists.
|
|
405
495
|
|
|
496
|
+
## Show whether the last push passed
|
|
497
|
+
|
|
498
|
+
The Update the badge job keeps a badge that turns red when Duel fails.
|
|
499
|
+
After each push to your default branch, it writes `badge.svg` to a
|
|
500
|
+
branch of its own, `duel/badge`, so your README can show it:
|
|
501
|
+
|
|
502
|
+
```markdown
|
|
503
|
+
[](https://github.com/sferik/duel-ruby)
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
The badge says "perf guarded", in green, when Duel passed the push. It
|
|
507
|
+
says "perf regressed", in red, when the Duel job failed or found a
|
|
508
|
+
regression, even one that passed, with
|
|
509
|
+
[`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
|
|
510
|
+
label accepted. A push with nothing to compare leaves the badge as it
|
|
511
|
+
was, and so does a run that finishes after a newer push's, so the
|
|
512
|
+
badge is always the newest push's that Duel compared. The branch holds
|
|
513
|
+
one commit, with the badge alone, whose message names that push. The
|
|
514
|
+
job checks nothing out, and it gets the `contents: write` permission
|
|
515
|
+
that moving a branch takes. For a project in a subdirectory of its
|
|
516
|
+
repository, the branch names the directory, such as
|
|
517
|
+
`duel/badge-gems-parser`.
|
|
518
|
+
|
|
519
|
+
Until the first push to your default branch after you add the
|
|
520
|
+
workflow, the branch doesn't exist, and the badge doesn't show.
|
|
521
|
+
|
|
406
522
|
## Guard other branches
|
|
407
523
|
|
|
408
524
|
The workflow compares the pushes to `main`. To guard another branch,
|
|
@@ -595,8 +711,9 @@ duel:commit-comment:
|
|
|
595
711
|
# With c.push_base = :last_passed in .duel.rb, a push is compared with the
|
|
596
712
|
# last commit Duel passed, which a branch marks (duel/passed/main for
|
|
597
713
|
# main), so slowdowns too small for any one push to fail add up until they
|
|
598
|
-
# fail one. This job moves the branch to a push that passed (or makes it)
|
|
599
|
-
#
|
|
714
|
+
# fail one. This job moves the branch to a push that passed (or makes it),
|
|
715
|
+
# but never back: where a newer push's job moved it first, it stays. To
|
|
716
|
+
# accept a regression, move the branch yourself (for main, git push
|
|
600
717
|
# origin main:duel/passed/main).
|
|
601
718
|
duel:passed:
|
|
602
719
|
image: alpine:3
|
|
@@ -607,14 +724,32 @@ duel:passed:
|
|
|
607
724
|
variables:
|
|
608
725
|
GIT_STRATEGY: none # it runs none of the code
|
|
609
726
|
script:
|
|
610
|
-
- apk add --no-cache --quiet curl
|
|
727
|
+
- apk add --no-cache --quiet curl jq
|
|
611
728
|
- |
|
|
612
729
|
[ "$DUEL_PASSED" = true ] || exit 0
|
|
613
730
|
api="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
|
|
614
731
|
token="PRIVATE-TOKEN: $DUEL_TOKEN"
|
|
615
732
|
branch="duel/passed/$CI_COMMIT_BRANCH"
|
|
733
|
+
at=$(curl -sf -H "$token" "$api/repository/branches/$(echo "$branch" | sed 's|/|%2F|g')" | jq -r '.commit.id // empty') || at=
|
|
734
|
+
if [ -n "$at" ]; then
|
|
735
|
+
base=$(curl -sf -H "$token" "$api/repository/merge_base?refs%5B%5D=$CI_COMMIT_SHA&refs%5B%5D=$at" | jq -r '.id // empty') || base=
|
|
736
|
+
[ -n "$base" ] || { echo "Couldn't compare $branch with this push" && exit 1; }
|
|
737
|
+
# (A newer push's job moved it, or this one's did, run again.)
|
|
738
|
+
[ "$base" = "$CI_COMMIT_SHA" ] && exit 0
|
|
739
|
+
fi
|
|
616
740
|
curl -s -o /dev/null -H "$token" -X DELETE "$api/repository/branches/$(echo "$branch" | sed 's|/|%2F|g')"
|
|
617
|
-
curl -sf -o /dev/null -H "$token" --data-urlencode "branch=$branch" --data-urlencode "ref=$CI_COMMIT_SHA" "$api/repository/branches"
|
|
741
|
+
curl -sf -o /dev/null -H "$token" --data-urlencode "branch=$branch" --data-urlencode "ref=$CI_COMMIT_SHA" "$api/repository/branches" && exit 0
|
|
742
|
+
# (Git can't make a branch where another's name is a folder of its
|
|
743
|
+
# name: a branch named duel blocks duel/passed/main. If one does,
|
|
744
|
+
# this names it.)
|
|
745
|
+
prefix=$branch
|
|
746
|
+
while [ "${prefix%/*}" != "$prefix" ]; do
|
|
747
|
+
prefix=${prefix%/*}
|
|
748
|
+
if curl -sf -o /dev/null -H "$token" "$api/repository/branches/$(echo "$prefix" | sed 's|/|%2F|g')"; then
|
|
749
|
+
echo "The branch $prefix keeps Duel from making $branch, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
|
|
750
|
+
fi
|
|
751
|
+
done
|
|
752
|
+
exit 1
|
|
618
753
|
```
|
|
619
754
|
|
|
620
755
|
## Run each version on its own Ruby on CI
|
data/docs/configuration.md
CHANGED
|
@@ -201,7 +201,7 @@ A prepare block can use the following methods:
|
|
|
201
201
|
| `w.failed("message")` | Fails the same way, for any other reason. |
|
|
202
202
|
| `w.at_stop { ... }` | Runs the block when the worker stops. |
|
|
203
203
|
| `w.cached(key) { \|dir\| ... }` | Makes something once, for every worker. See the following example. |
|
|
204
|
-
| `w.name` | The worker's name, for example `"
|
|
204
|
+
| `w.name` | The worker's name, for example `"base-1"`. |
|
|
205
205
|
| `w.dir` | The worker's directory. `w.path("name")` returns the path of a file in it. |
|
|
206
206
|
| `w.root` | The tree of the version that the worker runs. |
|
|
207
207
|
| `w.ruby` | The path of the Ruby that the version runs on. See [Run each version on its own Ruby](#run-each-version-on-its-own-ruby). |
|
|
@@ -223,7 +223,11 @@ Some things take long to make, and are the same for every worker, such
|
|
|
223
223
|
as a database's initial files. `w.cached` makes them once. The first
|
|
224
224
|
worker that asks builds the directory, and the other workers, in this
|
|
225
225
|
run and in later ones, get its path, to copy or to start from. The key
|
|
226
|
-
names what's cached, so put a version in it.
|
|
226
|
+
names what's cached, so put a version in it. If the key ends with a hash
|
|
227
|
+
of what the build depends on, such as `postgres-16.2-3f9c2a1b`, then
|
|
228
|
+
when a run ends, Duel removes the other builds of that name, the part
|
|
229
|
+
before the version and the hash, so an upgrade doesn't leave the old
|
|
230
|
+
build behind. Each build that the run used stays. `duel --clean` removes
|
|
227
231
|
every cached directory.
|
|
228
232
|
|
|
229
233
|
```ruby
|
|
@@ -267,8 +271,11 @@ Duel.configure do |c|
|
|
|
267
271
|
end
|
|
268
272
|
```
|
|
269
273
|
|
|
270
|
-
`v.name` is `"
|
|
271
|
-
|
|
274
|
+
`v.name` is `"base"` (the baseline) or `"head"` (the candidate), and
|
|
275
|
+
`v.root` is the version's directory. The two names are the same length
|
|
276
|
+
on purpose: where a process's stack starts depends on how long its
|
|
277
|
+
arguments and environment are, so a name one byte longer can make every
|
|
278
|
+
worker of one version measurably slower. What the block spawns stops when the run ends. A worker's
|
|
272
279
|
prepare blocks get the version's environment too, as `w.env`. If a
|
|
273
280
|
worker's prepare block sets a variable that the version's environment
|
|
274
281
|
sets to a different value, the run fails.
|
data/docs/how-it-works.md
CHANGED
|
@@ -73,7 +73,7 @@ Duel judges every benchmark on the following metrics:
|
|
|
73
73
|
| `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
|
|
74
74
|
| `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
|
|
75
75
|
| `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
|
|
76
|
-
| `peak_memory` | The most resident memory, above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
|
|
76
|
+
| `peak_memory` | The most resident memory (on JRuby, the JVM's heap in use), above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
|
|
77
77
|
| Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
|
|
78
78
|
|
|
79
79
|
Instruction counts barely vary from run to run, so they catch slowdowns
|
|
@@ -150,7 +150,9 @@ under a hidden name that's as long as your project's directory name, for
|
|
|
150
150
|
example `.3f9c2a1b` next to `my_parser`. Duel keeps it there for the
|
|
151
151
|
next run. If your project has an `ext/` directory, Duel builds the
|
|
152
152
|
baseline with `bundle exec rake compile`. To remove the baseline, git's
|
|
153
|
-
record of it, and the rest of `tmp/duel
|
|
153
|
+
record of it, and the rest of `tmp/duel` (see
|
|
154
|
+
[What Duel keeps in tmp/duel](#what-duel-keeps-in-tmpduel)), run
|
|
155
|
+
`duel --clean`.
|
|
154
156
|
|
|
155
157
|
If your project is in a subdirectory of its repository, as in a
|
|
156
158
|
monorepo, run `duel` in that subdirectory. Duel checks out the whole
|
|
@@ -172,6 +174,27 @@ that small. If Duel can't write next to your project, or something else
|
|
|
172
174
|
already has that name, it uses `tmp/duel/base` instead. Whenever the two
|
|
173
175
|
versions' paths differ in length, Duel says so.
|
|
174
176
|
|
|
177
|
+
### What Duel keeps in tmp/duel
|
|
178
|
+
|
|
179
|
+
Duel keeps what it needs between runs in your project's `tmp/duel`:
|
|
180
|
+
|
|
181
|
+
- `lock`, so that only one run at a time compares the project. A run
|
|
182
|
+
removes it when it ends.
|
|
183
|
+
- `cache/`, what plugins build once and reuse, such as a database's
|
|
184
|
+
initial files. A build is named for what it is, its version, and a
|
|
185
|
+
hash, such as `mysql-0.1.2-3f9c2a1b7e5d4c6a`. When a run ends, Duel
|
|
186
|
+
removes the builds of the same name that the ones it used replace,
|
|
187
|
+
such as the files of a database server that you've since upgraded, so
|
|
188
|
+
the cache keeps one build of each.
|
|
189
|
+
- `base/`, the baseline's checkout, when it can't go next to your
|
|
190
|
+
project.
|
|
191
|
+
|
|
192
|
+
When a run ends, Duel removes `tmp/duel` if nothing is left in it, and
|
|
193
|
+
`tmp` too if Duel made it. A project without plugins that build
|
|
194
|
+
something keeps nothing there. `duel --clean` removes the rest. Each
|
|
195
|
+
run's sockets and server directories are in a temporary directory,
|
|
196
|
+
which the run removes when it ends.
|
|
197
|
+
|
|
175
198
|
If nothing under `lib/` or `ext/`, your benchmarks, or `.duel.rb`
|
|
176
199
|
changed since the baseline, there's nothing to compare, and Duel says
|
|
177
200
|
so. To compare anyway, use `--force`. See
|
|
@@ -207,6 +230,9 @@ Duel runs each benchmark for five seconds before it measures it
|
|
|
207
230
|
(`--warm`). Their timings stay noisier than CRuby's, so Duel often
|
|
208
231
|
leaves small timing regressions on them unsettled.
|
|
209
232
|
|
|
210
|
-
Neither has an allocation count, an instruction count, or profiles
|
|
211
|
-
|
|
233
|
+
Neither has an allocation count, an instruction count, or profiles.
|
|
234
|
+
TruffleRuby has peak memory only on Linux. JRuby measures peak memory in
|
|
235
|
+
the JVM's heap in use, on any operating system, because the JVM takes
|
|
236
|
+
memory for its heap ahead of time, so resident memory barely moves as
|
|
237
|
+
a benchmark's objects fill it. StackProf runs on CRuby only: Duel
|
|
212
238
|
doesn't use it on JRuby or TruffleRuby, even where it's installed.
|
data/docs/json.md
CHANGED
|
@@ -227,6 +227,7 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
|
|
|
227
227
|
| `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
|
|
228
228
|
| `across` | The verdicts across the board, by metric (see below). |
|
|
229
229
|
| `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
|
|
230
|
+
| `cut_short` | What the run left undone when it reached its time limit (`--time-limit`): `{"unmeasured": [names], "unconfirmed": [names], "across": true or false}`. `unmeasured` are the benchmarks that it didn't measure, which are errors. `unconfirmed` are those that it judged without measuring them again as often as it usually does, and `across` is whether it judged across the board without measuring every benchmark again. `null` for a run that finished within its limit. |
|
|
230
231
|
|
|
231
232
|
A run fails when a benchmark regressed, a benchmark failed, or a metric
|
|
232
233
|
was slower across the board and confirmed (`"confirmed": true`).
|
|
@@ -241,6 +242,7 @@ was slower across the board and confirmed (`"confirmed": true`).
|
|
|
241
242
|
| `jobs` | Benchmarks measured at once (`--jobs`). |
|
|
242
243
|
| `processes` | Worker processes for each version (`--processes`). |
|
|
243
244
|
| `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
|
|
245
|
+
| `time_limit` | The seconds that a run measures for, at most (`--time-limit`). |
|
|
244
246
|
| `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default, which depends on the Ruby. |
|
|
245
247
|
| `force` | Whether `--force` was given. |
|
|
246
248
|
|
data/docs/plugins.md
CHANGED
|
@@ -303,10 +303,10 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
|
|
|
303
303
|
`Duel::Testing` has the following methods:
|
|
304
304
|
|
|
305
305
|
- `prepare(config)`: runs the configuration's `c.prepare(:version)`
|
|
306
|
-
blocks, named for the version (`"
|
|
306
|
+
blocks, named for the version (`"head"`), and then its
|
|
307
307
|
`c.prepare(:worker)` blocks, as Duel prepares a worker. It sets the
|
|
308
308
|
environment that they return while its block runs, and stops what
|
|
309
|
-
they started after. `name:` names the worker (`"
|
|
309
|
+
they started after. `name:` names the worker (`"head-1"` by
|
|
310
310
|
default), and `root:` is the version's tree (the configuration's root
|
|
311
311
|
by default). `cache:` is where `w.cached` builds go. By default,
|
|
312
312
|
that's one directory for the whole test process. To look inside it,
|
data/docs/troubleshooting.md
CHANGED
|
@@ -48,6 +48,20 @@ worker and reports the benchmark as an error. To change the limit, use
|
|
|
48
48
|
`--timeout`. If a benchmark's iterations are slow, split it into
|
|
49
49
|
smaller benchmarks.
|
|
50
50
|
|
|
51
|
+
## The run reached its time limit
|
|
52
|
+
|
|
53
|
+
A run measures for 30 minutes at most (`--time-limit`). Past that, Duel
|
|
54
|
+
starts no more measurements: it fails the benchmarks it hasn't measured,
|
|
55
|
+
and judges each benchmark that looked worse on the measurements it has.
|
|
56
|
+
The report starts the list of what it left undone with `TIME LIMIT`.
|
|
57
|
+
|
|
58
|
+
A run usually takes that long because a benchmark is slow to settle. A
|
|
59
|
+
benchmark that starts processes of its own, such as one that forks
|
|
60
|
+
workers, spends most of its time waiting for them, and Duel measures
|
|
61
|
+
each one again when it looks worse. Give it less work per call, or
|
|
62
|
+
fewer calls, so that each sample takes less time. If your benchmarks
|
|
63
|
+
need more time, raise the limit with `--time-limit SECONDS`.
|
|
64
|
+
|
|
51
65
|
## Duel tells you to set c.build
|
|
52
66
|
|
|
53
67
|
Duel checks that each version's workers load only native extensions that
|
data/lib/duel/baseline.rb
CHANGED
|
@@ -124,13 +124,13 @@ module Duel
|
|
|
124
124
|
sibling = File.join(File.dirname(top), ".#{Digest::SHA256.hexdigest(root).ljust(length, "0")[0, length]}")
|
|
125
125
|
return sibling if File.writable?(File.dirname(top)) && (!File.exist?(sibling) || ours?(sibling))
|
|
126
126
|
|
|
127
|
-
File.join(@config.
|
|
127
|
+
File.join(@config.state, "base")
|
|
128
128
|
end
|
|
129
129
|
|
|
130
|
-
# Removes what Duel keeps: tmp/duel, and the baseline's
|
|
131
|
-
# git's record of it as a worktree).
|
|
130
|
+
# Removes what Duel keeps (Config#state: tmp/duel), and the baseline's
|
|
131
|
+
# checkout (and git's record of it as a worktree).
|
|
132
132
|
def clean
|
|
133
|
-
FileUtils.rm_rf(
|
|
133
|
+
FileUtils.rm_rf(@config.state)
|
|
134
134
|
FileUtils.rm_rf(checkout)
|
|
135
135
|
git("worktree", "prune", allow_failure: true) # (not a git repository, say)
|
|
136
136
|
end
|
data/lib/duel/cli.rb
CHANGED
|
@@ -76,7 +76,7 @@ module Duel
|
|
|
76
76
|
|
|
77
77
|
settings = @settings = catch(:version) { parse } or return 0
|
|
78
78
|
config = @config = settings.config_path ? Duel.load_config(settings.config_path) : Config.new
|
|
79
|
-
Lock.hold(
|
|
79
|
+
Lock.hold(config.state) { compare(config, settings) }
|
|
80
80
|
rescue Error => e
|
|
81
81
|
@err.puts "duel: #{e}"
|
|
82
82
|
unfinished(Summary::FAILED_MARK, "couldn't compare", "error", e.to_s, 2)
|
|
@@ -451,6 +451,9 @@ module Duel
|
|
|
451
451
|
o.on("--jobs N", Integer, "Benchmarks measured at once (default: half the CPUs)") { |v| settings.jobs = v }
|
|
452
452
|
o.on("--processes N", Integer, "Worker processes per version (default: 3)") { |v| settings.processes = v }
|
|
453
453
|
o.on("--timeout SECONDS", Float, "Most a worker may take to load or answer (default: 60)") { |v| settings.timeout = v }
|
|
454
|
+
o.on("--time-limit SECONDS", Float, "Most a run measures for, then fails what's unmeasured or looks worse (default: 1800)") do |v|
|
|
455
|
+
settings.time_limit = v
|
|
456
|
+
end
|
|
454
457
|
o.on("--limit RATIO", Float, "Ratio above which a timing is slower (default: 1.02)") { |v| settings.limit = v }
|
|
455
458
|
o.on("--warm SECONDS", Float, "Warm-up per benchmark (default: 5 on JRuby and TruffleRuby, 2 with YJIT on, else 0)") { |v| settings.warm = v }
|
|
456
459
|
o.on("--ruby version-file|PATH", "Each version's Ruby: the one its version file names (.ruby-version, say), or a",
|
data/lib/duel/config.rb
CHANGED
|
@@ -31,6 +31,14 @@ module Duel
|
|
|
31
31
|
# Where the project is (and its config file).
|
|
32
32
|
attr_reader :root
|
|
33
33
|
|
|
34
|
+
# Where Duel keeps what it keeps for the project, tmp/duel: the lock
|
|
35
|
+
# that keeps two runs from sharing it (see Lock), what prepare blocks
|
|
36
|
+
# build once (Resource#cached), in cache/, and the baseline's checkout
|
|
37
|
+
# where it can't go next to the repository (see Baseline#checkout), in
|
|
38
|
+
# base/. A run removes the lock when it ends, and what that leaves
|
|
39
|
+
# empty. `duel --clean` removes the rest.
|
|
40
|
+
def state = File.join(root, "tmp", "duel")
|
|
41
|
+
|
|
34
42
|
# A glob, from the root, of the files defining benchmarks.
|
|
35
43
|
attr_accessor :benchmarks
|
|
36
44
|
|
|
@@ -312,18 +320,17 @@ module Duel
|
|
|
312
320
|
# stopped when it's done with. The block returns the environment
|
|
313
321
|
# variables it gets: a worker's, under its version's.
|
|
314
322
|
#
|
|
315
|
-
# A version's (named "
|
|
316
|
-
#
|
|
317
|
-
#
|
|
318
|
-
#
|
|
319
|
-
# its process benchmarks').
|
|
323
|
+
# A version's (named "base" or "head", its #root the version's tree
|
|
324
|
+
# and its #ruby the Ruby it runs on) is prepared once for a run, in
|
|
325
|
+
# the runner, before any of its workers start: installing its bundle,
|
|
326
|
+
# say. Its environment is its workers' (and its process benchmarks').
|
|
320
327
|
#
|
|
321
328
|
# c.prepare(:version) do |v|
|
|
322
329
|
# v.cached("bundle-#{Digest::SHA256.file(File.join(v.root, "Gemfile.lock"))}") { |_| v.run("bundle", "install", chdir: v.root) }
|
|
323
330
|
# {"BUNDLE_GEMFILE" => File.join(v.root, "Gemfile")}
|
|
324
331
|
# end
|
|
325
332
|
#
|
|
326
|
-
# A worker's (named "
|
|
333
|
+
# A worker's (named "base-1", say) is prepared by the runner as
|
|
327
334
|
# the worker starts: a server of its own, say.
|
|
328
335
|
#
|
|
329
336
|
# c.prepare(:worker) do |w|
|
data/lib/duel/context.rb
CHANGED
|
@@ -498,9 +498,10 @@ module Duel
|
|
|
498
498
|
end
|
|
499
499
|
|
|
500
500
|
# A stretch of a benchmark run for its peak memory: with the garbage
|
|
501
|
-
# collector working as usual, the most
|
|
502
|
-
#
|
|
503
|
-
# every
|
|
501
|
+
# collector working as usual, the most memory the process had in use
|
|
502
|
+
# (Native::Memory.used: resident memory, or on JRuby, the JVM's heap)
|
|
503
|
+
# above where it started (after a full collection), watched every
|
|
504
|
+
# millisecond.
|
|
504
505
|
# @api private
|
|
505
506
|
class MemorySample
|
|
506
507
|
attr_reader :started, :stopped, :peak
|
|
@@ -570,8 +571,8 @@ module Duel
|
|
|
570
571
|
end
|
|
571
572
|
|
|
572
573
|
def resident
|
|
573
|
-
# (
|
|
574
|
-
Native::Memory.
|
|
574
|
+
# (The memory in use is known: see Lane#peak_memory.)
|
|
575
|
+
Native::Memory.used #: Integer
|
|
575
576
|
end
|
|
576
577
|
|
|
577
578
|
def watch
|
data/lib/duel/init.rb
CHANGED
|
@@ -249,5 +249,9 @@ module Duel
|
|
|
249
249
|
# What names the branches that mark the last commit Duel passed in a
|
|
250
250
|
# push, before the branch pushed's name (see Baseline#passed_branch).
|
|
251
251
|
def passed_branch = ["duel/passed", working_directory&.tr("/", "-")].compact.join("-")
|
|
252
|
+
|
|
253
|
+
# The branch that holds the workflow's badge: duel/badge, or for a
|
|
254
|
+
# project in a subdirectory, duel/badge-<its path>.
|
|
255
|
+
def badge_branch = ["duel/badge", working_directory&.tr("/", "-")].compact.join("-")
|
|
252
256
|
end
|
|
253
257
|
end
|