duel 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4cc965d0ce98a08843bc8c1e1c2029045f029a9a4ae3cb05c6fa44c4a1dd7cc4
4
- data.tar.gz: 54a78bbafc884061376d691fe80f6db3e034d837dcf3d2a8448d50522758f669
3
+ metadata.gz: 3827f9485e70bb95c50b32ff9438042bd60cb26885d92f01fe6c1dc6de33ee72
4
+ data.tar.gz: 53e410c89687f6bcdec3c46be4d370409110ac134d90ef74ca8aaaad62394a7e
5
5
  SHA512:
6
- metadata.gz: 28a9a52155e5aa377428a359b2fb0c5c17b6518f9c2af07c473afb66b772c5e46825bb5f712df21590d60d65e35f2446e9412e93013cca2332bcc5bb99f94da0
7
- data.tar.gz: be20c6a67c205344149983ae8e01d49570eb2406d4ff2a4dcf828ea3ab03a260faa7ce782fec119f8257b9138b15bc877b2747f4f7fb1c9c4b3ae2a8c60a4a07
6
+ metadata.gz: 7710b0ab9cc84821d6e66af1d66e1005b98aa0e97a1f047e2a76d6c5d3a83defb413895e11476d067265b97d589dcbc2fb5bdf78550616da74cf92df0da2bd19
7
+ data.tar.gz: 6d9d0df5accaf420f0a4e554f21eb22e79e77f8865986af856759272c5358b6cacd22af27800827d6faf7360daeded8d58481a6ef3c22669279d13a85fe4ff83
data/CHANGELOG.md CHANGED
@@ -7,6 +7,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.3] - 2026-11-04
11
+
12
+ ### Added
13
+
14
+ - The workflow that `duel init` writes keeps a badge for the README
15
+ that turns red when Duel fails a push to the default branch. Its
16
+ Update the badge job writes `badge.svg` to a branch of its own,
17
+ `duel/badge`, green ("perf guarded") when Duel passed the push, and
18
+ red ("perf regressed") when the Duel job failed or found a
19
+ regression, even one that passed.
20
+ - A run measures for 30 minutes at most (`--time-limit SECONDS`), not
21
+ counting the builds. Past that, it starts no more measurements: a
22
+ benchmark it hasn't measured fails as an error, and one that looked
23
+ worse is judged on the measurements it has, so a run cut short never
24
+ passes a regression. The report (and the JSON report's `cut_short`)
25
+ says what it left undone. A benchmark that starts processes of its own
26
+ and sat near its limit could be measured again for hours before.
27
+
28
+ ### Changed
29
+
30
+ - The workers, their directories and sockets, and the resources that
31
+ prepare blocks get are named `base` and `head` (`base-1`, `head-1`),
32
+ not `baseline` and `candidate`: `Resource#name` (`w.name`, `v.name`)
33
+ says so, and `Duel::Testing.prepare` names its worker `head-1` by
34
+ default. The report still says baseline and candidate.
35
+ - A run removes its lock (`tmp/duel/lock`) when it ends, and
36
+ `tmp/duel` (and `tmp`, if Duel made it) when that leaves them empty, so
37
+ a project whose plugins build nothing to keep has nothing left there.
38
+ - When a run ends, the builds that prepare blocks made once
39
+ (`Resource#cached`) and the run used replace the others of their name:
40
+ a key's name is what's before its version and the hash it ends with
41
+ (`mysql`, of `mysql-0.1.2-3f9c2a1b7e5d4c6a`). A database server or a
42
+ plugin upgraded left its old build in `tmp/duel/cache` before, tens
43
+ of megabytes (82 MB for MySQL's) each time.
44
+
45
+ ### Fixed
46
+
47
+ - The candidate's workers could measure slower in every pair, with the
48
+ same instruction counts, on code that hadn't changed. Their names
49
+ (`candidate-1`, against `baseline-1`) were a byte longer, and so were
50
+ the paths that a plugin's servers put in their environment
51
+ (duel-postgresql's `DATABASE_URL`, `PGHOST` and
52
+ `DUEL_POSTGRESQL_URL`, say). Where a process's stack starts depends on
53
+ how long its arguments and environment are, and at some lengths, a
54
+ tight loop runs the same instructions in more cycles: on macOS,
55
+ server-free benchmarks of pg came out 1.2 to 1.55 times slower on the
56
+ candidate with the plugin, and the same as the baseline without it.
57
+ The two versions' names are now as long as each other.
58
+
59
+ - The workflow that `duel init` writes (and its GitLab CI jobs) no
60
+ longer moves the branch that marks the last commit Duel passed
61
+ backward. When two pushes ran at once and the older one finished
62
+ last, the branch went back to the older push, so the next push was
63
+ compared with it, and a regression already accepted counted again.
64
+ Now the branch only moves forward, or after a force push rewrites
65
+ its history.
66
+ - Where a branch named `duel` (or `duel/passed`) keeps the workflow that
67
+ `duel init` writes from making `duel/passed/main` or `duel/badge`, as
68
+ Git can't have both, the job's error now names that branch and says to
69
+ delete it, instead of GitHub's bare `Reference update failed`.
70
+ - On JRuby, peak memory (`peak_memory: true`) is the most of the JVM's
71
+ heap that a benchmark had in use, not resident memory, which barely
72
+ moves there, as the JVM takes memory for its heap ahead of time: a
73
+ benchmark that needed 24 MB more could pass. JRuby now has peak
74
+ memory on macOS too.
75
+
10
76
  ## [0.1.2] - 2026-11-02
11
77
 
12
78
  ### Added
@@ -422,7 +488,8 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
422
488
  Anything marked `@api private`, in lib/ and sig/, may change in any
423
489
  release.
424
490
 
425
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.2...HEAD
491
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.3...HEAD
492
+ [0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
426
493
  [0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
427
494
  [0.1.1]: https://github.com/sferik/duel-ruby/compare/v0.1.0...v0.1.1
428
495
  [0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # ⚔️ Duel: Performance regression guards for Ruby
2
2
 
3
- [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
3
+ [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/perf.yml)
4
4
  [![Test](https://github.com/sferik/duel-ruby/actions/workflows/test.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/test.yml)
5
5
  [![Lint](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml)
6
6
  [![Mutant](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml)
@@ -194,6 +194,16 @@ hang the run. If a worker takes more than a minute to load or to answer,
194
194
  Duel stops the worker and reports the benchmark as an error. To change
195
195
  the limit, use `--timeout`.
196
196
 
197
+ A run measures for 30 minutes at most, not counting the builds. Past
198
+ that, Duel starts no more measurements. A benchmark that it hasn't
199
+ measured yet fails as an error, and a benchmark that looked worse, but
200
+ that Duel hasn't measured again as often as it usually does, fails if it
201
+ was worse in every measurement so far. So a run that's cut short never
202
+ passes a regression. The report says what was left undone. To change
203
+ the limit, use `--time-limit SECONDS`. A run usually takes this long only
204
+ when some benchmark is slow to settle, such as one that starts processes
205
+ of its own, so making it quicker is the better fix.
206
+
197
207
  `duel` exits with one of the following statuses:
198
208
 
199
209
  | Status | Meaning |
@@ -325,14 +335,21 @@ says.
325
335
  ### Show that a project is guarded
326
336
 
327
337
  To show that Duel guards your project, add its badge to your README,
328
- next to your other badges:
338
+ next to your other badges. With the workflow that `duel init` writes,
339
+ the badge turns red when Duel fails a push to your default branch.
340
+ Replace `OWNER/REPO` with your repository's:
329
341
 
330
342
  ```markdown
331
- [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
343
+ [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
332
344
  ```
333
345
 
334
- The badge links to Duel. It doesn't show whether a comparison passed:
335
- your CI's status does that.
346
+ See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
347
+ for how the workflow keeps it. Without the workflow, such as on GitLab
348
+ CI, use Duel's own badge, which is always green:
349
+
350
+ ```markdown
351
+ [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
352
+ ```
336
353
 
337
354
  ## Plugins
338
355
 
data/docs/benchmarks.md CHANGED
@@ -217,7 +217,8 @@ To change that, pass these options to `duel`:
217
217
  benchmark thread's.
218
218
  - `peak_memory: true`: also judges the benchmark's peak memory, which is
219
219
  the most resident memory that its measured work reached, above where
220
- it started. Duel takes each version's lowest peak from three fresh
220
+ it started, or on JRuby, the most of the JVM's heap that it had in
221
+ use. Duel takes each version's lowest peak from three fresh
221
222
  workers. Like every other measure, peak memory leaves out what
222
223
  `once` and `untimed` blocks leave behind.
223
224
 
data/docs/ci.md CHANGED
@@ -167,7 +167,8 @@ jobs:
167
167
  # the last commit Duel passed, which a branch marks (each branch pushed
168
168
  # has its own), so slowdowns too small for any one push to fail add up
169
169
  # until they fail one. This job moves the branch to a push that passed
170
- # (or makes it). It checks nothing out and runs none of the code. To
170
+ # (or makes it), but never back: where a newer push's run moved it
171
+ # first, it stays. It checks nothing out and runs none of the code. To
171
172
  # accept a regression, move the branch yourself (for main, git push
172
173
  # origin main:duel/passed/main).
173
174
  passed:
@@ -183,8 +184,87 @@ jobs:
183
184
  GH_TOKEN: ${{ github.token }}
184
185
  BRANCH: duel/passed/${{ github.ref_name }}
185
186
  run: |
186
- gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" -F force=true > /dev/null 2>&1 ||
187
- gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null
187
+ # (Git can't make a branch where another's name is a folder of its
188
+ # name: a branch named duel blocks duel/passed/main. If one does,
189
+ # this names it.)
190
+ blocked() {
191
+ prefix=$BRANCH
192
+ while [ "${prefix%/*}" != "$prefix" ]; do
193
+ prefix=${prefix%/*}
194
+ if gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$prefix" > /dev/null 2>&1; then
195
+ echo "::error::The branch $prefix keeps Duel from making $BRANCH, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
196
+ fi
197
+ done
198
+ exit 1
199
+ }
200
+ # (Forward: GitHub moves it only where it's behind the push.)
201
+ gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
202
+ at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
203
+ if [ -z "$at" ]; then
204
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null || blocked
205
+ exit 0
206
+ fi
207
+ compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status)
208
+ case "$compared" in
209
+ # (A newer push's run moved it, or this one's did, run again.)
210
+ ahead | identical) ;;
211
+ # (History rewritten, a force push's, say: moved anyway.)
212
+ diverged | behind) gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" -F force=true > /dev/null ;;
213
+ *) echo "::error::Couldn't compare $BRANCH with this push: $compared" && exit 1 ;;
214
+ esac
215
+
216
+ # Duel's badge, for the README: green (perf guarded) or red (perf
217
+ # regressed), as the last push to the default branch that Duel compared
218
+ # left it. It's red where the Duel job failed or found a regression,
219
+ # even one that passed (with c.fail_on = :pull_requests in .duel.rb). A
220
+ # push with nothing to compare leaves it as it was, and so does a run
221
+ # for an older push than the badge's. This job writes the badge, alone,
222
+ # to a branch of its own, which the README shows:
223
+ #
224
+ # [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
225
+ #
226
+ # It checks nothing out and runs none of the code.
227
+ badge:
228
+ name: Update the badge
229
+ needs: duel
230
+ if: ${{ !cancelled() && github.event_name == 'push' && github.ref_name == github.event.repository.default_branch && (needs.duel.result == 'failure' || needs.duel.outputs.status == 'passed' || needs.duel.outputs.status == 'failed') }}
231
+ runs-on: ubuntu-latest
232
+ permissions:
233
+ contents: write # to move the badge's branch
234
+ steps:
235
+ - name: Update the badge
236
+ env:
237
+ GH_TOKEN: ${{ github.token }}
238
+ BRANCH: duel/badge
239
+ RESULT: ${{ needs.duel.result }}
240
+ STATUS: ${{ needs.duel.outputs.status }}
241
+ BADGES: https://raw.githubusercontent.com/sferik/duel-ruby/main/assets
242
+ run: |
243
+ # (Git can't make a branch where another's name is a folder of its
244
+ # name: a branch named duel blocks duel/passed/main. If one does,
245
+ # this names it.)
246
+ blocked() {
247
+ prefix=$BRANCH
248
+ while [ "${prefix%/*}" != "$prefix" ]; do
249
+ prefix=${prefix%/*}
250
+ if gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$prefix" > /dev/null 2>&1; then
251
+ echo "::error::The branch $prefix keeps Duel from making $BRANCH, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
252
+ fi
253
+ done
254
+ exit 1
255
+ }
256
+ # (Where the badge is a newer push's, that push's run has the say.
257
+ # Its commit names the push.)
258
+ last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
259
+ if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
260
+ badge="badge.svg"
261
+ if [ "$RESULT" = failure ] || [ "$STATUS" = failed ]; then badge="badge-regressed.svg"; fi
262
+ curl -fsSL "$BADGES/$badge" -o "$RUNNER_TEMP/badge.svg"
263
+ tree=$(jq -n --rawfile svg "$RUNNER_TEMP/badge.svg" '{tree: [{path: "badge.svg", mode: "100644", type: "blob", content: $svg}]}' |
264
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/trees" --input - --jq .sha)
265
+ commit=$(gh api --method POST "repos/$GITHUB_REPOSITORY/git/commits" -f message="Duel's badge, for $GITHUB_SHA" -f tree="$tree" --jq .sha)
266
+ gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$commit" -F force=true > /dev/null 2>&1 ||
267
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$commit" > /dev/null || blocked
188
268
  ```
189
269
 
190
270
  A newer push to a pull request cancels the pull request's run, if it's
@@ -383,13 +463,23 @@ branch that the workflow guards has its own (see
383
463
  [Guard other branches](#guard-other-branches)). When a push passes, or has
384
464
  nothing to compare, Duel sets the `passed` output to `true`, and the
385
465
  Mark the commit passed job moves the branch to the push, or creates the
386
- branch. That job checks nothing out either, and it gets the
387
- `contents: write` permission that moving a branch takes. Until the
388
- branch exists, and when the push's history doesn't have its commit,
389
- Duel compares the push with the commit before it. For a project in a
390
- subdirectory of its repository, the branch names the directory, such as
466
+ branch. It never moves the branch back: when two pushes run at once and
467
+ the older one finishes last, the branch stays at the newer push. After
468
+ a force push rewrites the branch's history, it moves the branch anyway.
469
+ That job checks nothing out either, and it gets the `contents: write`
470
+ permission that moving a branch takes. Until the branch exists, and
471
+ when the push's history doesn't have its commit, Duel compares the push
472
+ with the commit before it. For a project in a subdirectory of its
473
+ repository, the branch names the directory, such as
391
474
  `duel/passed-gems-parser/main`.
392
475
 
476
+ Git can't have a branch named `duel` and a branch named
477
+ `duel/passed/main` at once, because a branch's name is a path, and
478
+ `duel` can't be both a file and a folder. If your repository has a
479
+ branch named `duel`, say the one that added Duel, the job can't make
480
+ its branch, and its error says so. Delete that branch
481
+ (`git push origin --delete duel`), or rename it.
482
+
393
483
  After a regression, the branch stays where it is, so every push fails
394
484
  (or with [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit),
395
485
  is reported) until one is no slower. To accept a regression, move the
@@ -403,6 +493,32 @@ If you set `c.push_base` back to `:before`, delete the branches too
403
493
  (`git push origin --delete duel/passed/main`), because the step that
404
494
  finds the baseline's Ruby uses one whenever it exists.
405
495
 
496
+ ## Show whether the last push passed
497
+
498
+ The Update the badge job keeps a badge that turns red when Duel fails.
499
+ After each push to your default branch, it writes `badge.svg` to a
500
+ branch of its own, `duel/badge`, so your README can show it:
501
+
502
+ ```markdown
503
+ [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
504
+ ```
505
+
506
+ The badge says "perf guarded", in green, when Duel passed the push. It
507
+ says "perf regressed", in red, when the Duel job failed or found a
508
+ regression, even one that passed, with
509
+ [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
510
+ label accepted. A push with nothing to compare leaves the badge as it
511
+ was, and so does a run that finishes after a newer push's, so the
512
+ badge is always the newest push's that Duel compared. The branch holds
513
+ one commit, with the badge alone, whose message names that push. The
514
+ job checks nothing out, and it gets the `contents: write` permission
515
+ that moving a branch takes. For a project in a subdirectory of its
516
+ repository, the branch names the directory, such as
517
+ `duel/badge-gems-parser`.
518
+
519
+ Until the first push to your default branch after you add the
520
+ workflow, the branch doesn't exist, and the badge doesn't show.
521
+
406
522
  ## Guard other branches
407
523
 
408
524
  The workflow compares the pushes to `main`. To guard another branch,
@@ -595,8 +711,9 @@ duel:commit-comment:
595
711
  # With c.push_base = :last_passed in .duel.rb, a push is compared with the
596
712
  # last commit Duel passed, which a branch marks (duel/passed/main for
597
713
  # main), so slowdowns too small for any one push to fail add up until they
598
- # fail one. This job moves the branch to a push that passed (or makes it).
599
- # To accept a regression, move the branch yourself (for main, git push
714
+ # fail one. This job moves the branch to a push that passed (or makes it),
715
+ # but never back: where a newer push's job moved it first, it stays. To
716
+ # accept a regression, move the branch yourself (for main, git push
600
717
  # origin main:duel/passed/main).
601
718
  duel:passed:
602
719
  image: alpine:3
@@ -607,14 +724,32 @@ duel:passed:
607
724
  variables:
608
725
  GIT_STRATEGY: none # it runs none of the code
609
726
  script:
610
- - apk add --no-cache --quiet curl
727
+ - apk add --no-cache --quiet curl jq
611
728
  - |
612
729
  [ "$DUEL_PASSED" = true ] || exit 0
613
730
  api="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
614
731
  token="PRIVATE-TOKEN: $DUEL_TOKEN"
615
732
  branch="duel/passed/$CI_COMMIT_BRANCH"
733
+ at=$(curl -sf -H "$token" "$api/repository/branches/$(echo "$branch" | sed 's|/|%2F|g')" | jq -r '.commit.id // empty') || at=
734
+ if [ -n "$at" ]; then
735
+ base=$(curl -sf -H "$token" "$api/repository/merge_base?refs%5B%5D=$CI_COMMIT_SHA&refs%5B%5D=$at" | jq -r '.id // empty') || base=
736
+ [ -n "$base" ] || { echo "Couldn't compare $branch with this push" && exit 1; }
737
+ # (A newer push's job moved it, or this one's did, run again.)
738
+ [ "$base" = "$CI_COMMIT_SHA" ] && exit 0
739
+ fi
616
740
  curl -s -o /dev/null -H "$token" -X DELETE "$api/repository/branches/$(echo "$branch" | sed 's|/|%2F|g')"
617
- curl -sf -o /dev/null -H "$token" --data-urlencode "branch=$branch" --data-urlencode "ref=$CI_COMMIT_SHA" "$api/repository/branches"
741
+ curl -sf -o /dev/null -H "$token" --data-urlencode "branch=$branch" --data-urlencode "ref=$CI_COMMIT_SHA" "$api/repository/branches" && exit 0
742
+ # (Git can't make a branch where another's name is a folder of its
743
+ # name: a branch named duel blocks duel/passed/main. If one does,
744
+ # this names it.)
745
+ prefix=$branch
746
+ while [ "${prefix%/*}" != "$prefix" ]; do
747
+ prefix=${prefix%/*}
748
+ if curl -sf -o /dev/null -H "$token" "$api/repository/branches/$(echo "$prefix" | sed 's|/|%2F|g')"; then
749
+ echo "The branch $prefix keeps Duel from making $branch, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
750
+ fi
751
+ done
752
+ exit 1
618
753
  ```
619
754
 
620
755
  ## Run each version on its own Ruby on CI
@@ -201,7 +201,7 @@ A prepare block can use the following methods:
201
201
  | `w.failed("message")` | Fails the same way, for any other reason. |
202
202
  | `w.at_stop { ... }` | Runs the block when the worker stops. |
203
203
  | `w.cached(key) { \|dir\| ... }` | Makes something once, for every worker. See the following example. |
204
- | `w.name` | The worker's name, for example `"baseline-1"`. |
204
+ | `w.name` | The worker's name, for example `"base-1"`. |
205
205
  | `w.dir` | The worker's directory. `w.path("name")` returns the path of a file in it. |
206
206
  | `w.root` | The tree of the version that the worker runs. |
207
207
  | `w.ruby` | The path of the Ruby that the version runs on. See [Run each version on its own Ruby](#run-each-version-on-its-own-ruby). |
@@ -223,7 +223,11 @@ Some things take long to make, and are the same for every worker, such
223
223
  as a database's initial files. `w.cached` makes them once. The first
224
224
  worker that asks builds the directory, and the other workers, in this
225
225
  run and in later ones, get its path, to copy or to start from. The key
226
- names what's cached, so put a version in it. `duel --clean` removes
226
+ names what's cached, so put a version in it. If the key ends with a hash
227
+ of what the build depends on, such as `postgres-16.2-3f9c2a1b`, then
228
+ when a run ends, Duel removes the other builds of that name, the part
229
+ before the version and the hash, so an upgrade doesn't leave the old
230
+ build behind. Each build that the run used stays. `duel --clean` removes
227
231
  every cached directory.
228
232
 
229
233
  ```ruby
@@ -267,8 +271,11 @@ Duel.configure do |c|
267
271
  end
268
272
  ```
269
273
 
270
- `v.name` is `"baseline"` or `"candidate"`, and `v.root` is the version's
271
- directory. What the block spawns stops when the run ends. A worker's
274
+ `v.name` is `"base"` (the baseline) or `"head"` (the candidate), and
275
+ `v.root` is the version's directory. The two names are the same length
276
+ on purpose: where a process's stack starts depends on how long its
277
+ arguments and environment are, so a name one byte longer can make every
278
+ worker of one version measurably slower. What the block spawns stops when the run ends. A worker's
272
279
  prepare blocks get the version's environment too, as `w.env`. If a
273
280
  worker's prepare block sets a variable that the version's environment
274
281
  sets to a different value, the run fails.
data/docs/how-it-works.md CHANGED
@@ -73,7 +73,7 @@ Duel judges every benchmark on the following metrics:
73
73
  | `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
74
74
  | `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
75
75
  | `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
76
- | `peak_memory` | The most resident memory, above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
76
+ | `peak_memory` | The most resident memory (on JRuby, the JVM's heap in use), above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
77
77
  | Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
78
78
 
79
79
  Instruction counts barely vary from run to run, so they catch slowdowns
@@ -150,7 +150,9 @@ under a hidden name that's as long as your project's directory name, for
150
150
  example `.3f9c2a1b` next to `my_parser`. Duel keeps it there for the
151
151
  next run. If your project has an `ext/` directory, Duel builds the
152
152
  baseline with `bundle exec rake compile`. To remove the baseline, git's
153
- record of it, and the rest of `tmp/duel`, run `duel --clean`.
153
+ record of it, and the rest of `tmp/duel` (see
154
+ [What Duel keeps in tmp/duel](#what-duel-keeps-in-tmpduel)), run
155
+ `duel --clean`.
154
156
 
155
157
  If your project is in a subdirectory of its repository, as in a
156
158
  monorepo, run `duel` in that subdirectory. Duel checks out the whole
@@ -172,6 +174,27 @@ that small. If Duel can't write next to your project, or something else
172
174
  already has that name, it uses `tmp/duel/base` instead. Whenever the two
173
175
  versions' paths differ in length, Duel says so.
174
176
 
177
+ ### What Duel keeps in tmp/duel
178
+
179
+ Duel keeps what it needs between runs in your project's `tmp/duel`:
180
+
181
+ - `lock`, so that only one run at a time compares the project. A run
182
+ removes it when it ends.
183
+ - `cache/`, what plugins build once and reuse, such as a database's
184
+ initial files. A build is named for what it is, its version, and a
185
+ hash, such as `mysql-0.1.2-3f9c2a1b7e5d4c6a`. When a run ends, Duel
186
+ removes the builds of the same name that the ones it used replace,
187
+ such as the files of a database server that you've since upgraded, so
188
+ the cache keeps one build of each.
189
+ - `base/`, the baseline's checkout, when it can't go next to your
190
+ project.
191
+
192
+ When a run ends, Duel removes `tmp/duel` if nothing is left in it, and
193
+ `tmp` too if Duel made it. A project without plugins that build
194
+ something keeps nothing there. `duel --clean` removes the rest. Each
195
+ run's sockets and server directories are in a temporary directory,
196
+ which the run removes when it ends.
197
+
175
198
  If nothing under `lib/` or `ext/`, your benchmarks, or `.duel.rb`
176
199
  changed since the baseline, there's nothing to compare, and Duel says
177
200
  so. To compare anyway, use `--force`. See
@@ -207,6 +230,9 @@ Duel runs each benchmark for five seconds before it measures it
207
230
  (`--warm`). Their timings stay noisier than CRuby's, so Duel often
208
231
  leaves small timing regressions on them unsettled.
209
232
 
210
- Neither has an allocation count, an instruction count, or profiles, and
211
- each has peak memory only on Linux. StackProf runs on CRuby only: Duel
233
+ Neither has an allocation count, an instruction count, or profiles.
234
+ TruffleRuby has peak memory only on Linux. JRuby measures peak memory in
235
+ the JVM's heap in use, on any operating system, because the JVM takes
236
+ memory for its heap ahead of time, so resident memory barely moves as
237
+ a benchmark's objects fill it. StackProf runs on CRuby only: Duel
212
238
  doesn't use it on JRuby or TruffleRuby, even where it's installed.
data/docs/json.md CHANGED
@@ -227,6 +227,7 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
227
227
  | `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
228
228
  | `across` | The verdicts across the board, by metric (see below). |
229
229
  | `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
230
+ | `cut_short` | What the run left undone when it reached its time limit (`--time-limit`): `{"unmeasured": [names], "unconfirmed": [names], "across": true or false}`. `unmeasured` are the benchmarks that it didn't measure, which are errors. `unconfirmed` are those that it judged without measuring them again as often as it usually does, and `across` is whether it judged across the board without measuring every benchmark again. `null` for a run that finished within its limit. |
230
231
 
231
232
  A run fails when a benchmark regressed, a benchmark failed, or a metric
232
233
  was slower across the board and confirmed (`"confirmed": true`).
@@ -241,6 +242,7 @@ was slower across the board and confirmed (`"confirmed": true`).
241
242
  | `jobs` | Benchmarks measured at once (`--jobs`). |
242
243
  | `processes` | Worker processes for each version (`--processes`). |
243
244
  | `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
245
+ | `time_limit` | The seconds that a run measures for, at most (`--time-limit`). |
244
246
  | `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default, which depends on the Ruby. |
245
247
  | `force` | Whether `--force` was given. |
246
248
 
data/docs/plugins.md CHANGED
@@ -303,10 +303,10 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
303
303
  `Duel::Testing` has the following methods:
304
304
 
305
305
  - `prepare(config)`: runs the configuration's `c.prepare(:version)`
306
- blocks, named for the version (`"candidate"`), and then its
306
+ blocks, named for the version (`"head"`), and then its
307
307
  `c.prepare(:worker)` blocks, as Duel prepares a worker. It sets the
308
308
  environment that they return while its block runs, and stops what
309
- they started after. `name:` names the worker (`"candidate-1"` by
309
+ they started after. `name:` names the worker (`"head-1"` by
310
310
  default), and `root:` is the version's tree (the configuration's root
311
311
  by default). `cache:` is where `w.cached` builds go. By default,
312
312
  that's one directory for the whole test process. To look inside it,
@@ -48,6 +48,20 @@ worker and reports the benchmark as an error. To change the limit, use
48
48
  `--timeout`. If a benchmark's iterations are slow, split it into
49
49
  smaller benchmarks.
50
50
 
51
+ ## The run reached its time limit
52
+
53
+ A run measures for 30 minutes at most (`--time-limit`). Past that, Duel
54
+ starts no more measurements: it fails the benchmarks it hasn't measured,
55
+ and judges each benchmark that looked worse on the measurements it has.
56
+ The report starts the list of what it left undone with `TIME LIMIT`.
57
+
58
+ A run usually takes that long because a benchmark is slow to settle. A
59
+ benchmark that starts processes of its own, such as one that forks
60
+ workers, spends most of its time waiting for them, and Duel measures
61
+ each one again when it looks worse. Give it less work per call, or
62
+ fewer calls, so that each sample takes less time. If your benchmarks
63
+ need more time, raise the limit with `--time-limit SECONDS`.
64
+
51
65
  ## Duel tells you to set c.build
52
66
 
53
67
  Duel checks that each version's workers load only native extensions that
data/lib/duel/baseline.rb CHANGED
@@ -124,13 +124,13 @@ module Duel
124
124
  sibling = File.join(File.dirname(top), ".#{Digest::SHA256.hexdigest(root).ljust(length, "0")[0, length]}")
125
125
  return sibling if File.writable?(File.dirname(top)) && (!File.exist?(sibling) || ours?(sibling))
126
126
 
127
- File.join(@config.root, "tmp", "duel", "base")
127
+ File.join(@config.state, "base")
128
128
  end
129
129
 
130
- # Removes what Duel keeps: tmp/duel, and the baseline's checkout (and
131
- # git's record of it as a worktree).
130
+ # Removes what Duel keeps (Config#state: tmp/duel), and the baseline's
131
+ # checkout (and git's record of it as a worktree).
132
132
  def clean
133
- FileUtils.rm_rf(File.join(@config.root, "tmp", "duel"))
133
+ FileUtils.rm_rf(@config.state)
134
134
  FileUtils.rm_rf(checkout)
135
135
  git("worktree", "prune", allow_failure: true) # (not a git repository, say)
136
136
  end
data/lib/duel/cli.rb CHANGED
@@ -76,7 +76,7 @@ module Duel
76
76
 
77
77
  settings = @settings = catch(:version) { parse } or return 0
78
78
  config = @config = settings.config_path ? Duel.load_config(settings.config_path) : Config.new
79
- Lock.hold(File.join(config.root, "tmp", "duel")) { compare(config, settings) }
79
+ Lock.hold(config.state) { compare(config, settings) }
80
80
  rescue Error => e
81
81
  @err.puts "duel: #{e}"
82
82
  unfinished(Summary::FAILED_MARK, "couldn't compare", "error", e.to_s, 2)
@@ -451,6 +451,9 @@ module Duel
451
451
  o.on("--jobs N", Integer, "Benchmarks measured at once (default: half the CPUs)") { |v| settings.jobs = v }
452
452
  o.on("--processes N", Integer, "Worker processes per version (default: 3)") { |v| settings.processes = v }
453
453
  o.on("--timeout SECONDS", Float, "Most a worker may take to load or answer (default: 60)") { |v| settings.timeout = v }
454
+ o.on("--time-limit SECONDS", Float, "Most a run measures for, then fails what's unmeasured or looks worse (default: 1800)") do |v|
455
+ settings.time_limit = v
456
+ end
454
457
  o.on("--limit RATIO", Float, "Ratio above which a timing is slower (default: 1.02)") { |v| settings.limit = v }
455
458
  o.on("--warm SECONDS", Float, "Warm-up per benchmark (default: 5 on JRuby and TruffleRuby, 2 with YJIT on, else 0)") { |v| settings.warm = v }
456
459
  o.on("--ruby version-file|PATH", "Each version's Ruby: the one its version file names (.ruby-version, say), or a",
data/lib/duel/config.rb CHANGED
@@ -31,6 +31,14 @@ module Duel
31
31
  # Where the project is (and its config file).
32
32
  attr_reader :root
33
33
 
34
+ # Where Duel keeps what it keeps for the project, tmp/duel: the lock
35
+ # that keeps two runs from sharing it (see Lock), what prepare blocks
36
+ # build once (Resource#cached), in cache/, and the baseline's checkout
37
+ # where it can't go next to the repository (see Baseline#checkout), in
38
+ # base/. A run removes the lock when it ends, and what that leaves
39
+ # empty. `duel --clean` removes the rest.
40
+ def state = File.join(root, "tmp", "duel")
41
+
34
42
  # A glob, from the root, of the files defining benchmarks.
35
43
  attr_accessor :benchmarks
36
44
 
@@ -312,18 +320,17 @@ module Duel
312
320
  # stopped when it's done with. The block returns the environment
313
321
  # variables it gets: a worker's, under its version's.
314
322
  #
315
- # A version's (named "baseline" or "candidate", its #root the
316
- # version's tree and its #ruby the Ruby it runs on) is prepared once
317
- # for a run, in the runner, before any of its workers start:
318
- # installing its bundle, say. Its environment is its workers' (and
319
- # its process benchmarks').
323
+ # A version's (named "base" or "head", its #root the version's tree
324
+ # and its #ruby the Ruby it runs on) is prepared once for a run, in
325
+ # the runner, before any of its workers start: installing its bundle,
326
+ # say. Its environment is its workers' (and its process benchmarks').
320
327
  #
321
328
  # c.prepare(:version) do |v|
322
329
  # v.cached("bundle-#{Digest::SHA256.file(File.join(v.root, "Gemfile.lock"))}") { |_| v.run("bundle", "install", chdir: v.root) }
323
330
  # {"BUNDLE_GEMFILE" => File.join(v.root, "Gemfile")}
324
331
  # end
325
332
  #
326
- # A worker's (named "baseline-1", say) is prepared by the runner as
333
+ # A worker's (named "base-1", say) is prepared by the runner as
327
334
  # the worker starts: a server of its own, say.
328
335
  #
329
336
  # c.prepare(:worker) do |w|
data/lib/duel/context.rb CHANGED
@@ -498,9 +498,10 @@ module Duel
498
498
  end
499
499
 
500
500
  # A stretch of a benchmark run for its peak memory: with the garbage
501
- # collector working as usual, the most resident memory the process
502
- # reached above where it started (after a full collection), watched
503
- # every millisecond.
501
+ # collector working as usual, the most memory the process had in use
502
+ # (Native::Memory.used: resident memory, or on JRuby, the JVM's heap)
503
+ # above where it started (after a full collection), watched every
504
+ # millisecond.
504
505
  # @api private
505
506
  class MemorySample
506
507
  attr_reader :started, :stopped, :peak
@@ -570,8 +571,8 @@ module Duel
570
571
  end
571
572
 
572
573
  def resident
573
- # (Resident memory is known: see Lane#peak_memory.)
574
- Native::Memory.resident #: Integer
574
+ # (The memory in use is known: see Lane#peak_memory.)
575
+ Native::Memory.used #: Integer
575
576
  end
576
577
 
577
578
  def watch
data/lib/duel/init.rb CHANGED
@@ -249,5 +249,9 @@ module Duel
249
249
  # What names the branches that mark the last commit Duel passed in a
250
250
  # push, before the branch pushed's name (see Baseline#passed_branch).
251
251
  def passed_branch = ["duel/passed", working_directory&.tr("/", "-")].compact.join("-")
252
+
253
+ # The branch that holds the workflow's badge: duel/badge, or for a
254
+ # project in a subdirectory, duel/badge-<its path>.
255
+ def badge_branch = ["duel/badge", working_directory&.tr("/", "-")].compact.join("-")
252
256
  end
253
257
  end