duel 0.1.2 → 0.1.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4cc965d0ce98a08843bc8c1e1c2029045f029a9a4ae3cb05c6fa44c4a1dd7cc4
4
- data.tar.gz: 54a78bbafc884061376d691fe80f6db3e034d837dcf3d2a8448d50522758f669
3
+ metadata.gz: 15e688ef3d4a4d3c89f10bfef8621961c6025933456c845c3cdca1d048b3d2dd
4
+ data.tar.gz: 71d90787a6a0993a5161acd2f7e9594498bc2e6e297d07d3970acd86c07471c5
5
5
  SHA512:
6
- metadata.gz: 28a9a52155e5aa377428a359b2fb0c5c17b6518f9c2af07c473afb66b772c5e46825bb5f712df21590d60d65e35f2446e9412e93013cca2332bcc5bb99f94da0
7
- data.tar.gz: be20c6a67c205344149983ae8e01d49570eb2406d4ff2a4dcf828ea3ab03a260faa7ce782fec119f8257b9138b15bc877b2747f4f7fb1c9c4b3ae2a8c60a4a07
6
+ metadata.gz: 3275eb44aeec1133c699604b391c716a989242e657acd51fb48fb2ebb8aabad55f658b09151e5c07250a07f4d9d1e5e7b201bbd47f6e760b4b30a9516a6681f7
7
+ data.tar.gz: 795438481087f53a0cb8faee6a61155d787ef5d808c079f2a222e1f0b2a12d957526080103d7d144990f096f1482dcd60bc27c9321f252f619a005a0eb4d2a3a
data/CHANGELOG.md CHANGED
@@ -7,6 +7,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.4] - 2026-11-06
11
+
12
+ ### Fixed
13
+
14
+ - In json's own repository, every baseline worker loaded the
15
+ candidate's json, extensions and all, before the baseline's, as the
16
+ worker loaded json for its own messages to the runner before it
17
+ loaded the version. A worker and the runner now send each other
18
+ their messages marshaled, so a worker loads nothing a version could
19
+ be before it loads the version.
20
+ - A worker loaded etc, a gem, before it loaded the version it
21
+ measures, for the page size that `/proc/self/statm` counts resident
22
+ memory in. In etc's own repository, a worker would load the
23
+ bundle's etc before the version's own. A worker now reads resident
24
+ memory, in kB, from `/proc/self/status` (VmRSS), and loads no etc.
25
+ - A worker loaded fiddle, a gem, as Duel loaded, before it loaded the
26
+ version it measures, so in fiddle's own repository, a worker would
27
+ load the bundle's fiddle before the version's own. Duel now loads
28
+ fiddle as it first calls into the operating system, once the
29
+ version has loaded (and on JRuby, which Duel makes no such calls
30
+ on, never).
31
+ - A worker that loaded a native extension twice, the candidate's that
32
+ something required before the worker loaded the version and then the
33
+ version's own over it, went unnoticed, as loading the version took
34
+ the candidate's off `$LOADED_FEATURES`. Duel now fails, and names the
35
+ extension and both of its files.
36
+ - On JRuby, peak memory came out short, by tens of megabytes at times,
37
+ and a benchmark that needed more memory could pass. A peak is
38
+ measured above what's in use after the garbage is collected, but
39
+ JRuby's `GC.start` collects nothing, so the garbage left from before
40
+ stayed in use until the benchmark's own objects made room. Duel now
41
+ collects it with the JVM's own call (`java.lang.System.gc`).
42
+ - Where a repository's history was replaced, so the commit that
43
+ `duel/passed/BRANCH` marks has no history in common with a push, the
44
+ Mark the commit passed job failed, as GitHub can't compare the two.
45
+ It now moves the branch to the push, as it does where history was
46
+ rewritten. Update the job in `.github/workflows/perf.yml` to pick
47
+ this up.
48
+
49
+ ## [0.1.3] - 2026-11-04
50
+
51
+ ### Added
52
+
53
+ - The workflow that `duel init` writes keeps a badge for the README
54
+ that turns red when Duel fails a push to the default branch. Its
55
+ Update the badge job writes `badge.svg` to a branch of its own,
56
+ `duel/badge`, green ("perf guarded") when Duel passed the push, and
57
+ red ("perf regressed") when the Duel job failed or found a
58
+ regression, even one that passed.
59
+ - A run measures for 30 minutes at most (`--time-limit SECONDS`), not
60
+ counting the builds. Past that, it starts no more measurements: a
61
+ benchmark it hasn't measured fails as an error, and one that looked
62
+ worse is judged on the measurements it has, so a run cut short never
63
+ passes a regression. The report (and the JSON report's `cut_short`)
64
+ says what it left undone. A benchmark that starts processes of its own
65
+ and sat near its limit could be measured again for hours before.
66
+
67
+ ### Changed
68
+
69
+ - The workers, their directories and sockets, and the resources that
70
+ prepare blocks get are named `base` and `head` (`base-1`, `head-1`),
71
+ not `baseline` and `candidate`: `Resource#name` (`w.name`, `v.name`)
72
+ says so, and `Duel::Testing.prepare` names its worker `head-1` by
73
+ default. The report still says baseline and candidate.
74
+ - A run removes its lock (`tmp/duel/lock`) when it ends, and
75
+ `tmp/duel` (and `tmp`, if Duel made it) when that leaves them empty, so
76
+ a project whose plugins build nothing to keep has nothing left there.
77
+ - When a run ends, the builds that prepare blocks made once
78
+ (`Resource#cached`) and the run used replace the others of their name:
79
+ a key's name is what's before its version and the hash it ends with
80
+ (`mysql`, of `mysql-0.1.2-3f9c2a1b7e5d4c6a`). A database server or a
81
+ plugin upgraded left its old build in `tmp/duel/cache` before, tens
82
+ of megabytes (82 MB for MySQL's) each time.
83
+
84
+ ### Fixed
85
+
86
+ - The candidate's workers could measure slower in every pair, with the
87
+ same instruction counts, on code that hadn't changed. Their names
88
+ (`candidate-1`, against `baseline-1`) were a byte longer, and so were
89
+ the paths that a plugin's servers put in their environment
90
+ (duel-postgresql's `DATABASE_URL`, `PGHOST` and
91
+ `DUEL_POSTGRESQL_URL`, say). Where a process's stack starts depends on
92
+ how long its arguments and environment are, and at some lengths, a
93
+ tight loop runs the same instructions in more cycles: on macOS,
94
+ server-free benchmarks of pg came out 1.2 to 1.55 times slower on the
95
+ candidate with the plugin, and the same as the baseline without it.
96
+ The two versions' names are now as long as each other.
97
+
98
+ - The workflow that `duel init` writes (and its GitLab CI jobs) no
99
+ longer moves the branch that marks the last commit Duel passed
100
+ backward. When two pushes ran at once and the older one finished
101
+ last, the branch went back to the older push, so the next push was
102
+ compared with it, and a regression already accepted counted again.
103
+ Now the branch only moves forward, or after a force push rewrites
104
+ its history.
105
+ - Where a branch named `duel` (or `duel/passed`) keeps the workflow that
106
+ `duel init` writes from making `duel/passed/main` or `duel/badge`, as
107
+ Git can't have both, the job's error now names that branch and says to
108
+ delete it, instead of GitHub's bare `Reference update failed`.
109
+ - On JRuby, peak memory (`peak_memory: true`) is the most of the JVM's
110
+ heap that a benchmark had in use, not resident memory, which barely
111
+ moves there, as the JVM takes memory for its heap ahead of time: a
112
+ benchmark that needed 24 MB more could pass. JRuby now has peak
113
+ memory on macOS too.
114
+
10
115
  ## [0.1.2] - 2026-11-02
11
116
 
12
117
  ### Added
@@ -422,7 +527,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
422
527
  Anything marked `@api private`, in lib/ and sig/, may change in any
423
528
  release.
424
529
 
425
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.2...HEAD
530
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.4...HEAD
531
+ [0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
532
+ [0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
426
533
  [0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
427
534
  [0.1.1]: https://github.com/sferik/duel-ruby/compare/v0.1.0...v0.1.1
428
535
  [0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # ⚔️ Duel: Performance regression guards for Ruby
2
2
 
3
- [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
3
+ [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/perf.yml)
4
4
  [![Test](https://github.com/sferik/duel-ruby/actions/workflows/test.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/test.yml)
5
5
  [![Lint](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml)
6
6
  [![Mutant](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml/badge.svg)](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml)
@@ -194,6 +194,16 @@ hang the run. If a worker takes more than a minute to load or to answer,
194
194
  Duel stops the worker and reports the benchmark as an error. To change
195
195
  the limit, use `--timeout`.
196
196
 
197
+ A run measures for 30 minutes at most, not counting the builds. Past
198
+ that, Duel starts no more measurements. A benchmark that it hasn't
199
+ measured yet fails as an error, and a benchmark that looked worse, but
200
+ that Duel hasn't measured again as often as it usually does, fails if it
201
+ was worse in every measurement so far. So a run that's cut short never
202
+ passes a regression. The report says what was left undone. To change
203
+ the limit, use `--time-limit SECONDS`. A run usually takes this long only
204
+ when some benchmark is slow to settle, such as one that starts processes
205
+ of its own, so making it quicker is the better fix.
206
+
197
207
  `duel` exits with one of the following statuses:
198
208
 
199
209
  | Status | Meaning |
@@ -325,14 +335,21 @@ says.
325
335
  ### Show that a project is guarded
326
336
 
327
337
  To show that Duel guards your project, add its badge to your README,
328
- next to your other badges:
338
+ next to your other badges. With the workflow that `duel init` writes,
339
+ the badge turns red when Duel fails a push to your default branch.
340
+ Replace `OWNER/REPO` with your repository's:
329
341
 
330
342
  ```markdown
331
- [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
343
+ [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
332
344
  ```
333
345
 
334
- The badge links to Duel. It doesn't show whether a comparison passed:
335
- your CI's status does that.
346
+ See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
347
+ for how the workflow keeps it. Without the workflow, such as on GitLab
348
+ CI, use Duel's own badge, which is always green:
349
+
350
+ ```markdown
351
+ [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
352
+ ```
336
353
 
337
354
  ## Plugins
338
355
 
data/docs/benchmarks.md CHANGED
@@ -217,7 +217,8 @@ To change that, pass these options to `duel`:
217
217
  benchmark thread's.
218
218
  - `peak_memory: true`: also judges the benchmark's peak memory, which is
219
219
  the most resident memory that its measured work reached, above where
220
- it started. Duel takes each version's lowest peak from three fresh
220
+ it started, or on JRuby, the most of the JVM's heap that it had in
221
+ use. Duel takes each version's lowest peak from three fresh
221
222
  workers. Like every other measure, peak memory leaves out what
222
223
  `once` and `untimed` blocks leave behind.
223
224
 
data/docs/ci.md CHANGED
@@ -167,7 +167,8 @@ jobs:
167
167
  # the last commit Duel passed, which a branch marks (each branch pushed
168
168
  # has its own), so slowdowns too small for any one push to fail add up
169
169
  # until they fail one. This job moves the branch to a push that passed
170
- # (or makes it). It checks nothing out and runs none of the code. To
170
+ # (or makes it), but never back: where a newer push's run moved it
171
+ # first, it stays. It checks nothing out and runs none of the code. To
171
172
  # accept a regression, move the branch yourself (for main, git push
172
173
  # origin main:duel/passed/main).
173
174
  passed:
@@ -183,8 +184,90 @@ jobs:
183
184
  GH_TOKEN: ${{ github.token }}
184
185
  BRANCH: duel/passed/${{ github.ref_name }}
185
186
  run: |
186
- gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" -F force=true > /dev/null 2>&1 ||
187
- gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null
187
+ # (Git can't make a branch where another's name is a folder of its
188
+ # name: a branch named duel blocks duel/passed/main. If one does,
189
+ # this names it.)
190
+ blocked() {
191
+ prefix=$BRANCH
192
+ while [ "${prefix%/*}" != "$prefix" ]; do
193
+ prefix=${prefix%/*}
194
+ if gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$prefix" > /dev/null 2>&1; then
195
+ echo "::error::The branch $prefix keeps Duel from making $BRANCH, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
196
+ fi
197
+ done
198
+ exit 1
199
+ }
200
+ # (Forward: GitHub moves it only where it's behind the push.)
201
+ gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
202
+ at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
203
+ if [ -z "$at" ]; then
204
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null || blocked
205
+ exit 0
206
+ fi
207
+ compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status 2>&1) || case "$compared" in
208
+ # (GitHub can't compare commits with no history in common.)
209
+ *"No common ancestor"*) compared=unrelated ;;
210
+ esac
211
+ case "$compared" in
212
+ # (A newer push's run moved it, or this one's did, run again.)
213
+ ahead | identical) ;;
214
+ # (History rewritten, a force push's, say, or replaced: moved anyway.)
215
+ diverged | behind | unrelated) gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" -F force=true > /dev/null ;;
216
+ *) echo "::error::Couldn't compare $BRANCH with this push: $compared" && exit 1 ;;
217
+ esac
218
+
219
+ # Duel's badge, for the README: green (perf guarded) or red (perf
220
+ # regressed), as the last push to the default branch that Duel compared
221
+ # left it. It's red where the Duel job failed or found a regression,
222
+ # even one that passed (with c.fail_on = :pull_requests in .duel.rb). A
223
+ # push with nothing to compare leaves it as it was, and so does a run
224
+ # for an older push than the badge's. This job writes the badge, alone,
225
+ # to a branch of its own, which the README shows:
226
+ #
227
+ # [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
228
+ #
229
+ # It checks nothing out and runs none of the code.
230
+ badge:
231
+ name: Update the badge
232
+ needs: duel
233
+ if: ${{ !cancelled() && github.event_name == 'push' && github.ref_name == github.event.repository.default_branch && (needs.duel.result == 'failure' || needs.duel.outputs.status == 'passed' || needs.duel.outputs.status == 'failed') }}
234
+ runs-on: ubuntu-latest
235
+ permissions:
236
+ contents: write # to move the badge's branch
237
+ steps:
238
+ - name: Update the badge
239
+ env:
240
+ GH_TOKEN: ${{ github.token }}
241
+ BRANCH: duel/badge
242
+ RESULT: ${{ needs.duel.result }}
243
+ STATUS: ${{ needs.duel.outputs.status }}
244
+ BADGES: https://raw.githubusercontent.com/sferik/duel-ruby/main/assets
245
+ run: |
246
+ # (Git can't make a branch where another's name is a folder of its
247
+ # name: a branch named duel blocks duel/passed/main. If one does,
248
+ # this names it.)
249
+ blocked() {
250
+ prefix=$BRANCH
251
+ while [ "${prefix%/*}" != "$prefix" ]; do
252
+ prefix=${prefix%/*}
253
+ if gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$prefix" > /dev/null 2>&1; then
254
+ echo "::error::The branch $prefix keeps Duel from making $BRANCH, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
255
+ fi
256
+ done
257
+ exit 1
258
+ }
259
+ # (Where the badge is a newer push's, that push's run has the say.
260
+ # Its commit names the push.)
261
+ last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
262
+ if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
263
+ badge="badge.svg"
264
+ if [ "$RESULT" = failure ] || [ "$STATUS" = failed ]; then badge="badge-regressed.svg"; fi
265
+ curl -fsSL "$BADGES/$badge" -o "$RUNNER_TEMP/badge.svg"
266
+ tree=$(jq -n --rawfile svg "$RUNNER_TEMP/badge.svg" '{tree: [{path: "badge.svg", mode: "100644", type: "blob", content: $svg}]}' |
267
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/trees" --input - --jq .sha)
268
+ commit=$(gh api --method POST "repos/$GITHUB_REPOSITORY/git/commits" -f message="Duel's badge, for $GITHUB_SHA" -f tree="$tree" --jq .sha)
269
+ gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$commit" -F force=true > /dev/null 2>&1 ||
270
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$commit" > /dev/null || blocked
188
271
  ```
189
272
 
190
273
  A newer push to a pull request cancels the pull request's run, if it's
@@ -383,13 +466,23 @@ branch that the workflow guards has its own (see
383
466
  [Guard other branches](#guard-other-branches)). When a push passes, or has
384
467
  nothing to compare, Duel sets the `passed` output to `true`, and the
385
468
  Mark the commit passed job moves the branch to the push, or creates the
386
- branch. That job checks nothing out either, and it gets the
387
- `contents: write` permission that moving a branch takes. Until the
388
- branch exists, and when the push's history doesn't have its commit,
389
- Duel compares the push with the commit before it. For a project in a
390
- subdirectory of its repository, the branch names the directory, such as
469
+ branch. It never moves the branch back: when two pushes run at once and
470
+ the older one finishes last, the branch stays at the newer push. After
471
+ a force push rewrites the branch's history, it moves the branch anyway.
472
+ That job checks nothing out either, and it gets the `contents: write`
473
+ permission that moving a branch takes. Until the branch exists, and
474
+ when the push's history doesn't have its commit, Duel compares the push
475
+ with the commit before it. For a project in a subdirectory of its
476
+ repository, the branch names the directory, such as
391
477
  `duel/passed-gems-parser/main`.
392
478
 
479
+ Git can't have a branch named `duel` and a branch named
480
+ `duel/passed/main` at once, because a branch's name is a path, and
481
+ `duel` can't be both a file and a folder. If your repository has a
482
+ branch named `duel`, say the one that added Duel, the job can't make
483
+ its branch, and its error says so. Delete that branch
484
+ (`git push origin --delete duel`), or rename it.
485
+
393
486
  After a regression, the branch stays where it is, so every push fails
394
487
  (or with [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit),
395
488
  is reported) until one is no slower. To accept a regression, move the
@@ -403,6 +496,32 @@ If you set `c.push_base` back to `:before`, delete the branches too
403
496
  (`git push origin --delete duel/passed/main`), because the step that
404
497
  finds the baseline's Ruby uses one whenever it exists.
405
498
 
499
+ ## Show whether the last push passed
500
+
501
+ The Update the badge job keeps a badge that turns red when Duel fails.
502
+ After each push to your default branch, it writes `badge.svg` to a
503
+ branch of its own, `duel/badge`, so your README can show it:
504
+
505
+ ```markdown
506
+ [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
507
+ ```
508
+
509
+ The badge says "perf guarded", in green, when Duel passed the push. It
510
+ says "perf regressed", in red, when the Duel job failed or found a
511
+ regression, even one that passed, with
512
+ [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
513
+ label accepted. A push with nothing to compare leaves the badge as it
514
+ was, and so does a run that finishes after a newer push's, so the
515
+ badge is always the newest push's that Duel compared. The branch holds
516
+ one commit, with the badge alone, whose message names that push. The
517
+ job checks nothing out, and it gets the `contents: write` permission
518
+ that moving a branch takes. For a project in a subdirectory of its
519
+ repository, the branch names the directory, such as
520
+ `duel/badge-gems-parser`.
521
+
522
+ Until the first push to your default branch after you add the
523
+ workflow, the branch doesn't exist, and the badge doesn't show.
524
+
406
525
  ## Guard other branches
407
526
 
408
527
  The workflow compares the pushes to `main`. To guard another branch,
@@ -595,8 +714,9 @@ duel:commit-comment:
595
714
  # With c.push_base = :last_passed in .duel.rb, a push is compared with the
596
715
  # last commit Duel passed, which a branch marks (duel/passed/main for
597
716
  # main), so slowdowns too small for any one push to fail add up until they
598
- # fail one. This job moves the branch to a push that passed (or makes it).
599
- # To accept a regression, move the branch yourself (for main, git push
717
+ # fail one. This job moves the branch to a push that passed (or makes it),
718
+ # but never back: where a newer push's job moved it first, it stays. To
719
+ # accept a regression, move the branch yourself (for main, git push
600
720
  # origin main:duel/passed/main).
601
721
  duel:passed:
602
722
  image: alpine:3
@@ -607,14 +727,32 @@ duel:passed:
607
727
  variables:
608
728
  GIT_STRATEGY: none # it runs none of the code
609
729
  script:
610
- - apk add --no-cache --quiet curl
730
+ - apk add --no-cache --quiet curl jq
611
731
  - |
612
732
  [ "$DUEL_PASSED" = true ] || exit 0
613
733
  api="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
614
734
  token="PRIVATE-TOKEN: $DUEL_TOKEN"
615
735
  branch="duel/passed/$CI_COMMIT_BRANCH"
736
+ at=$(curl -sf -H "$token" "$api/repository/branches/$(echo "$branch" | sed 's|/|%2F|g')" | jq -r '.commit.id // empty') || at=
737
+ if [ -n "$at" ]; then
738
+ base=$(curl -sf -H "$token" "$api/repository/merge_base?refs%5B%5D=$CI_COMMIT_SHA&refs%5B%5D=$at" | jq -r '.id // empty') || base=
739
+ [ -n "$base" ] || { echo "Couldn't compare $branch with this push" && exit 1; }
740
+ # (A newer push's job moved it, or this one's did, run again.)
741
+ [ "$base" = "$CI_COMMIT_SHA" ] && exit 0
742
+ fi
616
743
  curl -s -o /dev/null -H "$token" -X DELETE "$api/repository/branches/$(echo "$branch" | sed 's|/|%2F|g')"
617
- curl -sf -o /dev/null -H "$token" --data-urlencode "branch=$branch" --data-urlencode "ref=$CI_COMMIT_SHA" "$api/repository/branches"
744
+ curl -sf -o /dev/null -H "$token" --data-urlencode "branch=$branch" --data-urlencode "ref=$CI_COMMIT_SHA" "$api/repository/branches" && exit 0
745
+ # (Git can't make a branch where another's name is a folder of its
746
+ # name: a branch named duel blocks duel/passed/main. If one does,
747
+ # this names it.)
748
+ prefix=$branch
749
+ while [ "${prefix%/*}" != "$prefix" ]; do
750
+ prefix=${prefix%/*}
751
+ if curl -sf -o /dev/null -H "$token" "$api/repository/branches/$(echo "$prefix" | sed 's|/|%2F|g')"; then
752
+ echo "The branch $prefix keeps Duel from making $branch, as Git can't have both. Delete it (git push origin --delete $prefix), or rename it."
753
+ fi
754
+ done
755
+ exit 1
618
756
  ```
619
757
 
620
758
  ## Run each version on its own Ruby on CI
@@ -201,7 +201,7 @@ A prepare block can use the following methods:
201
201
  | `w.failed("message")` | Fails the same way, for any other reason. |
202
202
  | `w.at_stop { ... }` | Runs the block when the worker stops. |
203
203
  | `w.cached(key) { \|dir\| ... }` | Makes something once, for every worker. See the following example. |
204
- | `w.name` | The worker's name, for example `"baseline-1"`. |
204
+ | `w.name` | The worker's name, for example `"base-1"`. |
205
205
  | `w.dir` | The worker's directory. `w.path("name")` returns the path of a file in it. |
206
206
  | `w.root` | The tree of the version that the worker runs. |
207
207
  | `w.ruby` | The path of the Ruby that the version runs on. See [Run each version on its own Ruby](#run-each-version-on-its-own-ruby). |
@@ -223,7 +223,11 @@ Some things take long to make, and are the same for every worker, such
223
223
  as a database's initial files. `w.cached` makes them once. The first
224
224
  worker that asks builds the directory, and the other workers, in this
225
225
  run and in later ones, get its path, to copy or to start from. The key
226
- names what's cached, so put a version in it. `duel --clean` removes
226
+ names what's cached, so put a version in it. If the key ends with a hash
227
+ of what the build depends on, such as `postgres-16.2-3f9c2a1b`, then
228
+ when a run ends, Duel removes the other builds of that name, the part
229
+ before the version and the hash, so an upgrade doesn't leave the old
230
+ build behind. Each build that the run used stays. `duel --clean` removes
227
231
  every cached directory.
228
232
 
229
233
  ```ruby
@@ -267,8 +271,11 @@ Duel.configure do |c|
267
271
  end
268
272
  ```
269
273
 
270
- `v.name` is `"baseline"` or `"candidate"`, and `v.root` is the version's
271
- directory. What the block spawns stops when the run ends. A worker's
274
+ `v.name` is `"base"` (the baseline) or `"head"` (the candidate), and
275
+ `v.root` is the version's directory. The two names are the same length
276
+ on purpose: where a process's stack starts depends on how long its
277
+ arguments and environment are, so a name one byte longer can make every
278
+ worker of one version measurably slower. What the block spawns stops when the run ends. A worker's
272
279
  prepare blocks get the version's environment too, as `w.env`. If a
273
280
  worker's prepare block sets a variable that the version's environment
274
281
  sets to a different value, the run fails.
data/docs/how-it-works.md CHANGED
@@ -73,7 +73,7 @@ Duel judges every benchmark on the following metrics:
73
73
  | `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
74
74
  | `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
75
75
  | `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
76
- | `peak_memory` | The most resident memory, above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
76
+ | `peak_memory` | The most resident memory (on JRuby, the JVM's heap in use), above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
77
77
  | Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
78
78
 
79
79
  Instruction counts barely vary from run to run, so they catch slowdowns
@@ -150,7 +150,9 @@ under a hidden name that's as long as your project's directory name, for
150
150
  example `.3f9c2a1b` next to `my_parser`. Duel keeps it there for the
151
151
  next run. If your project has an `ext/` directory, Duel builds the
152
152
  baseline with `bundle exec rake compile`. To remove the baseline, git's
153
- record of it, and the rest of `tmp/duel`, run `duel --clean`.
153
+ record of it, and the rest of `tmp/duel` (see
154
+ [What Duel keeps in tmp/duel](#what-duel-keeps-in-tmpduel)), run
155
+ `duel --clean`.
154
156
 
155
157
  If your project is in a subdirectory of its repository, as in a
156
158
  monorepo, run `duel` in that subdirectory. Duel checks out the whole
@@ -172,6 +174,27 @@ that small. If Duel can't write next to your project, or something else
172
174
  already has that name, it uses `tmp/duel/base` instead. Whenever the two
173
175
  versions' paths differ in length, Duel says so.
174
176
 
177
+ ### What Duel keeps in tmp/duel
178
+
179
+ Duel keeps what it needs between runs in your project's `tmp/duel`:
180
+
181
+ - `lock`, so that only one run at a time compares the project. A run
182
+ removes it when it ends.
183
+ - `cache/`, what plugins build once and reuse, such as a database's
184
+ initial files. A build is named for what it is, its version, and a
185
+ hash, such as `mysql-0.1.2-3f9c2a1b7e5d4c6a`. When a run ends, Duel
186
+ removes the builds of the same name that the ones it used replace,
187
+ such as the files of a database server that you've since upgraded, so
188
+ the cache keeps one build of each.
189
+ - `base/`, the baseline's checkout, when it can't go next to your
190
+ project.
191
+
192
+ When a run ends, Duel removes `tmp/duel` if nothing is left in it, and
193
+ `tmp` too if Duel made it. A project without plugins that build
194
+ something keeps nothing there. `duel --clean` removes the rest. Each
195
+ run's sockets and server directories are in a temporary directory,
196
+ which the run removes when it ends.
197
+
175
198
  If nothing under `lib/` or `ext/`, your benchmarks, or `.duel.rb`
176
199
  changed since the baseline, there's nothing to compare, and Duel says
177
200
  so. To compare anyway, use `--force`. See
@@ -207,6 +230,11 @@ Duel runs each benchmark for five seconds before it measures it
207
230
  (`--warm`). Their timings stay noisier than CRuby's, so Duel often
208
231
  leaves small timing regressions on them unsettled.
209
232
 
210
- Neither has an allocation count, an instruction count, or profiles, and
211
- each has peak memory only on Linux. StackProf runs on CRuby only: Duel
233
+ Neither has an allocation count, an instruction count, or profiles.
234
+ TruffleRuby has peak memory only on Linux. JRuby measures peak memory in
235
+ the JVM's heap in use, on any operating system, because the JVM takes
236
+ memory for its heap ahead of time, so resident memory barely moves as
237
+ a benchmark's objects fill it. The peak starts from the heap in use
238
+ after a collection that Duel asks the JVM for, as JRuby's `GC.start`
239
+ collects nothing. StackProf runs on CRuby only: Duel
212
240
  doesn't use it on JRuby or TruffleRuby, even where it's installed.
data/docs/json.md CHANGED
@@ -227,6 +227,7 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
227
227
  | `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
228
228
  | `across` | The verdicts across the board, by metric (see below). |
229
229
  | `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
230
+ | `cut_short` | What the run left undone when it reached its time limit (`--time-limit`): `{"unmeasured": [names], "unconfirmed": [names], "across": true or false}`. `unmeasured` are the benchmarks that it didn't measure, which are errors. `unconfirmed` are those that it judged without measuring them again as often as it usually does, and `across` is whether it judged across the board without measuring every benchmark again. `null` for a run that finished within its limit. |
230
231
 
231
232
  A run fails when a benchmark regressed, a benchmark failed, or a metric
232
233
  was slower across the board and confirmed (`"confirmed": true`).
@@ -241,6 +242,7 @@ was slower across the board and confirmed (`"confirmed": true`).
241
242
  | `jobs` | Benchmarks measured at once (`--jobs`). |
242
243
  | `processes` | Worker processes for each version (`--processes`). |
243
244
  | `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
245
+ | `time_limit` | The seconds that a run measures for, at most (`--time-limit`). |
244
246
  | `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default, which depends on the Ruby. |
245
247
  | `force` | Whether `--force` was given. |
246
248
 
data/docs/plugins.md CHANGED
@@ -303,10 +303,10 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
303
303
  `Duel::Testing` has the following methods:
304
304
 
305
305
  - `prepare(config)`: runs the configuration's `c.prepare(:version)`
306
- blocks, named for the version (`"candidate"`), and then its
306
+ blocks, named for the version (`"head"`), and then its
307
307
  `c.prepare(:worker)` blocks, as Duel prepares a worker. It sets the
308
308
  environment that they return while its block runs, and stops what
309
- they started after. `name:` names the worker (`"candidate-1"` by
309
+ they started after. `name:` names the worker (`"head-1"` by
310
310
  default), and `root:` is the version's tree (the configuration's root
311
311
  by default). `cache:` is where `w.cached` builds go. By default,
312
312
  that's one directory for the whole test process. To look inside it,
@@ -48,6 +48,20 @@ worker and reports the benchmark as an error. To change the limit, use
48
48
  `--timeout`. If a benchmark's iterations are slow, split it into
49
49
  smaller benchmarks.
50
50
 
51
+ ## The run reached its time limit
52
+
53
+ A run measures for 30 minutes at most (`--time-limit`). Past that, Duel
54
+ starts no more measurements: it fails the benchmarks it hasn't measured,
55
+ and judges each benchmark that looked worse on the measurements it has.
56
+ The report starts the list of what it left undone with `TIME LIMIT`.
57
+
58
+ A run usually takes that long because a benchmark is slow to settle. A
59
+ benchmark that starts processes of its own, such as one that forks
60
+ workers, spends most of its time waiting for them, and Duel measures
61
+ each one again when it looks worse. Give it less work per call, or
62
+ fewer calls, so that each sample takes less time. If your benchmarks
63
+ need more time, raise the limit with `--time-limit SECONDS`.
64
+
51
65
  ## Duel tells you to set c.build
52
66
 
53
67
  Duel checks that each version's workers load only native extensions that
@@ -58,6 +72,20 @@ happens, Duel stops and tells you to set `c.build` to a command that
58
72
  builds each version, such as `c.build = "make"`. See
59
73
  [Prepare each version](configuration.md#prepare-each-version).
60
74
 
75
+ ## Duel says a version loaded an extension twice
76
+
77
+ A worker loads `.duel.rb` and your plugins before it loads its version.
78
+ If one of them requires your library, or a gem that requires it, the
79
+ worker finds the library wherever the load path leads, such as the
80
+ candidate's `lib/` or an installed release. Duel then loads the
81
+ version's own build, but a native extension can't be unloaded, so both
82
+ stay loaded, and the comparison wouldn't be of the version's code alone.
83
+ Duel stops and names the extension and both of its files.
84
+
85
+ To fix it, require what loads your library in a block that runs once the
86
+ version has loaded, such as `c.in_worker`, not at the top of `.duel.rb`
87
+ or as a plugin loads.
88
+
61
89
  ## Duel says that the versions' paths differ in length
62
90
 
63
91
  The baseline's path is as long and as deep as your project's, because
data/lib/duel/baseline.rb CHANGED
@@ -124,13 +124,13 @@ module Duel
124
124
  sibling = File.join(File.dirname(top), ".#{Digest::SHA256.hexdigest(root).ljust(length, "0")[0, length]}")
125
125
  return sibling if File.writable?(File.dirname(top)) && (!File.exist?(sibling) || ours?(sibling))
126
126
 
127
- File.join(@config.root, "tmp", "duel", "base")
127
+ File.join(@config.state, "base")
128
128
  end
129
129
 
130
- # Removes what Duel keeps: tmp/duel, and the baseline's checkout (and
131
- # git's record of it as a worktree).
130
+ # Removes what Duel keeps (Config#state: tmp/duel), and the baseline's
131
+ # checkout (and git's record of it as a worktree).
132
132
  def clean
133
- FileUtils.rm_rf(File.join(@config.root, "tmp", "duel"))
133
+ FileUtils.rm_rf(@config.state)
134
134
  FileUtils.rm_rf(checkout)
135
135
  git("worktree", "prune", allow_failure: true) # (not a git repository, say)
136
136
  end
data/lib/duel/cli.rb CHANGED
@@ -76,7 +76,7 @@ module Duel
76
76
 
77
77
  settings = @settings = catch(:version) { parse } or return 0
78
78
  config = @config = settings.config_path ? Duel.load_config(settings.config_path) : Config.new
79
- Lock.hold(File.join(config.root, "tmp", "duel")) { compare(config, settings) }
79
+ Lock.hold(config.state) { compare(config, settings) }
80
80
  rescue Error => e
81
81
  @err.puts "duel: #{e}"
82
82
  unfinished(Summary::FAILED_MARK, "couldn't compare", "error", e.to_s, 2)
@@ -451,6 +451,9 @@ module Duel
451
451
  o.on("--jobs N", Integer, "Benchmarks measured at once (default: half the CPUs)") { |v| settings.jobs = v }
452
452
  o.on("--processes N", Integer, "Worker processes per version (default: 3)") { |v| settings.processes = v }
453
453
  o.on("--timeout SECONDS", Float, "Most a worker may take to load or answer (default: 60)") { |v| settings.timeout = v }
454
+ o.on("--time-limit SECONDS", Float, "Most a run measures for, then fails what's unmeasured or looks worse (default: 1800)") do |v|
455
+ settings.time_limit = v
456
+ end
454
457
  o.on("--limit RATIO", Float, "Ratio above which a timing is slower (default: 1.02)") { |v| settings.limit = v }
455
458
  o.on("--warm SECONDS", Float, "Warm-up per benchmark (default: 5 on JRuby and TruffleRuby, 2 with YJIT on, else 0)") { |v| settings.warm = v }
456
459
  o.on("--ruby version-file|PATH", "Each version's Ruby: the one its version file names (.ruby-version, say), or a",