duel 0.1.5 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2528f9b36dd187ea88916fa507ab544b243a58b6c74cfe9c75083c127d9c3bbb
4
- data.tar.gz: ca3838705ef0db2b367c115e96e555fb346173be3aa7a1a1f1740e7a5c895a40
3
+ metadata.gz: a3bd63fc0a6b4eb33d7fe48f733446e7b35c3e961dfdc42e85531fdc680ca53c
4
+ data.tar.gz: d44709c7629208381a9e24235f2401dc53c69c11fd540c8b8ab5c4e02710bf7a
5
5
  SHA512:
6
- metadata.gz: 0663ad25b83ffa57ff1fdedb64c462c8fea600967d26ec49831071bf69fa4ebfbbf2a9a17f8ed1bdfbb3c8bb7a55757cea1fd9a6ad3937bb435ed849dda4ab55
7
- data.tar.gz: d2b0f54723cfe98dbffa56103b5699bc532b9d305c2d75f5304d2749de0f1c455ab12c6250afbf8932537b7a883a6101bf61f242a0e7c5f24f65f866d60e6fb5
6
+ metadata.gz: 44fd8dbe608296d4af112bd27525f38d544662e858e39abfc4913583fa1a0d5d74cd9c20c3745dcc226ef1c6f89c6deb9f2f6780bb457627da18e0b7bf82373e
7
+ data.tar.gz: 6984d61ce88c4042e8447a9ec03886bb68fa39d4e2d5bdef448a6b4dd1cd7ef6d258a940aac56b3bd08892907fc4d626cd4f91dcbbcbca634b29c78227c7f860
data/CHANGELOG.md CHANGED
@@ -7,6 +7,114 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.6] - 2026-11-10
11
+
12
+ ### Added
13
+
14
+ - The comment workflow that `duel init` writes runs the Duel job again
15
+ when the label that accepts a regression (`c.accept_label`) is added
16
+ to a pull request whose run failed, or taken off one whose run
17
+ passed. Before, the label did nothing until someone ran the job again
18
+ by hand. The job runs on `pull_request_target`, checks nothing out,
19
+ and has only `actions: write`.
20
+ - On a schedule, the workflow compares with the last commit since the
21
+ last release that merged a pull request with the label, instead of
22
+ the release, so a slowdown accepted in its pull request doesn't fail
23
+ every nightly run until the next release. Duel reads that commit from
24
+ `DUEL_ACCEPTED`, and the report says what it compared with.
25
+ - The JSON report says, for each timing and across the board, how many
26
+ worker processes of each version the interval is over (`processes`),
27
+ or `null` where it's over pairs or benchmarks.
28
+ - The nightly bias check (`tasks/bias_check.rb`) also fails where more
29
+ of a metric's intervals of identical code leave out 1 than their
30
+ confidence allows, and runs on JRuby too, and on a pull request that
31
+ changes how Duel measures or judges.
32
+
33
+ - On JRuby, Duel counts the bytes each benchmark allocates
34
+ (`allocated_bytes`), as the JVM counts them for each thread (or every
35
+ thread, for a benchmark with threads of its own), and fails one that
36
+ allocates more than 32 bytes more an iteration. Before, JRuby had no
37
+ allocation count. Where each JVM's JIT does away with different
38
+ allocations (escape analysis), workers' counts differ, and Duel finds
39
+ only as many more as they can tell apart.
40
+ - A count is over workers too: each worker's fewest in three rounds of
41
+ a sample from every worker, and it fails when an interval for how many
42
+ more the candidate's workers count than the baseline's lies above the
43
+ tolerance. Before, it was each version's fewest of 3 samples, and on
44
+ JRuby, where one worker's JIT can do away with an allocation another's
45
+ doesn't, one worker's luck could fail identical code. Where every
46
+ worker counts the same, as on CRuby, it's as exact as before. With
47
+ fewer than 5 workers of each version, it's the fewest of 3 samples (10
48
+ on JRuby), as before.
49
+ - On JRuby and TruffleRuby, a worker may take 5 minutes to load or to
50
+ answer (`--timeout`), not 1: with `-Xbatch`, 16 JVMs starting at once
51
+ could take longer than a minute.
52
+ - Counts of 9999.5 or more an iteration show as whole numbers (25736, not
53
+ 2.574e+04).
54
+
55
+ ### Changed
56
+
57
+ - A timing's interval is over worker processes, not pairs of samples.
58
+ Where things land in a process's memory (or on JRuby, what its JIT
59
+ compiles) can make the same code faster or slower for the process's
60
+ whole life, so the pairs of 3 workers of each version were far from
61
+ independent, and with identical code on both sides, their intervals
62
+ left out 1 several times as often as 99% intervals should. Each
63
+ worker's median is now a sample, and the interval is the Mann-Whitney
64
+ interval for the ratio of the candidate's workers to the baseline's.
65
+ Duel takes rounds of a sample from every worker, up to some 120
66
+ pairs, and starts 12 workers of each version by default (8 on JRuby
67
+ and TruffleRuby). CPU-time intervals are about as wide as before, or
68
+ narrower, and instruction counts' (which a process repeats almost
69
+ exactly) a little wider. On JRuby, they're some 5% to 8% wide, as
70
+ wide as each JVM's JIT makes them, so Duel leaves smaller timing
71
+ regressions there unsettled. With fewer than 5 workers of each version
72
+ (`--processes`, and `Duel::Testing.compare`'s 2), and for a process
73
+ benchmark, the interval is over the pairs, as before, and the first
74
+ line of the output says which.
75
+ - Across the board, the interval is over the workers too: each worker's
76
+ score is how much faster or slower it was than its lane's other
77
+ workers, at the median of the benchmarks it measured. A worker whose
78
+ luck slowed it at every benchmark could fail identical code across
79
+ the board.
80
+ - Without `--warm`, each worker warms a benchmark up until it's steady
81
+ (the median of its samples over each of three half seconds in a row
82
+ is no more than 1% faster than over the last half second that was),
83
+ for up to a minute on JRuby and TruffleRuby, or 10 seconds with YJIT
84
+ on. Before, it warmed up for 5 seconds, or 2, every time, and on
85
+ JRuby, some code was still several times faster after that, so it
86
+ was measured while it got faster. `--warm` still warms up for the time
87
+ it says.
88
+ - On JRuby, each worker's JVM compiles in the foreground (`-Xbatch`) and
89
+ collects garbage with the serial collector, after any options in
90
+ `JRUBY_OPTS` (and without the serial collector where they choose
91
+ another). Each JVM's JIT compiles the same code a little differently,
92
+ and that made JRuby's intervals over workers twice as wide.
93
+ - A run measures for two hours at most by default (`--time-limit`), not
94
+ half an hour, as on JRuby a benchmark can take half a minute to warm
95
+ up.
96
+ - On JRuby and TruffleRuby, Duel measures one benchmark at a time for
97
+ every four CPUs (`--jobs`), not every two: their JIT compilers keep the
98
+ other CPUs busy, so more were no faster, and each worker is a JVM of
99
+ some 500 MB.
100
+
101
+ ### Fixed
102
+
103
+ - The comment workflow fences a fork's report with more backticks than
104
+ any run of them in it even when the report has a NUL byte. GNU grep
105
+ took such a report for binary and found no backticks, so the fence
106
+ was three, and a line of three backticks in the report closed it. The
107
+ workflow now drops NUL bytes, which GitHub doesn't take in a comment,
108
+ and reads the report as text whatever's in it.
109
+ - The Mark the commit passed job runs one at a time for each branch
110
+ (and on GitLab CI, one at a time). Before the branch that marks the
111
+ last commit Duel passed existed, two pushes that passed at once could
112
+ both try to make it, and the second failed, saying another branch
113
+ kept it from being made. Where making the branch fails, the job now
114
+ also looks for it again before it says so. Update the job in
115
+ `.github/workflows/perf.yml` (or `.gitlab/duel.gitlab-ci.yml`) to pick
116
+ this up.
117
+
10
118
  ## [0.1.5] - 2026-11-08
11
119
 
12
120
  ### Changed
@@ -574,7 +682,8 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
574
682
  Anything marked `@api private`, in lib/ and sig/, may change in any
575
683
  release.
576
684
 
577
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.5...HEAD
685
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.6...HEAD
686
+ [0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
578
687
  [0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
579
688
  [0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
580
689
  [0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
data/README.md CHANGED
@@ -14,7 +14,7 @@ code from becoming slower.
14
14
 
15
15
  ```sh
16
16
  $ duel
17
- Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02)
17
+ Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02, over 12 worker processes of each version)
18
18
  parse: a small document 0.6s n=412 cpu 0.998 [0.991, 1.006] same allocations 20
19
19
  ! parse: a large document 1.1s n=9 cpu 1.081 [1.062, 1.097] slower allocations 5790
20
20
  ...
@@ -193,11 +193,11 @@ and how it compares a project in a subdirectory of its repository, see
193
193
  ### Timeouts and exit status
194
194
 
195
195
  A benchmark that hangs, or that waits on a server that died, doesn't
196
- hang the run. If a worker takes more than a minute to load or to answer,
197
- Duel stops the worker and reports the benchmark as an error. To change
198
- the limit, use `--timeout`.
196
+ hang the run. If a worker takes more than a minute (five on JRuby and
197
+ TruffleRuby) to load or to answer, Duel stops the worker and reports the
198
+ benchmark as an error. To change the limit, use `--timeout`.
199
199
 
200
- A run measures for 30 minutes at most, not counting the builds. Past
200
+ A run measures for two hours at most, not counting the builds. Past
201
201
  that, Duel starts no more measurements. A benchmark that it hasn't
202
202
  measured yet fails as an error, and a benchmark that looked worse, but
203
203
  that Duel hasn't measured again as often as it usually does, fails if it
data/docs/ci.md CHANGED
@@ -61,23 +61,33 @@ jobs:
61
61
  with:
62
62
  fetch-depth: 0 # the baseline comes from history, tags included
63
63
  persist-credentials: false
64
- # (And whatever else your tests need set up: packages, services.)
65
- - uses: ruby/setup-ruby@v1
66
- with:
67
- ruby-version: "3.4" # (or none, with a .ruby-version)
68
- bundler-cache: true
69
64
  # The labels of the pull request (or for a push, of the one that
70
65
  # merged it), for c.accept_label in .duel.rb, as a JSON array. They're
71
- # read as the job runs, so after adding the label, run it again.
66
+ # read as the job runs, and adding the label (or taking it off) runs
67
+ # the job again: see the Perf comment workflow.
68
+ # On a schedule, the last commit since the last release that merged a
69
+ # pull request with the label (ACCEPT_LABEL, as .duel.rb names it):
70
+ # what that slowed down was accepted, so it's compared with instead.
72
71
  - name: Find the pull request's labels
73
72
  id: labels
74
- if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push'
73
+ if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push' || github.event_name == 'schedule'
75
74
  continue-on-error: true # (without them, nothing's accepted)
76
75
  env:
77
76
  GH_TOKEN: ${{ github.token }}
78
77
  PR: ${{ github.event.pull_request.number }}
79
78
  QUEUED: ${{ github.event.merge_group.head_ref }}
79
+ ACCEPT_LABEL: 'duel: accept'
80
+ RELEASE_TAGS: "v*"
80
81
  run: |
82
+ if [ "$GITHUB_EVENT_NAME" = schedule ]; then
83
+ tag=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || exit 0
84
+ since=$(TZ=UTC git log -1 --date=format-local:%Y-%m-%dT%H:%M:%SZ --format=%cd "$tag")
85
+ merged=$(gh pr list --repo "$GITHUB_REPOSITORY" --state merged --label "$ACCEPT_LABEL" --search "merged:>=$since" --limit 1000 \
86
+ --json mergeCommit --jq '.[].mergeCommit.oid')
87
+ accepted=$(git rev-list "$tag..HEAD" | grep -Fx -m 1 -f <(echo "$merged")) || accepted=
88
+ echo "accepted=$accepted" >> "$GITHUB_OUTPUT"
89
+ exit 0
90
+ fi
81
91
  # (In a merge queue, the last pull request queued, which the queue's
82
92
  # branch names: gh-readonly-queue/main/pr-123-<sha>.)
83
93
  if [ -n "$QUEUED" ]; then
@@ -90,6 +100,11 @@ jobs:
90
100
  labels=$(gh api "repos/$GITHUB_REPOSITORY/commits/$GITHUB_SHA/pulls" --jq '[.[0].labels[]?.name]')
91
101
  fi
92
102
  echo "labels=$labels" >> "$GITHUB_OUTPUT"
103
+ # (And whatever else your tests need set up: packages, services.)
104
+ - uses: ruby/setup-ruby@v1
105
+ with:
106
+ ruby-version: "3.4" # (or none, with a .ruby-version)
107
+ bundler-cache: true
93
108
  # Against the branch it targets for a pull request, the queue's base
94
109
  # for a merge queue, what was there before for a push (or as
95
110
  # c.push_base in .duel.rb says, the last commit Duel passed), and the
@@ -105,6 +120,7 @@ jobs:
105
120
  # with itself, which checks the setup.)
106
121
  DUEL_FORCE: ${{ github.event_name == 'workflow_dispatch' && '1' || '' }}
107
122
  DUEL_LABELS: ${{ steps.labels.outputs.labels }}
123
+ DUEL_ACCEPTED: ${{ steps.labels.outputs.accepted }}
108
124
  run: bundle exec duel --markdown "$RUNNER_TEMP/duel/duel.md"
109
125
  # For a pull request, what Duel said, for the Perf comment workflow,
110
126
  # which can't see this job's outputs.
@@ -149,8 +165,11 @@ The workflow compares the following:
149
165
  the last commit that Duel passed. If that commit isn't in the history,
150
166
  such as after a force push or for a branch's first push, Duel reports
151
167
  that there's nothing to compare against, and passes.
152
- - On a schedule, the last release. Until you tag your first release,
153
- Duel reports that there's nothing to compare against, and passes.
168
+ - On a schedule, the last release, or with
169
+ [`c.accept_label`](#accept-a-regression-with-a-label), the last commit
170
+ since then that merged a pull request with the label. Until you tag
171
+ your first release, Duel reports that there's nothing to compare
172
+ against, and passes.
154
173
  - When you run the workflow manually, your code against itself. This
155
174
  checks that the benchmarks run on the CI machines.
156
175
 
@@ -237,12 +256,23 @@ name: Perf comment
237
256
  # ran), it trusts only what that job could have said: the pull request's
238
257
  # number, if it names the pull request the run compared, and the comment
239
258
  # and the status, if they're ones Duel says. A fork's report, which the
240
- # fork's code wrote, is shown as text. (The workflow must be on the
241
- # default branch before it runs at all.)
259
+ # fork's code wrote, is shown as text.
260
+ #
261
+ # And when the label that accepts a regression (c.accept_label in
262
+ # .duel.rb) is added to a pull request, or taken off, it runs the Duel
263
+ # job again, which reads the labels as it runs: a run that failed, once
264
+ # the label's added, or one that passed, once it's taken off.
265
+ #
266
+ # (The workflow must be on the default branch before it runs at all.)
242
267
  on:
243
268
  workflow_run:
244
269
  workflows: ["Perf"]
245
270
  types: [completed]
271
+ # (This repository's own workflow, as for the comment, with a token
272
+ # that can run a workflow again: it checks nothing out, and runs none
273
+ # of the pull request's code.)
274
+ pull_request_target:
275
+ types: [labeled, unlabeled]
246
276
 
247
277
  permissions: {}
248
278
 
@@ -283,12 +313,16 @@ jobs:
283
313
  # ran the fork's code, which could have written anything in it. In
284
314
  # a code block, fenced with more backticks than any run of them in
285
315
  # the report, it can't mention anyone, link anywhere or show an
286
- # image. (And it's cut short where a comment couldn't hold it.)
316
+ # image. It's cut short where a comment couldn't hold it, and loses
317
+ # any NUL bytes, which GitHub wouldn't take, and grep reads it as
318
+ # text whatever's in it, so it finds every run of backticks.
287
319
  report() {
288
320
  if [ "${head#* }" = "$GITHUB_REPOSITORY" ]; then cat "$RUNNER_TEMP/report/duel.md" && return; fi
289
- fence=$(grep -o '``*' "$RUNNER_TEMP/report/duel.md" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
321
+ text="$RUNNER_TEMP/report.txt"
322
+ LC_ALL=C tr -d '\000' < "$RUNNER_TEMP/report/duel.md" | head -c 60000 > "$text"
323
+ fence=$(LC_ALL=C grep -ao '``*' "$text" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
290
324
  printf 'From a fork, so shown as text:\n\n%s\n' "$fence"
291
- head -c 60000 "$RUNNER_TEMP/report/duel.md"
325
+ cat "$text"
292
326
  printf '\n%s\n' "$fence"
293
327
  }
294
328
  marker="<!-- duel"
@@ -300,6 +334,33 @@ jobs:
300
334
  { echo "$marker status=$STATUS -->"; report; } > "$RUNNER_TEMP/comment.md"
301
335
  gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
302
336
  fi
337
+
338
+ again:
339
+ name: Run Duel again
340
+ if: >-
341
+ github.event_name == 'pull_request_target' && github.event.label.name == 'duel: accept'
342
+ runs-on: ubuntu-latest
343
+ permissions:
344
+ actions: write # to run the Perf workflow's jobs again
345
+ steps:
346
+ - name: Run Duel again
347
+ env:
348
+ GH_TOKEN: ${{ github.token }}
349
+ GH_REPO: ${{ github.repository }}
350
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
351
+ ACTION: ${{ github.event.action }}
352
+ run: |
353
+ # The newest Perf run of the pull request's head commit, once it's
354
+ # done (one that's running may have read the labels already).
355
+ id=$(gh run list --workflow "Perf" --event pull_request --commit "$HEAD_SHA" --limit 1 \
356
+ --json databaseId --jq '.[0].databaseId // empty')
357
+ [ -n "$id" ] || exit 0
358
+ gh run watch "$id" --interval 30 > /dev/null || true
359
+ conclusion=$(gh run view "$id" --json conclusion --jq .conclusion)
360
+ case "$ACTION:$conclusion" in
361
+ labeled:failure) gh run rerun "$id" --failed ;;
362
+ unlabeled:success) gh run rerun "$id" ;;
363
+ esac
303
364
  ```
304
365
 
305
366
  ## Comment on a pushed commit
@@ -374,6 +435,23 @@ or for a push, of the pull request that merged it, and passes them to
374
435
  `pull-requests: read` permission. Without the labels, such as when
375
436
  GitHub's API fails, nothing is accepted.
376
437
 
438
+ The labels are read as the Duel job runs, so `perf-comment.yml` runs
439
+ the job again when the label changes. Its Run Duel again job runs on
440
+ `pull_request_target`, with this repository's own workflow, when a
441
+ label is added to a pull request or taken off. For the label `.duel.rb`
442
+ names, it waits for the Perf run of the pull request's head commit to
443
+ finish. Then it runs the failed jobs again where the label was added
444
+ to a run that failed, or the whole run again where the label was
445
+ taken off one that passed. It checks nothing out and runs none of the
446
+ pull request's code, and its only permission is `actions: write`. The
447
+ workflows name the label as `.duel.rb` did when `duel init` wrote them,
448
+ so if you rename it, rename it in both workflows too.
449
+
450
+ On a schedule, the same step looks for the pull requests merged since
451
+ the last release with the label, and passes `duel` the last commit
452
+ that merged one, in `DUEL_ACCEPTED`. Duel compares with that commit
453
+ instead of the release, as what it slowed down was accepted.
454
+
377
455
  ## Compare a push with the last commit that passed
378
456
 
379
457
  By default, the workflow compares each push with the commit before it.
@@ -390,8 +468,10 @@ branch that the workflow guards has its own (see
390
468
  nothing to compare, Duel sets the `passed` output to `true`, and the
391
469
  Mark the commit passed job moves the branch to the push, or creates the
392
470
  branch. It never moves the branch back: when two pushes run at once and
393
- the older one finishes last, the branch stays at the newer push. After
394
- a force push rewrites the branch's history, it moves the branch anyway.
471
+ the older one finishes last, the branch stays at the newer push. The
472
+ job runs one at a time for each branch, so two runs can't race to
473
+ create the branch. After a force push rewrites the branch's history,
474
+ it moves the branch anyway.
395
475
  That job checks nothing out either, and it gets the `contents: write`
396
476
  permission that moving a branch takes. Until the branch exists, and
397
477
  when the push's history doesn't have its commit, Duel compares the push
@@ -438,6 +518,13 @@ yourself:
438
518
  passed:
439
519
  name: Mark the commit passed
440
520
  needs: duel
521
+ # One at a time for each branch, so that two runs can't both find no
522
+ # branch and race to make it (a push's run doesn't wait for the one
523
+ # before it). One that's running is never cancelled, and one that's
524
+ # waiting gives way to a newer push's, which moves it further anyway.
525
+ concurrency:
526
+ group: ${{ github.workflow }}-passed-${{ github.ref_name }}
527
+ cancel-in-progress: false
441
528
  if: ${{ !cancelled() && github.event_name == 'push' && needs.duel.outputs.passed == 'true' }}
442
529
  runs-on: ubuntu-latest
443
530
  permissions:
@@ -461,13 +548,16 @@ yourself:
461
548
  done
462
549
  exit 1
463
550
  }
464
- # (Forward: GitHub moves it only where it's behind the push.)
465
- gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
466
- at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
467
- if [ -z "$at" ]; then
468
- gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null || blocked
469
- exit 0
470
- fi
551
+ # (Forward: GitHub moves it only where it's behind the push. Where
552
+ # there's no branch, and making it fails, it looks once more, as
553
+ # something else may have made it: a push of it, say.)
554
+ for _ in 1 2; do
555
+ gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
556
+ at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
557
+ [ -n "$at" ] && break
558
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null && exit 0
559
+ done
560
+ [ -n "$at" ] || blocked
471
561
  compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status 2>&1) || case "$compared" in
472
562
  # (GitHub can't compare commits with no history in common.)
473
563
  *"No common ancestor"*) compared=unrelated ;;
@@ -825,6 +915,9 @@ duel:passed:
825
915
  image: alpine:3
826
916
  stage: .post
827
917
  needs: [duel]
918
+ # One at a time, as each deletes the branch and makes it again: two at
919
+ # once could both delete it, and the second to make it would fail.
920
+ resource_group: duel/passed
828
921
  rules:
829
922
  - if: $DUEL_TOKEN && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
830
923
  variables:
@@ -894,7 +987,8 @@ match. `duel init` adds these steps for a project that has a
894
987
  # request targets (or run by hand, the main branch), a merge queue's
895
988
  # base, the commit before a push (or the last commit Duel passed, PASSED, if there is
896
989
  # one: see the Mark the commit passed job), or on a schedule the last
897
- # release, the newest tag before HEAD that RELEASE_TAGS matches. (If
990
+ # release, the newest tag before HEAD that RELEASE_TAGS matches
991
+ # (or ACCEPTED, the last change accepted since then). (If
898
992
  # .duel.rb sets c.main_branch or c.release_tags, set MAIN_BRANCH or
899
993
  # RELEASE_TAGS to match.)
900
994
  - name: Find the baseline's Ruby
@@ -905,13 +999,14 @@ match. `duel init` adds these steps for a project that has a
905
999
  PASSED: origin/duel/passed/${{ github.ref_name }}
906
1000
  MAIN_BRANCH: origin/${{ github.base_ref || github.event.repository.default_branch }}
907
1001
  RELEASE_TAGS: "v*"
1002
+ ACCEPTED: ${{ steps.labels.outputs.accepted }}
908
1003
  run: |
909
1004
  case "$GITHUB_EVENT_NAME" in
910
1005
  push)
911
1006
  base=$BEFORE
912
1007
  if git merge-base --is-ancestor "$PASSED" HEAD 2>/dev/null; then base=$PASSED; fi ;;
913
1008
  merge_group) base=$QUEUE_BASE ;;
914
- schedule) base=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || base= ;;
1009
+ schedule) base=${ACCEPTED:-$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null)} || base= ;;
915
1010
  *) base=$(git merge-base HEAD "$MAIN_BRANCH" 2>/dev/null) || base= ;;
916
1011
  esac
917
1012
  ruby_of() { git show "$1:./.ruby-version" 2>/dev/null | awk 'NF { print $1; exit }'; }
@@ -140,12 +140,16 @@ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
140
140
  that `duel init` writes passes Duel the pull request's labels. With
141
141
  that label, a regression passes, with a warning, and the report says
142
142
  that the label accepted it. The labels are read when the workflow runs,
143
- so after you add the label, run the workflow again (Re-run jobs, on the
144
- pull request's Checks tab). For a push, Duel reads the labels of the
145
- pull request that merged it, so with
143
+ so the comment workflow runs the Duel job again when you add the label
144
+ to a pull request whose run failed, or take it off one whose run
145
+ passed. For a push, Duel reads the labels of the pull request that
146
+ merged it, so with
146
147
  [`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
147
148
  the merged regression becomes the commit that later pushes are compared
148
- with. Duel's default is no such label, but the `.duel.rb` that
149
+ with. On a schedule, Duel compares with the last commit since your last
150
+ release that merged a pull request with the label, instead of the
151
+ release, so a slowdown you accepted doesn't fail every night until
152
+ your next release. Duel's default is no such label, but the `.duel.rb` that
149
153
  `duel init` writes (and duel-rails's generator) names `duel: accept`.
150
154
  Create the label in your repository to use it, or set
151
155
  `c.accept_label = nil` so that nothing can let a regression through.
data/docs/how-it-works.md CHANGED
@@ -17,25 +17,40 @@ never on noise, so that it can gate every pull request:
17
17
  only ever adds time. Garbage collection is kept out of samples, and
18
18
  its cost shows in the allocation count instead, which is exact.
19
19
  - **Side by side, many times over.** The two versions run in worker
20
- processes of their own, three of each by default, at the same time on
21
- the same machine. They take turns: one sample from each, in
22
- alternating order, and each pair from a different pair of processes.
23
- That way, a busy moment weighs on both versions alike, and no one
24
- process's luck decides anything. (Where things land in a process's
25
- memory can make the very same code a few percent faster or slower for
26
- the process's whole life.) There are no stored numbers and no
27
- thresholds to maintain, and any machine will do.
28
- - **A confidence interval, not an average.** A timing is slower only
29
- when a 99% distribution-free confidence interval for the median of
30
- the paired ratios (candidate / baseline) lies entirely above the
31
- limit, which is 1.02 by default. No one outlier sample can decide it.
32
- Duel takes samples in rounds of 10 pairs, up to 60, until the interval
33
- settles the question. A process benchmark's sample is the fastest of
34
- a few fresh processes, and it can also stop after any pair once the
20
+ processes of their own, at the same time on the same machine: twelve
21
+ of each by default, or eight on JRuby and TruffleRuby, whose workers
22
+ are JVMs. They take turns, one sample from each in alternating order,
23
+ in rounds. Each round takes a sample from every worker, and pairs
24
+ each baseline worker with a different candidate worker than the
25
+ last round did. That way, a busy moment weighs on both versions
26
+ alike. There are no stored numbers and no thresholds to maintain,
27
+ and any machine will do.
28
+ - **A confidence interval over processes, not an average.** Where
29
+ things land in a process's memory can make the very same code a few
30
+ percent faster or slower for the process's whole life, and on JRuby,
31
+ so can what its JIT compiled. So a sample from one process isn't
32
+ independent of the next sample from it, and the unit Duel judges is
33
+ the process: each worker's median. A timing is slower only when a
34
+ 99% distribution-free confidence interval for the ratio of the
35
+ candidate's workers to the baseline's (the Mann-Whitney interval,
36
+ over every candidate worker's median against every baseline
37
+ worker's) lies entirely above the limit, which is 1.02 by default.
38
+ No one outlier sample can decide it, and no one lucky or unlucky
39
+ worker can either: the interval is as wide as the workers make it.
40
+ An interval over the pairs would be as narrow as if every pair had
41
+ processes of its own, and with identical code on both sides, it
42
+ would often leave out 1. Duel takes rounds until the interval
43
+ settles the question, from the fourth round on, and up to some 120
44
+ pairs. Duel reports a timing that's still too close to the limit to
45
+ call, and doesn't fail on it.
46
+
47
+ The interval over workers takes five of each at least. With fewer
48
+ (`--processes`), Duel judges over the pairs, as it does a process
49
+ benchmark, and its first line says so. A process benchmark's
50
+ sample is the fastest of a few fresh processes, so its pairs are
51
+ independent already. It can also stop after any pair once the
35
52
  interval shows that it isn't slower, which only ever ends a
36
53
  measurement before it could find a slowdown.
37
- Duel reports a timing that's still too close to the limit to call, and
38
- doesn't fail on it.
39
54
  - **Measured again before failing.** Duel measures whatever looks
40
55
  slower again from scratch, up to three times, in fresh processes. It
41
56
  fails only if every measurement finds the same metric worse. A
@@ -54,15 +69,27 @@ never on noise, so that it can gate every pull request:
54
69
  than it otherwise would.
55
70
  - **Exact counts.** Allocations (on CRuby), and any counted metric of
56
71
  your own, fail on any increase: one more allocation per two
57
- iterations.
72
+ iterations. Each worker's count is its fewest in three rounds of a
73
+ sample from every worker, and a count fails when an interval over the
74
+ workers, for how many more the candidate's are, lies above the
75
+ tolerance. Where every worker counts the same, as CRuby's workers
76
+ do, that's exactly how many more. Where workers differ, it's as few
77
+ as they can tell apart, so one worker's luck can't fail a benchmark.
78
+ With fewer than five workers of each version, a count is each
79
+ version's fewest in three samples instead.
58
80
  - **Across the board, too.** A slowdown that's too small for any one
59
81
  benchmark to settle, such as 1% everywhere, still shows in all of the
60
- benchmarks together. For each metric that's judged by ratio, every
61
- benchmark's median ratio is a sample. The candidate fails when a
62
- confidence interval for their median lies above a quarter of the
63
- metric's margin: 1.005, for CPU time's 1.02. This takes 8 benchmarks,
64
- so with fewer, Duel doesn't report it. Duel measures again before it
65
- fails, as it does for the rest.
82
+ benchmarks together. The same workers measure every benchmark, so a
83
+ worker whose luck makes it slower is slower at all of them, and the
84
+ unit is the worker again. Each worker has a score: the median, over
85
+ the benchmarks it measured, of how much faster or slower it was than
86
+ the other workers of its lane. The candidate fails when a confidence
87
+ interval for the ratio of its workers' scores to the baseline's lies
88
+ above a quarter of the metric's margin: 1.005, for CPU time's 1.02.
89
+ Where the benchmarks were judged over pairs, every benchmark's median
90
+ ratio is a sample instead, and the interval is for their median.
91
+ Either takes 8 benchmarks, so with fewer, Duel doesn't report it.
92
+ Duel measures again before it fails, as it does for the rest.
66
93
 
67
94
  ## What Duel measures
68
95
 
@@ -73,6 +100,7 @@ Duel judges every benchmark on the following metrics:
73
100
  | `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
74
101
  | `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
75
102
  | `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
103
+ | `allocated_bytes` | Bytes allocated (on JRuby), by the JVM's count | More than 32 bytes more per iteration |
76
104
  | `peak_memory` | The most resident memory (on JRuby, the JVM's heap in use), above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
77
105
  | Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
78
106
 
@@ -216,21 +244,67 @@ when the gems are installed in the project's directory, as
216
244
 
217
245
  ## JIT compilers
218
246
 
219
- If YJIT is on in a worker, Duel runs each benchmark for two seconds
220
- before it measures it. A small library's methods are compiled within
221
- Duel's first samples, but in a big app, YJIT compiles a method only after
222
- 120 calls. YJIT can be on because you turned it on (`RUBY_YJIT_ENABLE=1`,
223
- for example), or because your code did as it loaded, as Rails 7.2 and
224
- later do. To warm up for a different time, use `--warm`.
247
+ If YJIT is on in a worker, Duel runs each benchmark until it's steady
248
+ before it measures it, for up to ten seconds. A small library's methods
249
+ are compiled within Duel's first samples, but in a big app, YJIT
250
+ compiles a method only after 120 calls. YJIT can be on because you
251
+ turned it on (`RUBY_YJIT_ENABLE=1`, for example), or because your code
252
+ did as it loaded, as Rails 7.2 and later do.
253
+
254
+ A worker is steady once the median of its samples over each of three
255
+ half seconds in a row is no more than 1% faster than over the last half
256
+ second that was, so a pause (a collection, say) doesn't end its warm-up,
257
+ and code that creeps faster by less a half second isn't steady. Duel
258
+ measures once every worker is steady, or the time is up. To warm up for
259
+ a set time instead, use `--warm`.
225
260
 
226
261
  ### JRuby and TruffleRuby
227
262
 
228
263
  JRuby's and TruffleRuby's JIT compilers keep compiling for seconds, so
229
- Duel runs each benchmark for five seconds before it measures it
230
- (`--warm`). Their timings stay noisier than CRuby's, so Duel often
231
- leaves small timing regressions on them unsettled.
232
-
233
- Neither has an allocation count, an instruction count, or profiles.
264
+ Duel runs each benchmark until it's steady, for up to a minute, before
265
+ it measures it (`--warm`). Code that a JIT compiles quickly is steady
266
+ in a few seconds, but some code is still several times faster after
267
+ five seconds than it was, and steady only after 10 or 20.
268
+
269
+ Each JVM's JIT also compiles the same code a little differently, so one
270
+ worker can run a benchmark faster or slower than another for its whole
271
+ life, however long it warms up. On JRuby, Duel starts each worker's JVM
272
+ with `-Xbatch`, which compiles in the foreground, so what it compiles is
273
+ less a matter of timing, and with the serial garbage collector, which
274
+ runs no threads of its own beside the code. That makes the workers about
275
+ twice as alike, but a CPU-time interval on JRuby is still some 5% to 8%
276
+ wide, as wide as the workers make it, so Duel leaves a timing regression
277
+ of a few percent unsettled there, where it would find it on CRuby. It
278
+ doesn't fail identical code on that noise: an interval over the pairs
279
+ would be narrower, but it would often leave out 1 for identical code.
280
+ The options come after any in `JRUBY_OPTS`, and Duel doesn't add the
281
+ serial collector where `JRUBY_OPTS` chooses another. The processes a
282
+ process benchmark starts don't get them, as their boot is what's
283
+ measured.
284
+
285
+ Their workers are big JVMs, and their compilers keep other CPUs busy,
286
+ so Duel starts eight workers of each version rather than twelve, and
287
+ measures one benchmark at a time for every four CPUs (`--jobs`), not
288
+ every two. More at once is no faster there, and takes more memory: some
289
+ 500 MB a worker.
290
+
291
+ JRuby counts no objects allocated, but the JVM counts the bytes each
292
+ thread allocates, and Duel judges those instead (`allocated_bytes`):
293
+ more than 32 bytes more an iteration fails, about a small object's, and
294
+ a change that makes objects bigger counts too. Where every worker counts
295
+ the same, Duel finds one more small object an iteration. But the JIT can
296
+ do away with an allocation that never leaves a method (escape analysis),
297
+ and each JVM's JIT does away with different ones, so on some code,
298
+ workers' counts differ by thousands of bytes, and an interval over the
299
+ workers (see How Duel tells a regression from noise) is too wide to find
300
+ a few more. And an allocation the JIT does away with in every worker
301
+ isn't counted at all, as it isn't made. Turning escape analysis off
302
+ would count every allocation, but makes CPU time some three times
303
+ noisier, so Duel leaves it on. The count falls as the JIT compiles the
304
+ code, so it's taken after the warm-up. With fewer than five workers of
305
+ each version, it's each version's fewest in 10 samples, as a sample now
306
+ and then is some 31 bytes an iteration more than the rest. TruffleRuby
307
+ has no allocation count. Neither has an instruction count or profiles.
234
308
  TruffleRuby has peak memory only on Linux. JRuby measures peak memory in
235
309
  the JVM's heap in use, on any operating system, because the JVM takes
236
310
  memory for its heap ahead of time, so resident memory barely moves as