duel 0.1.5 → 0.1.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2528f9b36dd187ea88916fa507ab544b243a58b6c74cfe9c75083c127d9c3bbb
4
- data.tar.gz: ca3838705ef0db2b367c115e96e555fb346173be3aa7a1a1f1740e7a5c895a40
3
+ metadata.gz: 2c2e7b0e6e07fe45ff7a77955035a06be6850e761c9ff2f0b10c0ab4b5130eed
4
+ data.tar.gz: fe086be96d5238e8befcffd0bc7adfb8ef94d0a83790f1ab74a82f881cc3786e
5
5
  SHA512:
6
- metadata.gz: 0663ad25b83ffa57ff1fdedb64c462c8fea600967d26ec49831071bf69fa4ebfbbf2a9a17f8ed1bdfbb3c8bb7a55757cea1fd9a6ad3937bb435ed849dda4ab55
7
- data.tar.gz: d2b0f54723cfe98dbffa56103b5699bc532b9d305c2d75f5304d2749de0f1c455ab12c6250afbf8932537b7a883a6101bf61f242a0e7c5f24f65f866d60e6fb5
6
+ metadata.gz: 731c24cb25017b1f1fba0546523c8c895400eb2750d260e58072c9537afc952fdf2b29bbc90d6242ef5b9463004203782c637cb21c5ea4bf075291866a44a939
7
+ data.tar.gz: 6b98fb68a77b594457ea5b0f9b3bb88fab03c55a606ce7741ae7c970a4cab829000521d62effce598ed64ca362fca1e0b7e1ee50b209fa9b800fb4e6e16e2839
data/CHANGELOG.md CHANGED
@@ -7,6 +7,138 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [0.1.7] - 2026-11-10
11
+
12
+ ### Fixed
13
+
14
+ - Duel measures fewer benchmarks at once, and where even one
15
+ benchmark's workers wouldn't fit, starts fewer workers, so that they
16
+ fit in the memory available. Since 0.1.6 starts twelve workers of
17
+ each version for each benchmark it measures at once, a plugin whose
18
+ workers each start a server could run a machine out of memory: on a
19
+ GitHub runner, duel-mysql's 48 MySQL servers took more than its
20
+ 16 GB, and the runner stopped answering until its job was lost. Duel
21
+ starts one worker of each version first to find what a pair takes
22
+ (with its resources), and allows each pair a quarter more, and 1 GB
23
+ for everything else. It reads what's available from Linux's
24
+ MemAvailable (or a container's cgroup limit, if lower) or macOS's
25
+ `vm_stat`, and its first line says when it lowered either. A
26
+ `--jobs` or `--processes` you give is kept as it is.
27
+ - The badge (in the workflow that `duel init --badge` writes) says
28
+ "not measured", in grey, where the Duel job failed
29
+ before Duel judged the push (its runner lost, say, or a step before
30
+ Duel's failed), instead of "perf regressed". It's red only where
31
+ Duel failed the push or found a regression, and a job that fails
32
+ after Duel passed the push leaves it green.
33
+
34
+ ## [0.1.6] - 2026-11-10
35
+
36
+ ### Added
37
+
38
+ - The comment workflow that `duel init` writes runs the Duel job again
39
+ when the label that accepts a regression (`c.accept_label`) is added
40
+ to a pull request whose run failed, or taken off one whose run
41
+ passed. Before, the label did nothing until someone ran the job again
42
+ by hand. The job runs on `pull_request_target`, checks nothing out,
43
+ and has only `actions: write`.
44
+ - On a schedule, the workflow compares with the last commit since the
45
+ last release that merged a pull request with the label, instead of
46
+ the release, so a slowdown accepted in its pull request doesn't fail
47
+ every nightly run until the next release. Duel reads that commit from
48
+ `DUEL_ACCEPTED`, and the report says what it compared with.
49
+ - The JSON report says, for each timing and across the board, how many
50
+ worker processes of each version the interval is over (`processes`),
51
+ or `null` where it's over pairs or benchmarks.
52
+ - The nightly bias check (`tasks/bias_check.rb`) also fails where more
53
+ of a metric's intervals of identical code leave out 1 than their
54
+ confidence allows, and runs on JRuby too, and on a pull request that
55
+ changes how Duel measures or judges.
56
+
57
+ - On JRuby, Duel counts the bytes each benchmark allocates
58
+ (`allocated_bytes`), as the JVM counts them for each thread (or every
59
+ thread, for a benchmark with threads of its own), and fails one that
60
+ allocates more than 32 bytes more an iteration. Before, JRuby had no
61
+ allocation count. Where each JVM's JIT does away with different
62
+ allocations (escape analysis), workers' counts differ, and Duel finds
63
+ only as many more as they can tell apart.
64
+ - A count is over workers too: each worker's fewest in three rounds of
65
+ a sample from every worker, and it fails when an interval for how many
66
+ more the candidate's workers count than the baseline's lies above the
67
+ tolerance. Before, it was each version's fewest of 3 samples, and on
68
+ JRuby, where one worker's JIT can do away with an allocation another's
69
+ doesn't, one worker's luck could fail identical code. Where every
70
+ worker counts the same, as on CRuby, it's as exact as before. With
71
+ fewer than 5 workers of each version, it's the fewest of 3 samples (10
72
+ on JRuby), as before.
73
+ - On JRuby and TruffleRuby, a worker may take 5 minutes to load or to
74
+ answer (`--timeout`), not 1: with `-Xbatch`, 16 JVMs starting at once
75
+ could take longer than a minute.
76
+ - Counts of 9999.5 or more an iteration show as whole numbers (25736, not
77
+ 2.574e+04).
78
+
79
+ ### Changed
80
+
81
+ - A timing's interval is over worker processes, not pairs of samples.
82
+ Where things land in a process's memory (or on JRuby, what its JIT
83
+ compiles) can make the same code faster or slower for the process's
84
+ whole life, so the pairs of 3 workers of each version were far from
85
+ independent, and with identical code on both sides, their intervals
86
+ left out 1 several times as often as 99% intervals should. Each
87
+ worker's median is now a sample, and the interval is the Mann-Whitney
88
+ interval for the ratio of the candidate's workers to the baseline's.
89
+ Duel takes rounds of a sample from every worker, up to some 120
90
+ pairs, and starts 12 workers of each version by default (8 on JRuby
91
+ and TruffleRuby). CPU-time intervals are about as wide as before, or
92
+ narrower, and instruction counts' (which a process repeats almost
93
+ exactly) a little wider. On JRuby, they're some 5% to 8% wide, as
94
+ wide as each JVM's JIT makes them, so Duel leaves smaller timing
95
+ regressions there unsettled. With fewer than 5 workers of each version
96
+ (`--processes`, and `Duel::Testing.compare`'s 2), and for a process
97
+ benchmark, the interval is over the pairs, as before, and the first
98
+ line of the output says which.
99
+ - Across the board, the interval is over the workers too: each worker's
100
+ score is how much faster or slower it was than its lane's other
101
+ workers, at the median of the benchmarks it measured. A worker whose
102
+ luck slowed it at every benchmark could fail identical code across
103
+ the board.
104
+ - Without `--warm`, each worker warms a benchmark up until it's steady
105
+ (the median of its samples over each of three half seconds in a row
106
+ is no more than 1% faster than over the last half second that was),
107
+ for up to a minute on JRuby and TruffleRuby, or 10 seconds with YJIT
108
+ on. Before, it warmed up for 5 seconds, or 2, every time, and on
109
+ JRuby, some code was still several times faster after that, so it
110
+ was measured while it got faster. `--warm` still warms up for the time
111
+ it says.
112
+ - On JRuby, each worker's JVM compiles in the foreground (`-Xbatch`) and
113
+ collects garbage with the serial collector, after any options in
114
+ `JRUBY_OPTS` (and without the serial collector where they choose
115
+ another). Each JVM's JIT compiles the same code a little differently,
116
+ and that made JRuby's intervals over workers twice as wide.
117
+ - A run measures for two hours at most by default (`--time-limit`), not
118
+ half an hour, as on JRuby a benchmark can take half a minute to warm
119
+ up.
120
+ - On JRuby and TruffleRuby, Duel measures one benchmark at a time for
121
+ every four CPUs (`--jobs`), not every two: their JIT compilers keep the
122
+ other CPUs busy, so more were no faster, and each worker is a JVM of
123
+ some 500 MB.
124
+
125
+ ### Fixed
126
+
127
+ - The comment workflow fences a fork's report with more backticks than
128
+ any run of them in it even when the report has a NUL byte. GNU grep
129
+ took such a report for binary and found no backticks, so the fence
130
+ was three, and a line of three backticks in the report closed it. The
131
+ workflow now drops NUL bytes, which GitHub doesn't take in a comment,
132
+ and reads the report as text whatever's in it.
133
+ - The Mark the commit passed job runs one at a time for each branch
134
+ (and on GitLab CI, one at a time). Before the branch that marks the
135
+ last commit Duel passed existed, two pushes that passed at once could
136
+ both try to make it, and the second failed, saying another branch
137
+ kept it from being made. Where making the branch fails, the job now
138
+ also looks for it again before it says so. Update the job in
139
+ `.github/workflows/perf.yml` (or `.gitlab/duel.gitlab-ci.yml`) to pick
140
+ this up.
141
+
10
142
  ## [0.1.5] - 2026-11-08
11
143
 
12
144
  ### Changed
@@ -574,7 +706,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
574
706
  Anything marked `@api private`, in lib/ and sig/, may change in any
575
707
  release.
576
708
 
577
- [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.5...HEAD
709
+ [Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.7...HEAD
710
+ [0.1.7]: https://github.com/sferik/duel-ruby/compare/v0.1.6...v0.1.7
711
+ [0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
578
712
  [0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
579
713
  [0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
580
714
  [0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
data/README.md CHANGED
@@ -14,7 +14,7 @@ code from becoming slower.
14
14
 
15
15
  ```sh
16
16
  $ duel
17
- Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02)
17
+ Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02, over 12 worker processes of each version)
18
18
  parse: a small document 0.6s n=412 cpu 0.998 [0.991, 1.006] same allocations 20
19
19
  ! parse: a large document 1.1s n=9 cpu 1.081 [1.062, 1.097] slower allocations 5790
20
20
  ...
@@ -193,11 +193,11 @@ and how it compares a project in a subdirectory of its repository, see
193
193
  ### Timeouts and exit status
194
194
 
195
195
  A benchmark that hangs, or that waits on a server that died, doesn't
196
- hang the run. If a worker takes more than a minute to load or to answer,
197
- Duel stops the worker and reports the benchmark as an error. To change
198
- the limit, use `--timeout`.
196
+ hang the run. If a worker takes more than a minute (five on JRuby and
197
+ TruffleRuby) to load or to answer, Duel stops the worker and reports the
198
+ benchmark as an error. To change the limit, use `--timeout`.
199
199
 
200
- A run measures for 30 minutes at most, not counting the builds. Past
200
+ A run measures for two hours at most, not counting the builds. Past
201
201
  that, Duel starts no more measurements. A benchmark that it hasn't
202
202
  measured yet fails as an error, and a benchmark that looked worse, but
203
203
  that Duel hasn't measured again as often as it usually does, fails if it
data/docs/ci.md CHANGED
@@ -61,23 +61,33 @@ jobs:
61
61
  with:
62
62
  fetch-depth: 0 # the baseline comes from history, tags included
63
63
  persist-credentials: false
64
- # (And whatever else your tests need set up: packages, services.)
65
- - uses: ruby/setup-ruby@v1
66
- with:
67
- ruby-version: "3.4" # (or none, with a .ruby-version)
68
- bundler-cache: true
69
64
  # The labels of the pull request (or for a push, of the one that
70
65
  # merged it), for c.accept_label in .duel.rb, as a JSON array. They're
71
- # read as the job runs, so after adding the label, run it again.
66
+ # read as the job runs, and adding the label (or taking it off) runs
67
+ # the job again: see the Perf comment workflow.
68
+ # On a schedule, the last commit since the last release that merged a
69
+ # pull request with the label (ACCEPT_LABEL, as .duel.rb names it):
70
+ # what that slowed down was accepted, so it's compared with instead.
72
71
  - name: Find the pull request's labels
73
72
  id: labels
74
- if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push'
73
+ if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push' || github.event_name == 'schedule'
75
74
  continue-on-error: true # (without them, nothing's accepted)
76
75
  env:
77
76
  GH_TOKEN: ${{ github.token }}
78
77
  PR: ${{ github.event.pull_request.number }}
79
78
  QUEUED: ${{ github.event.merge_group.head_ref }}
79
+ ACCEPT_LABEL: 'duel: accept'
80
+ RELEASE_TAGS: "v*"
80
81
  run: |
82
+ if [ "$GITHUB_EVENT_NAME" = schedule ]; then
83
+ tag=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || exit 0
84
+ since=$(TZ=UTC git log -1 --date=format-local:%Y-%m-%dT%H:%M:%SZ --format=%cd "$tag")
85
+ merged=$(gh pr list --repo "$GITHUB_REPOSITORY" --state merged --label "$ACCEPT_LABEL" --search "merged:>=$since" --limit 1000 \
86
+ --json mergeCommit --jq '.[].mergeCommit.oid')
87
+ accepted=$(git rev-list "$tag..HEAD" | grep -Fx -m 1 -f <(echo "$merged")) || accepted=
88
+ echo "accepted=$accepted" >> "$GITHUB_OUTPUT"
89
+ exit 0
90
+ fi
81
91
  # (In a merge queue, the last pull request queued, which the queue's
82
92
  # branch names: gh-readonly-queue/main/pr-123-<sha>.)
83
93
  if [ -n "$QUEUED" ]; then
@@ -90,6 +100,11 @@ jobs:
90
100
  labels=$(gh api "repos/$GITHUB_REPOSITORY/commits/$GITHUB_SHA/pulls" --jq '[.[0].labels[]?.name]')
91
101
  fi
92
102
  echo "labels=$labels" >> "$GITHUB_OUTPUT"
103
+ # (And whatever else your tests need set up: packages, services.)
104
+ - uses: ruby/setup-ruby@v1
105
+ with:
106
+ ruby-version: "3.4" # (or none, with a .ruby-version)
107
+ bundler-cache: true
93
108
  # Against the branch it targets for a pull request, the queue's base
94
109
  # for a merge queue, what was there before for a push (or as
95
110
  # c.push_base in .duel.rb says, the last commit Duel passed), and the
@@ -105,6 +120,7 @@ jobs:
105
120
  # with itself, which checks the setup.)
106
121
  DUEL_FORCE: ${{ github.event_name == 'workflow_dispatch' && '1' || '' }}
107
122
  DUEL_LABELS: ${{ steps.labels.outputs.labels }}
123
+ DUEL_ACCEPTED: ${{ steps.labels.outputs.accepted }}
108
124
  run: bundle exec duel --markdown "$RUNNER_TEMP/duel/duel.md"
109
125
  # For a pull request, what Duel said, for the Perf comment workflow,
110
126
  # which can't see this job's outputs.
@@ -149,8 +165,11 @@ The workflow compares the following:
149
165
  the last commit that Duel passed. If that commit isn't in the history,
150
166
  such as after a force push or for a branch's first push, Duel reports
151
167
  that there's nothing to compare against, and passes.
152
- - On a schedule, the last release. Until you tag your first release,
153
- Duel reports that there's nothing to compare against, and passes.
168
+ - On a schedule, the last release, or with
169
+ [`c.accept_label`](#accept-a-regression-with-a-label), the last commit
170
+ since then that merged a pull request with the label. Until you tag
171
+ your first release, Duel reports that there's nothing to compare
172
+ against, and passes.
154
173
  - When you run the workflow manually, your code against itself. This
155
174
  checks that the benchmarks run on the CI machines.
156
175
 
@@ -237,12 +256,23 @@ name: Perf comment
237
256
  # ran), it trusts only what that job could have said: the pull request's
238
257
  # number, if it names the pull request the run compared, and the comment
239
258
  # and the status, if they're ones Duel says. A fork's report, which the
240
- # fork's code wrote, is shown as text. (The workflow must be on the
241
- # default branch before it runs at all.)
259
+ # fork's code wrote, is shown as text.
260
+ #
261
+ # And when the label that accepts a regression (c.accept_label in
262
+ # .duel.rb) is added to a pull request, or taken off, it runs the Duel
263
+ # job again, which reads the labels as it runs: a run that failed, once
264
+ # the label's added, or one that passed, once it's taken off.
265
+ #
266
+ # (The workflow must be on the default branch before it runs at all.)
242
267
  on:
243
268
  workflow_run:
244
269
  workflows: ["Perf"]
245
270
  types: [completed]
271
+ # (This repository's own workflow, as for the comment, with a token
272
+ # that can run a workflow again: it checks nothing out, and runs none
273
+ # of the pull request's code.)
274
+ pull_request_target:
275
+ types: [labeled, unlabeled]
246
276
 
247
277
  permissions: {}
248
278
 
@@ -283,12 +313,16 @@ jobs:
283
313
  # ran the fork's code, which could have written anything in it. In
284
314
  # a code block, fenced with more backticks than any run of them in
285
315
  # the report, it can't mention anyone, link anywhere or show an
286
- # image. (And it's cut short where a comment couldn't hold it.)
316
+ # image. It's cut short where a comment couldn't hold it, and loses
317
+ # any NUL bytes, which GitHub wouldn't take, and grep reads it as
318
+ # text whatever's in it, so it finds every run of backticks.
287
319
  report() {
288
320
  if [ "${head#* }" = "$GITHUB_REPOSITORY" ]; then cat "$RUNNER_TEMP/report/duel.md" && return; fi
289
- fence=$(grep -o '``*' "$RUNNER_TEMP/report/duel.md" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
321
+ text="$RUNNER_TEMP/report.txt"
322
+ LC_ALL=C tr -d '\000' < "$RUNNER_TEMP/report/duel.md" | head -c 60000 > "$text"
323
+ fence=$(LC_ALL=C grep -ao '``*' "$text" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
290
324
  printf 'From a fork, so shown as text:\n\n%s\n' "$fence"
291
- head -c 60000 "$RUNNER_TEMP/report/duel.md"
325
+ cat "$text"
292
326
  printf '\n%s\n' "$fence"
293
327
  }
294
328
  marker="<!-- duel"
@@ -300,6 +334,33 @@ jobs:
300
334
  { echo "$marker status=$STATUS -->"; report; } > "$RUNNER_TEMP/comment.md"
301
335
  gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
302
336
  fi
337
+
338
+ again:
339
+ name: Run Duel again
340
+ if: >-
341
+ github.event_name == 'pull_request_target' && github.event.label.name == 'duel: accept'
342
+ runs-on: ubuntu-latest
343
+ permissions:
344
+ actions: write # to run the Perf workflow's jobs again
345
+ steps:
346
+ - name: Run Duel again
347
+ env:
348
+ GH_TOKEN: ${{ github.token }}
349
+ GH_REPO: ${{ github.repository }}
350
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
351
+ ACTION: ${{ github.event.action }}
352
+ run: |
353
+ # The newest Perf run of the pull request's head commit, once it's
354
+ # done (one that's running may have read the labels already).
355
+ id=$(gh run list --workflow "Perf" --event pull_request --commit "$HEAD_SHA" --limit 1 \
356
+ --json databaseId --jq '.[0].databaseId // empty')
357
+ [ -n "$id" ] || exit 0
358
+ gh run watch "$id" --interval 30 > /dev/null || true
359
+ conclusion=$(gh run view "$id" --json conclusion --jq .conclusion)
360
+ case "$ACTION:$conclusion" in
361
+ labeled:failure) gh run rerun "$id" --failed ;;
362
+ unlabeled:success) gh run rerun "$id" ;;
363
+ esac
303
364
  ```
304
365
 
305
366
  ## Comment on a pushed commit
@@ -374,6 +435,23 @@ or for a push, of the pull request that merged it, and passes them to
374
435
  `pull-requests: read` permission. Without the labels, such as when
375
436
  GitHub's API fails, nothing is accepted.
376
437
 
438
+ The labels are read as the Duel job runs, so `perf-comment.yml` runs
439
+ the job again when the label changes. Its Run Duel again job runs on
440
+ `pull_request_target`, with this repository's own workflow, when a
441
+ label is added to a pull request or taken off. For the label `.duel.rb`
442
+ names, it waits for the Perf run of the pull request's head commit to
443
+ finish. Then it runs the failed jobs again where the label was added
444
+ to a run that failed, or the whole run again where the label was
445
+ taken off one that passed. It checks nothing out and runs none of the
446
+ pull request's code, and its only permission is `actions: write`. The
447
+ workflows name the label as `.duel.rb` did when `duel init` wrote them,
448
+ so if you rename it, rename it in both workflows too.
449
+
450
+ On a schedule, the same step looks for the pull requests merged since
451
+ the last release with the label, and passes `duel` the last commit
452
+ that merged one, in `DUEL_ACCEPTED`. Duel compares with that commit
453
+ instead of the release, as what it slowed down was accepted.
454
+
377
455
  ## Compare a push with the last commit that passed
378
456
 
379
457
  By default, the workflow compares each push with the commit before it.
@@ -390,8 +468,10 @@ branch that the workflow guards has its own (see
390
468
  nothing to compare, Duel sets the `passed` output to `true`, and the
391
469
  Mark the commit passed job moves the branch to the push, or creates the
392
470
  branch. It never moves the branch back: when two pushes run at once and
393
- the older one finishes last, the branch stays at the newer push. After
394
- a force push rewrites the branch's history, it moves the branch anyway.
471
+ the older one finishes last, the branch stays at the newer push. The
472
+ job runs one at a time for each branch, so two runs can't race to
473
+ create the branch. After a force push rewrites the branch's history,
474
+ it moves the branch anyway.
395
475
  That job checks nothing out either, and it gets the `contents: write`
396
476
  permission that moving a branch takes. Until the branch exists, and
397
477
  when the push's history doesn't have its commit, Duel compares the push
@@ -438,6 +518,13 @@ yourself:
438
518
  passed:
439
519
  name: Mark the commit passed
440
520
  needs: duel
521
+ # One at a time for each branch, so that two runs can't both find no
522
+ # branch and race to make it (a push's run doesn't wait for the one
523
+ # before it). One that's running is never cancelled, and one that's
524
+ # waiting gives way to a newer push's, which moves it further anyway.
525
+ concurrency:
526
+ group: ${{ github.workflow }}-passed-${{ github.ref_name }}
527
+ cancel-in-progress: false
441
528
  if: ${{ !cancelled() && github.event_name == 'push' && needs.duel.outputs.passed == 'true' }}
442
529
  runs-on: ubuntu-latest
443
530
  permissions:
@@ -461,13 +548,16 @@ yourself:
461
548
  done
462
549
  exit 1
463
550
  }
464
- # (Forward: GitHub moves it only where it's behind the push.)
465
- gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
466
- at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
467
- if [ -z "$at" ]; then
468
- gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null || blocked
469
- exit 0
470
- fi
551
+ # (Forward: GitHub moves it only where it's behind the push. Where
552
+ # there's no branch, and making it fails, it looks once more, as
553
+ # something else may have made it: a push of it, say.)
554
+ for _ in 1 2; do
555
+ gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
556
+ at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
557
+ [ -n "$at" ] && break
558
+ gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null && exit 0
559
+ done
560
+ [ -n "$at" ] || blocked
471
561
  compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status 2>&1) || case "$compared" in
472
562
  # (GitHub can't compare commits with no history in common.)
473
563
  *"No common ancestor"*) compared=unrelated ;;
@@ -494,16 +584,19 @@ your README can show it:
494
584
  ```
495
585
 
496
586
  The badge says "perf guarded", in green, when Duel passed the push. It
497
- says "perf regressed", in red, when the Duel job failed or found a
587
+ says "perf regressed", in red, when Duel failed the push or found a
498
588
  regression, even one that passed, with
499
589
  [`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
500
- label accepted. A push with nothing to compare leaves the badge as it
501
- was, and so does a run that finishes after a newer push's, so the
502
- badge is always the newest push's that Duel compared. The branch holds
503
- one commit, with the badge alone, whose message names that push. The
504
- job checks nothing out, and it gets the `contents: write` permission
505
- that moving a branch takes. For a project in a subdirectory of its
506
- repository, the branch names the directory, such as
590
+ label accepted. It says "not measured", in grey, when the Duel job
591
+ failed before Duel could judge the push, as when its runner ran out of
592
+ memory or a step before Duel's failed. A job that fails after Duel
593
+ passed the push leaves it green. A push with nothing to compare leaves
594
+ the badge as it was, and so does a run that finishes after a newer
595
+ push's, so the badge is always the newest push's that Duel compared.
596
+ The branch holds one commit, with the badge alone, whose message names
597
+ that push. The job checks nothing out, and it gets the `contents: write`
598
+ permission that moving a branch takes. For a project in a subdirectory
599
+ of its repository, the branch names the directory, such as
507
600
  `duel/badge-gems-parser`.
508
601
 
509
602
  Until the first push to your default branch after you add the job, the
@@ -517,13 +610,15 @@ keeps `.duel.rb`), or add the job yourself:
517
610
 
518
611
  ```yaml
519
612
  # .github/workflows/perf.yml, in jobs: (with duel init --badge)
520
- # Duel's badge, for the README: green (perf guarded) or red (perf
521
- # regressed), as the last push to the default branch that Duel compared
522
- # left it. It's red where the Duel job failed or found a regression,
523
- # even one that passed (with c.fail_on = :pull_requests in .duel.rb). A
524
- # push with nothing to compare leaves it as it was, and so does a run
525
- # for an older push than the badge's. This job writes the badge, alone,
526
- # to a branch of its own, which the README shows:
613
+ # Duel's badge, for the README: green (perf guarded), red (perf
614
+ # regressed) or grey (not measured), as the last push to the default
615
+ # branch that Duel compared left it. It's red where Duel failed the push
616
+ # or found a regression, even one that passed (with
617
+ # c.fail_on = :pull_requests in .duel.rb), and grey where the Duel job
618
+ # failed before Duel could judge the push (its runner ran out of memory,
619
+ # say). A push with nothing to compare leaves it as it was, and so does
620
+ # a run for an older push than the badge's. This job writes the badge,
621
+ # alone, to a branch of its own, which the README shows:
527
622
  #
528
623
  # [![Duel](https://raw.githubusercontent.com/OWNER/REPO/duel/badge/badge.svg)](https://github.com/sferik/duel-ruby)
529
624
  #
@@ -569,10 +664,13 @@ keeps `.duel.rb`), or add the job yourself:
569
664
  last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
570
665
  if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
571
666
  # (Its image, written here rather than fetched from anywhere: green,
572
- # or red where the Duel job failed or found a regression.)
667
+ # red where Duel failed the push or found a regression, or grey
668
+ # where the Duel job failed before Duel said either.)
573
669
  state="perf guarded" width=133 top="#34D058" bottom="#28A745" right="76.939C78.629 0 80 1.343 80"
574
- if [ "$RESULT" = failure ] || [ "$STATUS" = failed ]; then
670
+ if [ "$STATUS" = failed ]; then
575
671
  state="perf regressed" width=142 top="#EA4A5A" bottom="#D73A49" right="85.939C87.629 0 89 1.343 89"
672
+ elif [ "$RESULT" = failure ] && [ "$STATUS" != passed ]; then
673
+ state="not measured" width=138 top="#959DA5" bottom="#6A737D" right="81.939C83.629 0 85 1.343 85"
576
674
  fi
577
675
  cat > "$RUNNER_TEMP/badge.svg" <<SVG
578
676
  <svg xmlns="http://www.w3.org/2000/svg" width="$width" height="20">
@@ -825,6 +923,9 @@ duel:passed:
825
923
  image: alpine:3
826
924
  stage: .post
827
925
  needs: [duel]
926
+ # One at a time, as each deletes the branch and makes it again: two at
927
+ # once could both delete it, and the second to make it would fail.
928
+ resource_group: duel/passed
828
929
  rules:
829
930
  - if: $DUEL_TOKEN && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
830
931
  variables:
@@ -894,7 +995,8 @@ match. `duel init` adds these steps for a project that has a
894
995
  # request targets (or run by hand, the main branch), a merge queue's
895
996
  # base, the commit before a push (or the last commit Duel passed, PASSED, if there is
896
997
  # one: see the Mark the commit passed job), or on a schedule the last
897
- # release, the newest tag before HEAD that RELEASE_TAGS matches. (If
998
+ # release, the newest tag before HEAD that RELEASE_TAGS matches
999
+ # (or ACCEPTED, the last change accepted since then). (If
898
1000
  # .duel.rb sets c.main_branch or c.release_tags, set MAIN_BRANCH or
899
1001
  # RELEASE_TAGS to match.)
900
1002
  - name: Find the baseline's Ruby
@@ -905,13 +1007,14 @@ match. `duel init` adds these steps for a project that has a
905
1007
  PASSED: origin/duel/passed/${{ github.ref_name }}
906
1008
  MAIN_BRANCH: origin/${{ github.base_ref || github.event.repository.default_branch }}
907
1009
  RELEASE_TAGS: "v*"
1010
+ ACCEPTED: ${{ steps.labels.outputs.accepted }}
908
1011
  run: |
909
1012
  case "$GITHUB_EVENT_NAME" in
910
1013
  push)
911
1014
  base=$BEFORE
912
1015
  if git merge-base --is-ancestor "$PASSED" HEAD 2>/dev/null; then base=$PASSED; fi ;;
913
1016
  merge_group) base=$QUEUE_BASE ;;
914
- schedule) base=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || base= ;;
1017
+ schedule) base=${ACCEPTED:-$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null)} || base= ;;
915
1018
  *) base=$(git merge-base HEAD "$MAIN_BRANCH" 2>/dev/null) || base= ;;
916
1019
  esac
917
1020
  ruby_of() { git show "$1:./.ruby-version" 2>/dev/null | awk 'NF { print $1; exit }'; }
@@ -140,12 +140,16 @@ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
140
140
  that `duel init` writes passes Duel the pull request's labels. With
141
141
  that label, a regression passes, with a warning, and the report says
142
142
  that the label accepted it. The labels are read when the workflow runs,
143
- so after you add the label, run the workflow again (Re-run jobs, on the
144
- pull request's Checks tab). For a push, Duel reads the labels of the
145
- pull request that merged it, so with
143
+ so the comment workflow runs the Duel job again when you add the label
144
+ to a pull request whose run failed, or take it off one whose run
145
+ passed. For a push, Duel reads the labels of the pull request that
146
+ merged it, so with
146
147
  [`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
147
148
  the merged regression becomes the commit that later pushes are compared
148
- with. Duel's default is no such label, but the `.duel.rb` that
149
+ with. On a schedule, Duel compares with the last commit since your last
150
+ release that merged a pull request with the label, instead of the
151
+ release, so a slowdown you accepted doesn't fail every night until
152
+ your next release. Duel's default is no such label, but the `.duel.rb` that
149
153
  `duel init` writes (and duel-rails's generator) names `duel: accept`.
150
154
  Create the label in your repository to use it, or set
151
155
  `c.accept_label = nil` so that nothing can let a regression through.