duel 0.1.5 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +110 -1
- data/README.md +5 -5
- data/docs/ci.md +120 -25
- data/docs/configuration.md +8 -4
- data/docs/how-it-works.md +109 -35
- data/docs/json.md +16 -10
- data/docs/plugins.md +3 -1
- data/docs/troubleshooting.md +6 -6
- data/lib/duel/across.rb +100 -9
- data/lib/duel/baseline.rb +26 -2
- data/lib/duel/cli.rb +18 -7
- data/lib/duel/config.rb +4 -1
- data/lib/duel/init.rb +19 -3
- data/lib/duel/json_report.rb +2 -2
- data/lib/duel/metric.rb +32 -1
- data/lib/duel/rubies.rb +39 -0
- data/lib/duel/runner.rb +337 -49
- data/lib/duel/stats.rb +76 -0
- data/lib/duel/summary.rb +2 -2
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +3 -0
- data/lib/duel/templates/perf-comment.yml.erb +53 -5
- data/lib/duel/templates/perf.yml.erb +71 -33
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +5 -4
- data/sig/duel/across.rbs +24 -2
- data/sig/duel/baseline.rbs +12 -0
- data/sig/duel/cli.rbs +4 -0
- data/sig/duel/init.rbs +6 -0
- data/sig/duel/metric.rbs +19 -1
- data/sig/duel/rubies.rbs +13 -0
- data/sig/duel/runner.rbs +112 -6
- data/sig/duel/stats.rbs +25 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a3bd63fc0a6b4eb33d7fe48f733446e7b35c3e961dfdc42e85531fdc680ca53c
|
|
4
|
+
data.tar.gz: d44709c7629208381a9e24235f2401dc53c69c11fd540c8b8ab5c4e02710bf7a
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 44fd8dbe608296d4af112bd27525f38d544662e858e39abfc4913583fa1a0d5d74cd9c20c3745dcc226ef1c6f89c6deb9f2f6780bb457627da18e0b7bf82373e
|
|
7
|
+
data.tar.gz: 6984d61ce88c4042e8447a9ec03886bb68fa39d4e2d5bdef448a6b4dd1cd7ef6d258a940aac56b3bd08892907fc4d626cd4f91dcbbcbca634b29c78227c7f860
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,114 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.6] - 2026-11-10
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- The comment workflow that `duel init` writes runs the Duel job again
|
|
15
|
+
when the label that accepts a regression (`c.accept_label`) is added
|
|
16
|
+
to a pull request whose run failed, or taken off one whose run
|
|
17
|
+
passed. Before, the label did nothing until someone ran the job again
|
|
18
|
+
by hand. The job runs on `pull_request_target`, checks nothing out,
|
|
19
|
+
and has only `actions: write`.
|
|
20
|
+
- On a schedule, the workflow compares with the last commit since the
|
|
21
|
+
last release that merged a pull request with the label, instead of
|
|
22
|
+
the release, so a slowdown accepted in its pull request doesn't fail
|
|
23
|
+
every nightly run until the next release. Duel reads that commit from
|
|
24
|
+
`DUEL_ACCEPTED`, and the report says what it compared with.
|
|
25
|
+
- The JSON report says, for each timing and across the board, how many
|
|
26
|
+
worker processes of each version the interval is over (`processes`),
|
|
27
|
+
or `null` where it's over pairs or benchmarks.
|
|
28
|
+
- The nightly bias check (`tasks/bias_check.rb`) also fails where more
|
|
29
|
+
of a metric's intervals of identical code leave out 1 than their
|
|
30
|
+
confidence allows, and runs on JRuby too, and on a pull request that
|
|
31
|
+
changes how Duel measures or judges.
|
|
32
|
+
|
|
33
|
+
- On JRuby, Duel counts the bytes each benchmark allocates
|
|
34
|
+
(`allocated_bytes`), as the JVM counts them for each thread (or every
|
|
35
|
+
thread, for a benchmark with threads of its own), and fails one that
|
|
36
|
+
allocates more than 32 bytes more an iteration. Before, JRuby had no
|
|
37
|
+
allocation count. Where each JVM's JIT does away with different
|
|
38
|
+
allocations (escape analysis), workers' counts differ, and Duel finds
|
|
39
|
+
only as many more as they can tell apart.
|
|
40
|
+
- A count is over workers too: each worker's fewest in three rounds of
|
|
41
|
+
a sample from every worker, and it fails when an interval for how many
|
|
42
|
+
more the candidate's workers count than the baseline's lies above the
|
|
43
|
+
tolerance. Before, it was each version's fewest of 3 samples, and on
|
|
44
|
+
JRuby, where one worker's JIT can do away with an allocation another's
|
|
45
|
+
doesn't, one worker's luck could fail identical code. Where every
|
|
46
|
+
worker counts the same, as on CRuby, it's as exact as before. With
|
|
47
|
+
fewer than 5 workers of each version, it's the fewest of 3 samples (10
|
|
48
|
+
on JRuby), as before.
|
|
49
|
+
- On JRuby and TruffleRuby, a worker may take 5 minutes to load or to
|
|
50
|
+
answer (`--timeout`), not 1: with `-Xbatch`, 16 JVMs starting at once
|
|
51
|
+
could take longer than a minute.
|
|
52
|
+
- Counts of 9999.5 or more an iteration show as whole numbers (25736, not
|
|
53
|
+
2.574e+04).
|
|
54
|
+
|
|
55
|
+
### Changed
|
|
56
|
+
|
|
57
|
+
- A timing's interval is over worker processes, not pairs of samples.
|
|
58
|
+
Where things land in a process's memory (or on JRuby, what its JIT
|
|
59
|
+
compiles) can make the same code faster or slower for the process's
|
|
60
|
+
whole life, so the pairs of 3 workers of each version were far from
|
|
61
|
+
independent, and with identical code on both sides, their intervals
|
|
62
|
+
left out 1 several times as often as 99% intervals should. Each
|
|
63
|
+
worker's median is now a sample, and the interval is the Mann-Whitney
|
|
64
|
+
interval for the ratio of the candidate's workers to the baseline's.
|
|
65
|
+
Duel takes rounds of a sample from every worker, up to some 120
|
|
66
|
+
pairs, and starts 12 workers of each version by default (8 on JRuby
|
|
67
|
+
and TruffleRuby). CPU-time intervals are about as wide as before, or
|
|
68
|
+
narrower, and instruction counts' (which a process repeats almost
|
|
69
|
+
exactly) a little wider. On JRuby, they're some 5% to 8% wide, as
|
|
70
|
+
wide as each JVM's JIT makes them, so Duel leaves smaller timing
|
|
71
|
+
regressions there unsettled. With fewer than 5 workers of each version
|
|
72
|
+
(`--processes`, and `Duel::Testing.compare`'s 2), and for a process
|
|
73
|
+
benchmark, the interval is over the pairs, as before, and the first
|
|
74
|
+
line of the output says which.
|
|
75
|
+
- Across the board, the interval is over the workers too: each worker's
|
|
76
|
+
score is how much faster or slower it was than its lane's other
|
|
77
|
+
workers, at the median of the benchmarks it measured. A worker whose
|
|
78
|
+
luck slowed it at every benchmark could fail identical code across
|
|
79
|
+
the board.
|
|
80
|
+
- Without `--warm`, each worker warms a benchmark up until it's steady
|
|
81
|
+
(the median of its samples over each of three half seconds in a row
|
|
82
|
+
is no more than 1% faster than over the last half second that was),
|
|
83
|
+
for up to a minute on JRuby and TruffleRuby, or 10 seconds with YJIT
|
|
84
|
+
on. Before, it warmed up for 5 seconds, or 2, every time, and on
|
|
85
|
+
JRuby, some code was still several times faster after that, so it
|
|
86
|
+
was measured while it got faster. `--warm` still warms up for the time
|
|
87
|
+
it says.
|
|
88
|
+
- On JRuby, each worker's JVM compiles in the foreground (`-Xbatch`) and
|
|
89
|
+
collects garbage with the serial collector, after any options in
|
|
90
|
+
`JRUBY_OPTS` (and without the serial collector where they choose
|
|
91
|
+
another). Each JVM's JIT compiles the same code a little differently,
|
|
92
|
+
and that made JRuby's intervals over workers twice as wide.
|
|
93
|
+
- A run measures for two hours at most by default (`--time-limit`), not
|
|
94
|
+
half an hour, as on JRuby a benchmark can take half a minute to warm
|
|
95
|
+
up.
|
|
96
|
+
- On JRuby and TruffleRuby, Duel measures one benchmark at a time for
|
|
97
|
+
every four CPUs (`--jobs`), not every two: their JIT compilers keep the
|
|
98
|
+
other CPUs busy, so more were no faster, and each worker is a JVM of
|
|
99
|
+
some 500 MB.
|
|
100
|
+
|
|
101
|
+
### Fixed
|
|
102
|
+
|
|
103
|
+
- The comment workflow fences a fork's report with more backticks than
|
|
104
|
+
any run of them in it even when the report has a NUL byte. GNU grep
|
|
105
|
+
took such a report for binary and found no backticks, so the fence
|
|
106
|
+
was three, and a line of three backticks in the report closed it. The
|
|
107
|
+
workflow now drops NUL bytes, which GitHub doesn't take in a comment,
|
|
108
|
+
and reads the report as text whatever's in it.
|
|
109
|
+
- The Mark the commit passed job runs one at a time for each branch
|
|
110
|
+
(and on GitLab CI, one at a time). Before the branch that marks the
|
|
111
|
+
last commit Duel passed existed, two pushes that passed at once could
|
|
112
|
+
both try to make it, and the second failed, saying another branch
|
|
113
|
+
kept it from being made. Where making the branch fails, the job now
|
|
114
|
+
also looks for it again before it says so. Update the job in
|
|
115
|
+
`.github/workflows/perf.yml` (or `.gitlab/duel.gitlab-ci.yml`) to pick
|
|
116
|
+
this up.
|
|
117
|
+
|
|
10
118
|
## [0.1.5] - 2026-11-08
|
|
11
119
|
|
|
12
120
|
### Changed
|
|
@@ -574,7 +682,8 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
574
682
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
575
683
|
release.
|
|
576
684
|
|
|
577
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.
|
|
685
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.6...HEAD
|
|
686
|
+
[0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
|
|
578
687
|
[0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
|
|
579
688
|
[0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
|
|
580
689
|
[0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
|
data/README.md
CHANGED
|
@@ -14,7 +14,7 @@ code from becoming slower.
|
|
|
14
14
|
|
|
15
15
|
```sh
|
|
16
16
|
$ duel
|
|
17
|
-
Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02)
|
|
17
|
+
Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02, over 12 worker processes of each version)
|
|
18
18
|
parse: a small document 0.6s n=412 cpu 0.998 [0.991, 1.006] same allocations 20
|
|
19
19
|
! parse: a large document 1.1s n=9 cpu 1.081 [1.062, 1.097] slower allocations 5790
|
|
20
20
|
...
|
|
@@ -193,11 +193,11 @@ and how it compares a project in a subdirectory of its repository, see
|
|
|
193
193
|
### Timeouts and exit status
|
|
194
194
|
|
|
195
195
|
A benchmark that hangs, or that waits on a server that died, doesn't
|
|
196
|
-
hang the run. If a worker takes more than a minute
|
|
197
|
-
Duel stops the worker and reports the
|
|
198
|
-
the limit, use `--timeout`.
|
|
196
|
+
hang the run. If a worker takes more than a minute (five on JRuby and
|
|
197
|
+
TruffleRuby) to load or to answer, Duel stops the worker and reports the
|
|
198
|
+
benchmark as an error. To change the limit, use `--timeout`.
|
|
199
199
|
|
|
200
|
-
A run measures for
|
|
200
|
+
A run measures for two hours at most, not counting the builds. Past
|
|
201
201
|
that, Duel starts no more measurements. A benchmark that it hasn't
|
|
202
202
|
measured yet fails as an error, and a benchmark that looked worse, but
|
|
203
203
|
that Duel hasn't measured again as often as it usually does, fails if it
|
data/docs/ci.md
CHANGED
|
@@ -61,23 +61,33 @@ jobs:
|
|
|
61
61
|
with:
|
|
62
62
|
fetch-depth: 0 # the baseline comes from history, tags included
|
|
63
63
|
persist-credentials: false
|
|
64
|
-
# (And whatever else your tests need set up: packages, services.)
|
|
65
|
-
- uses: ruby/setup-ruby@v1
|
|
66
|
-
with:
|
|
67
|
-
ruby-version: "3.4" # (or none, with a .ruby-version)
|
|
68
|
-
bundler-cache: true
|
|
69
64
|
# The labels of the pull request (or for a push, of the one that
|
|
70
65
|
# merged it), for c.accept_label in .duel.rb, as a JSON array. They're
|
|
71
|
-
# read as the job runs,
|
|
66
|
+
# read as the job runs, and adding the label (or taking it off) runs
|
|
67
|
+
# the job again: see the Perf comment workflow.
|
|
68
|
+
# On a schedule, the last commit since the last release that merged a
|
|
69
|
+
# pull request with the label (ACCEPT_LABEL, as .duel.rb names it):
|
|
70
|
+
# what that slowed down was accepted, so it's compared with instead.
|
|
72
71
|
- name: Find the pull request's labels
|
|
73
72
|
id: labels
|
|
74
|
-
if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push'
|
|
73
|
+
if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push' || github.event_name == 'schedule'
|
|
75
74
|
continue-on-error: true # (without them, nothing's accepted)
|
|
76
75
|
env:
|
|
77
76
|
GH_TOKEN: ${{ github.token }}
|
|
78
77
|
PR: ${{ github.event.pull_request.number }}
|
|
79
78
|
QUEUED: ${{ github.event.merge_group.head_ref }}
|
|
79
|
+
ACCEPT_LABEL: 'duel: accept'
|
|
80
|
+
RELEASE_TAGS: "v*"
|
|
80
81
|
run: |
|
|
82
|
+
if [ "$GITHUB_EVENT_NAME" = schedule ]; then
|
|
83
|
+
tag=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || exit 0
|
|
84
|
+
since=$(TZ=UTC git log -1 --date=format-local:%Y-%m-%dT%H:%M:%SZ --format=%cd "$tag")
|
|
85
|
+
merged=$(gh pr list --repo "$GITHUB_REPOSITORY" --state merged --label "$ACCEPT_LABEL" --search "merged:>=$since" --limit 1000 \
|
|
86
|
+
--json mergeCommit --jq '.[].mergeCommit.oid')
|
|
87
|
+
accepted=$(git rev-list "$tag..HEAD" | grep -Fx -m 1 -f <(echo "$merged")) || accepted=
|
|
88
|
+
echo "accepted=$accepted" >> "$GITHUB_OUTPUT"
|
|
89
|
+
exit 0
|
|
90
|
+
fi
|
|
81
91
|
# (In a merge queue, the last pull request queued, which the queue's
|
|
82
92
|
# branch names: gh-readonly-queue/main/pr-123-<sha>.)
|
|
83
93
|
if [ -n "$QUEUED" ]; then
|
|
@@ -90,6 +100,11 @@ jobs:
|
|
|
90
100
|
labels=$(gh api "repos/$GITHUB_REPOSITORY/commits/$GITHUB_SHA/pulls" --jq '[.[0].labels[]?.name]')
|
|
91
101
|
fi
|
|
92
102
|
echo "labels=$labels" >> "$GITHUB_OUTPUT"
|
|
103
|
+
# (And whatever else your tests need set up: packages, services.)
|
|
104
|
+
- uses: ruby/setup-ruby@v1
|
|
105
|
+
with:
|
|
106
|
+
ruby-version: "3.4" # (or none, with a .ruby-version)
|
|
107
|
+
bundler-cache: true
|
|
93
108
|
# Against the branch it targets for a pull request, the queue's base
|
|
94
109
|
# for a merge queue, what was there before for a push (or as
|
|
95
110
|
# c.push_base in .duel.rb says, the last commit Duel passed), and the
|
|
@@ -105,6 +120,7 @@ jobs:
|
|
|
105
120
|
# with itself, which checks the setup.)
|
|
106
121
|
DUEL_FORCE: ${{ github.event_name == 'workflow_dispatch' && '1' || '' }}
|
|
107
122
|
DUEL_LABELS: ${{ steps.labels.outputs.labels }}
|
|
123
|
+
DUEL_ACCEPTED: ${{ steps.labels.outputs.accepted }}
|
|
108
124
|
run: bundle exec duel --markdown "$RUNNER_TEMP/duel/duel.md"
|
|
109
125
|
# For a pull request, what Duel said, for the Perf comment workflow,
|
|
110
126
|
# which can't see this job's outputs.
|
|
@@ -149,8 +165,11 @@ The workflow compares the following:
|
|
|
149
165
|
the last commit that Duel passed. If that commit isn't in the history,
|
|
150
166
|
such as after a force push or for a branch's first push, Duel reports
|
|
151
167
|
that there's nothing to compare against, and passes.
|
|
152
|
-
- On a schedule, the last release
|
|
153
|
-
|
|
168
|
+
- On a schedule, the last release, or with
|
|
169
|
+
[`c.accept_label`](#accept-a-regression-with-a-label), the last commit
|
|
170
|
+
since then that merged a pull request with the label. Until you tag
|
|
171
|
+
your first release, Duel reports that there's nothing to compare
|
|
172
|
+
against, and passes.
|
|
154
173
|
- When you run the workflow manually, your code against itself. This
|
|
155
174
|
checks that the benchmarks run on the CI machines.
|
|
156
175
|
|
|
@@ -237,12 +256,23 @@ name: Perf comment
|
|
|
237
256
|
# ran), it trusts only what that job could have said: the pull request's
|
|
238
257
|
# number, if it names the pull request the run compared, and the comment
|
|
239
258
|
# and the status, if they're ones Duel says. A fork's report, which the
|
|
240
|
-
# fork's code wrote, is shown as text.
|
|
241
|
-
#
|
|
259
|
+
# fork's code wrote, is shown as text.
|
|
260
|
+
#
|
|
261
|
+
# And when the label that accepts a regression (c.accept_label in
|
|
262
|
+
# .duel.rb) is added to a pull request, or taken off, it runs the Duel
|
|
263
|
+
# job again, which reads the labels as it runs: a run that failed, once
|
|
264
|
+
# the label's added, or one that passed, once it's taken off.
|
|
265
|
+
#
|
|
266
|
+
# (The workflow must be on the default branch before it runs at all.)
|
|
242
267
|
on:
|
|
243
268
|
workflow_run:
|
|
244
269
|
workflows: ["Perf"]
|
|
245
270
|
types: [completed]
|
|
271
|
+
# (This repository's own workflow, as for the comment, with a token
|
|
272
|
+
# that can run a workflow again: it checks nothing out, and runs none
|
|
273
|
+
# of the pull request's code.)
|
|
274
|
+
pull_request_target:
|
|
275
|
+
types: [labeled, unlabeled]
|
|
246
276
|
|
|
247
277
|
permissions: {}
|
|
248
278
|
|
|
@@ -283,12 +313,16 @@ jobs:
|
|
|
283
313
|
# ran the fork's code, which could have written anything in it. In
|
|
284
314
|
# a code block, fenced with more backticks than any run of them in
|
|
285
315
|
# the report, it can't mention anyone, link anywhere or show an
|
|
286
|
-
# image.
|
|
316
|
+
# image. It's cut short where a comment couldn't hold it, and loses
|
|
317
|
+
# any NUL bytes, which GitHub wouldn't take, and grep reads it as
|
|
318
|
+
# text whatever's in it, so it finds every run of backticks.
|
|
287
319
|
report() {
|
|
288
320
|
if [ "${head#* }" = "$GITHUB_REPOSITORY" ]; then cat "$RUNNER_TEMP/report/duel.md" && return; fi
|
|
289
|
-
|
|
321
|
+
text="$RUNNER_TEMP/report.txt"
|
|
322
|
+
LC_ALL=C tr -d '\000' < "$RUNNER_TEMP/report/duel.md" | head -c 60000 > "$text"
|
|
323
|
+
fence=$(LC_ALL=C grep -ao '``*' "$text" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
|
|
290
324
|
printf 'From a fork, so shown as text:\n\n%s\n' "$fence"
|
|
291
|
-
|
|
325
|
+
cat "$text"
|
|
292
326
|
printf '\n%s\n' "$fence"
|
|
293
327
|
}
|
|
294
328
|
marker="<!-- duel"
|
|
@@ -300,6 +334,33 @@ jobs:
|
|
|
300
334
|
{ echo "$marker status=$STATUS -->"; report; } > "$RUNNER_TEMP/comment.md"
|
|
301
335
|
gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
|
|
302
336
|
fi
|
|
337
|
+
|
|
338
|
+
again:
|
|
339
|
+
name: Run Duel again
|
|
340
|
+
if: >-
|
|
341
|
+
github.event_name == 'pull_request_target' && github.event.label.name == 'duel: accept'
|
|
342
|
+
runs-on: ubuntu-latest
|
|
343
|
+
permissions:
|
|
344
|
+
actions: write # to run the Perf workflow's jobs again
|
|
345
|
+
steps:
|
|
346
|
+
- name: Run Duel again
|
|
347
|
+
env:
|
|
348
|
+
GH_TOKEN: ${{ github.token }}
|
|
349
|
+
GH_REPO: ${{ github.repository }}
|
|
350
|
+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
|
351
|
+
ACTION: ${{ github.event.action }}
|
|
352
|
+
run: |
|
|
353
|
+
# The newest Perf run of the pull request's head commit, once it's
|
|
354
|
+
# done (one that's running may have read the labels already).
|
|
355
|
+
id=$(gh run list --workflow "Perf" --event pull_request --commit "$HEAD_SHA" --limit 1 \
|
|
356
|
+
--json databaseId --jq '.[0].databaseId // empty')
|
|
357
|
+
[ -n "$id" ] || exit 0
|
|
358
|
+
gh run watch "$id" --interval 30 > /dev/null || true
|
|
359
|
+
conclusion=$(gh run view "$id" --json conclusion --jq .conclusion)
|
|
360
|
+
case "$ACTION:$conclusion" in
|
|
361
|
+
labeled:failure) gh run rerun "$id" --failed ;;
|
|
362
|
+
unlabeled:success) gh run rerun "$id" ;;
|
|
363
|
+
esac
|
|
303
364
|
```
|
|
304
365
|
|
|
305
366
|
## Comment on a pushed commit
|
|
@@ -374,6 +435,23 @@ or for a push, of the pull request that merged it, and passes them to
|
|
|
374
435
|
`pull-requests: read` permission. Without the labels, such as when
|
|
375
436
|
GitHub's API fails, nothing is accepted.
|
|
376
437
|
|
|
438
|
+
The labels are read as the Duel job runs, so `perf-comment.yml` runs
|
|
439
|
+
the job again when the label changes. Its Run Duel again job runs on
|
|
440
|
+
`pull_request_target`, with this repository's own workflow, when a
|
|
441
|
+
label is added to a pull request or taken off. For the label `.duel.rb`
|
|
442
|
+
names, it waits for the Perf run of the pull request's head commit to
|
|
443
|
+
finish. Then it runs the failed jobs again where the label was added
|
|
444
|
+
to a run that failed, or the whole run again where the label was
|
|
445
|
+
taken off one that passed. It checks nothing out and runs none of the
|
|
446
|
+
pull request's code, and its only permission is `actions: write`. The
|
|
447
|
+
workflows name the label as `.duel.rb` did when `duel init` wrote them,
|
|
448
|
+
so if you rename it, rename it in both workflows too.
|
|
449
|
+
|
|
450
|
+
On a schedule, the same step looks for the pull requests merged since
|
|
451
|
+
the last release with the label, and passes `duel` the last commit
|
|
452
|
+
that merged one, in `DUEL_ACCEPTED`. Duel compares with that commit
|
|
453
|
+
instead of the release, as what it slowed down was accepted.
|
|
454
|
+
|
|
377
455
|
## Compare a push with the last commit that passed
|
|
378
456
|
|
|
379
457
|
By default, the workflow compares each push with the commit before it.
|
|
@@ -390,8 +468,10 @@ branch that the workflow guards has its own (see
|
|
|
390
468
|
nothing to compare, Duel sets the `passed` output to `true`, and the
|
|
391
469
|
Mark the commit passed job moves the branch to the push, or creates the
|
|
392
470
|
branch. It never moves the branch back: when two pushes run at once and
|
|
393
|
-
the older one finishes last, the branch stays at the newer push.
|
|
394
|
-
a
|
|
471
|
+
the older one finishes last, the branch stays at the newer push. The
|
|
472
|
+
job runs one at a time for each branch, so two runs can't race to
|
|
473
|
+
create the branch. After a force push rewrites the branch's history,
|
|
474
|
+
it moves the branch anyway.
|
|
395
475
|
That job checks nothing out either, and it gets the `contents: write`
|
|
396
476
|
permission that moving a branch takes. Until the branch exists, and
|
|
397
477
|
when the push's history doesn't have its commit, Duel compares the push
|
|
@@ -438,6 +518,13 @@ yourself:
|
|
|
438
518
|
passed:
|
|
439
519
|
name: Mark the commit passed
|
|
440
520
|
needs: duel
|
|
521
|
+
# One at a time for each branch, so that two runs can't both find no
|
|
522
|
+
# branch and race to make it (a push's run doesn't wait for the one
|
|
523
|
+
# before it). One that's running is never cancelled, and one that's
|
|
524
|
+
# waiting gives way to a newer push's, which moves it further anyway.
|
|
525
|
+
concurrency:
|
|
526
|
+
group: ${{ github.workflow }}-passed-${{ github.ref_name }}
|
|
527
|
+
cancel-in-progress: false
|
|
441
528
|
if: ${{ !cancelled() && github.event_name == 'push' && needs.duel.outputs.passed == 'true' }}
|
|
442
529
|
runs-on: ubuntu-latest
|
|
443
530
|
permissions:
|
|
@@ -461,13 +548,16 @@ yourself:
|
|
|
461
548
|
done
|
|
462
549
|
exit 1
|
|
463
550
|
}
|
|
464
|
-
# (Forward: GitHub moves it only where it's behind the push.
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
gh api --method
|
|
469
|
-
|
|
470
|
-
|
|
551
|
+
# (Forward: GitHub moves it only where it's behind the push. Where
|
|
552
|
+
# there's no branch, and making it fails, it looks once more, as
|
|
553
|
+
# something else may have made it: a push of it, say.)
|
|
554
|
+
for _ in 1 2; do
|
|
555
|
+
gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
|
|
556
|
+
at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
|
|
557
|
+
[ -n "$at" ] && break
|
|
558
|
+
gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null && exit 0
|
|
559
|
+
done
|
|
560
|
+
[ -n "$at" ] || blocked
|
|
471
561
|
compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status 2>&1) || case "$compared" in
|
|
472
562
|
# (GitHub can't compare commits with no history in common.)
|
|
473
563
|
*"No common ancestor"*) compared=unrelated ;;
|
|
@@ -825,6 +915,9 @@ duel:passed:
|
|
|
825
915
|
image: alpine:3
|
|
826
916
|
stage: .post
|
|
827
917
|
needs: [duel]
|
|
918
|
+
# One at a time, as each deletes the branch and makes it again: two at
|
|
919
|
+
# once could both delete it, and the second to make it would fail.
|
|
920
|
+
resource_group: duel/passed
|
|
828
921
|
rules:
|
|
829
922
|
- if: $DUEL_TOKEN && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
|
|
830
923
|
variables:
|
|
@@ -894,7 +987,8 @@ match. `duel init` adds these steps for a project that has a
|
|
|
894
987
|
# request targets (or run by hand, the main branch), a merge queue's
|
|
895
988
|
# base, the commit before a push (or the last commit Duel passed, PASSED, if there is
|
|
896
989
|
# one: see the Mark the commit passed job), or on a schedule the last
|
|
897
|
-
# release, the newest tag before HEAD that RELEASE_TAGS matches
|
|
990
|
+
# release, the newest tag before HEAD that RELEASE_TAGS matches
|
|
991
|
+
# (or ACCEPTED, the last change accepted since then). (If
|
|
898
992
|
# .duel.rb sets c.main_branch or c.release_tags, set MAIN_BRANCH or
|
|
899
993
|
# RELEASE_TAGS to match.)
|
|
900
994
|
- name: Find the baseline's Ruby
|
|
@@ -905,13 +999,14 @@ match. `duel init` adds these steps for a project that has a
|
|
|
905
999
|
PASSED: origin/duel/passed/${{ github.ref_name }}
|
|
906
1000
|
MAIN_BRANCH: origin/${{ github.base_ref || github.event.repository.default_branch }}
|
|
907
1001
|
RELEASE_TAGS: "v*"
|
|
1002
|
+
ACCEPTED: ${{ steps.labels.outputs.accepted }}
|
|
908
1003
|
run: |
|
|
909
1004
|
case "$GITHUB_EVENT_NAME" in
|
|
910
1005
|
push)
|
|
911
1006
|
base=$BEFORE
|
|
912
1007
|
if git merge-base --is-ancestor "$PASSED" HEAD 2>/dev/null; then base=$PASSED; fi ;;
|
|
913
1008
|
merge_group) base=$QUEUE_BASE ;;
|
|
914
|
-
schedule) base=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || base= ;;
|
|
1009
|
+
schedule) base=${ACCEPTED:-$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null)} || base= ;;
|
|
915
1010
|
*) base=$(git merge-base HEAD "$MAIN_BRANCH" 2>/dev/null) || base= ;;
|
|
916
1011
|
esac
|
|
917
1012
|
ruby_of() { git show "$1:./.ruby-version" 2>/dev/null | awk 'NF { print $1; exit }'; }
|
data/docs/configuration.md
CHANGED
|
@@ -140,12 +140,16 @@ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
|
|
|
140
140
|
that `duel init` writes passes Duel the pull request's labels. With
|
|
141
141
|
that label, a regression passes, with a warning, and the report says
|
|
142
142
|
that the label accepted it. The labels are read when the workflow runs,
|
|
143
|
-
so
|
|
144
|
-
pull request
|
|
145
|
-
|
|
143
|
+
so the comment workflow runs the Duel job again when you add the label
|
|
144
|
+
to a pull request whose run failed, or take it off one whose run
|
|
145
|
+
passed. For a push, Duel reads the labels of the pull request that
|
|
146
|
+
merged it, so with
|
|
146
147
|
[`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
|
|
147
148
|
the merged regression becomes the commit that later pushes are compared
|
|
148
|
-
with. Duel
|
|
149
|
+
with. On a schedule, Duel compares with the last commit since your last
|
|
150
|
+
release that merged a pull request with the label, instead of the
|
|
151
|
+
release, so a slowdown you accepted doesn't fail every night until
|
|
152
|
+
your next release. Duel's default is no such label, but the `.duel.rb` that
|
|
149
153
|
`duel init` writes (and duel-rails's generator) names `duel: accept`.
|
|
150
154
|
Create the label in your repository to use it, or set
|
|
151
155
|
`c.accept_label = nil` so that nothing can let a regression through.
|
data/docs/how-it-works.md
CHANGED
|
@@ -17,25 +17,40 @@ never on noise, so that it can gate every pull request:
|
|
|
17
17
|
only ever adds time. Garbage collection is kept out of samples, and
|
|
18
18
|
its cost shows in the allocation count instead, which is exact.
|
|
19
19
|
- **Side by side, many times over.** The two versions run in worker
|
|
20
|
-
processes of their own,
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
- **A confidence interval, not an average.**
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
20
|
+
processes of their own, at the same time on the same machine: twelve
|
|
21
|
+
of each by default, or eight on JRuby and TruffleRuby, whose workers
|
|
22
|
+
are JVMs. They take turns, one sample from each in alternating order,
|
|
23
|
+
in rounds. Each round takes a sample from every worker, and pairs
|
|
24
|
+
each baseline worker with a different candidate worker than the
|
|
25
|
+
last round did. That way, a busy moment weighs on both versions
|
|
26
|
+
alike. There are no stored numbers and no thresholds to maintain,
|
|
27
|
+
and any machine will do.
|
|
28
|
+
- **A confidence interval over processes, not an average.** Where
|
|
29
|
+
things land in a process's memory can make the very same code a few
|
|
30
|
+
percent faster or slower for the process's whole life, and on JRuby,
|
|
31
|
+
so can what its JIT compiled. So a sample from one process isn't
|
|
32
|
+
independent of the next sample from it, and the unit Duel judges is
|
|
33
|
+
the process: each worker's median. A timing is slower only when a
|
|
34
|
+
99% distribution-free confidence interval for the ratio of the
|
|
35
|
+
candidate's workers to the baseline's (the Mann-Whitney interval,
|
|
36
|
+
over every candidate worker's median against every baseline
|
|
37
|
+
worker's) lies entirely above the limit, which is 1.02 by default.
|
|
38
|
+
No one outlier sample can decide it, and no one lucky or unlucky
|
|
39
|
+
worker can either: the interval is as wide as the workers make it.
|
|
40
|
+
An interval over the pairs would be as narrow as if every pair had
|
|
41
|
+
processes of its own, and with identical code on both sides, it
|
|
42
|
+
would often leave out 1. Duel takes rounds until the interval
|
|
43
|
+
settles the question, from the fourth round on, and up to some 120
|
|
44
|
+
pairs. Duel reports a timing that's still too close to the limit to
|
|
45
|
+
call, and doesn't fail on it.
|
|
46
|
+
|
|
47
|
+
The interval over workers takes five of each at least. With fewer
|
|
48
|
+
(`--processes`), Duel judges over the pairs, as it does a process
|
|
49
|
+
benchmark, and its first line says so. A process benchmark's
|
|
50
|
+
sample is the fastest of a few fresh processes, so its pairs are
|
|
51
|
+
independent already. It can also stop after any pair once the
|
|
35
52
|
interval shows that it isn't slower, which only ever ends a
|
|
36
53
|
measurement before it could find a slowdown.
|
|
37
|
-
Duel reports a timing that's still too close to the limit to call, and
|
|
38
|
-
doesn't fail on it.
|
|
39
54
|
- **Measured again before failing.** Duel measures whatever looks
|
|
40
55
|
slower again from scratch, up to three times, in fresh processes. It
|
|
41
56
|
fails only if every measurement finds the same metric worse. A
|
|
@@ -54,15 +69,27 @@ never on noise, so that it can gate every pull request:
|
|
|
54
69
|
than it otherwise would.
|
|
55
70
|
- **Exact counts.** Allocations (on CRuby), and any counted metric of
|
|
56
71
|
your own, fail on any increase: one more allocation per two
|
|
57
|
-
iterations.
|
|
72
|
+
iterations. Each worker's count is its fewest in three rounds of a
|
|
73
|
+
sample from every worker, and a count fails when an interval over the
|
|
74
|
+
workers, for how many more the candidate's are, lies above the
|
|
75
|
+
tolerance. Where every worker counts the same, as CRuby's workers
|
|
76
|
+
do, that's exactly how many more. Where workers differ, it's as few
|
|
77
|
+
as they can tell apart, so one worker's luck can't fail a benchmark.
|
|
78
|
+
With fewer than five workers of each version, a count is each
|
|
79
|
+
version's fewest in three samples instead.
|
|
58
80
|
- **Across the board, too.** A slowdown that's too small for any one
|
|
59
81
|
benchmark to settle, such as 1% everywhere, still shows in all of the
|
|
60
|
-
benchmarks together.
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
82
|
+
benchmarks together. The same workers measure every benchmark, so a
|
|
83
|
+
worker whose luck makes it slower is slower at all of them, and the
|
|
84
|
+
unit is the worker again. Each worker has a score: the median, over
|
|
85
|
+
the benchmarks it measured, of how much faster or slower it was than
|
|
86
|
+
the other workers of its lane. The candidate fails when a confidence
|
|
87
|
+
interval for the ratio of its workers' scores to the baseline's lies
|
|
88
|
+
above a quarter of the metric's margin: 1.005, for CPU time's 1.02.
|
|
89
|
+
Where the benchmarks were judged over pairs, every benchmark's median
|
|
90
|
+
ratio is a sample instead, and the interval is for their median.
|
|
91
|
+
Either takes 8 benchmarks, so with fewer, Duel doesn't report it.
|
|
92
|
+
Duel measures again before it fails, as it does for the rest.
|
|
66
93
|
|
|
67
94
|
## What Duel measures
|
|
68
95
|
|
|
@@ -73,6 +100,7 @@ Duel judges every benchmark on the following metrics:
|
|
|
73
100
|
| `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
|
|
74
101
|
| `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
|
|
75
102
|
| `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
|
|
103
|
+
| `allocated_bytes` | Bytes allocated (on JRuby), by the JVM's count | More than 32 bytes more per iteration |
|
|
76
104
|
| `peak_memory` | The most resident memory (on JRuby, the JVM's heap in use), above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
|
|
77
105
|
| Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
|
|
78
106
|
|
|
@@ -216,21 +244,67 @@ when the gems are installed in the project's directory, as
|
|
|
216
244
|
|
|
217
245
|
## JIT compilers
|
|
218
246
|
|
|
219
|
-
If YJIT is on in a worker, Duel runs each benchmark
|
|
220
|
-
before it measures it. A small library's methods
|
|
221
|
-
Duel's first samples, but in a big app, YJIT
|
|
222
|
-
120 calls. YJIT can be on because you
|
|
223
|
-
for example), or because your code
|
|
224
|
-
|
|
247
|
+
If YJIT is on in a worker, Duel runs each benchmark until it's steady
|
|
248
|
+
before it measures it, for up to ten seconds. A small library's methods
|
|
249
|
+
are compiled within Duel's first samples, but in a big app, YJIT
|
|
250
|
+
compiles a method only after 120 calls. YJIT can be on because you
|
|
251
|
+
turned it on (`RUBY_YJIT_ENABLE=1`, for example), or because your code
|
|
252
|
+
did as it loaded, as Rails 7.2 and later do.
|
|
253
|
+
|
|
254
|
+
A worker is steady once the median of its samples over each of three
|
|
255
|
+
half seconds in a row is no more than 1% faster than over the last half
|
|
256
|
+
second that was, so a pause (a collection, say) doesn't end its warm-up,
|
|
257
|
+
and code that creeps faster by less a half second isn't steady. Duel
|
|
258
|
+
measures once every worker is steady, or the time is up. To warm up for
|
|
259
|
+
a set time instead, use `--warm`.
|
|
225
260
|
|
|
226
261
|
### JRuby and TruffleRuby
|
|
227
262
|
|
|
228
263
|
JRuby's and TruffleRuby's JIT compilers keep compiling for seconds, so
|
|
229
|
-
Duel runs each benchmark for
|
|
230
|
-
(`--warm`).
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
264
|
+
Duel runs each benchmark until it's steady, for up to a minute, before
|
|
265
|
+
it measures it (`--warm`). Code that a JIT compiles quickly is steady
|
|
266
|
+
in a few seconds, but some code is still several times faster after
|
|
267
|
+
five seconds than it was, and steady only after 10 or 20.
|
|
268
|
+
|
|
269
|
+
Each JVM's JIT also compiles the same code a little differently, so one
|
|
270
|
+
worker can run a benchmark faster or slower than another for its whole
|
|
271
|
+
life, however long it warms up. On JRuby, Duel starts each worker's JVM
|
|
272
|
+
with `-Xbatch`, which compiles in the foreground, so what it compiles is
|
|
273
|
+
less a matter of timing, and with the serial garbage collector, which
|
|
274
|
+
runs no threads of its own beside the code. That makes the workers about
|
|
275
|
+
twice as alike, but a CPU-time interval on JRuby is still some 5% to 8%
|
|
276
|
+
wide, as wide as the workers make it, so Duel leaves a timing regression
|
|
277
|
+
of a few percent unsettled there, where it would find it on CRuby. It
|
|
278
|
+
doesn't fail identical code on that noise: an interval over the pairs
|
|
279
|
+
would be narrower, but it would often leave out 1 for identical code.
|
|
280
|
+
The options come after any in `JRUBY_OPTS`, and Duel doesn't add the
|
|
281
|
+
serial collector where `JRUBY_OPTS` chooses another. The processes a
|
|
282
|
+
process benchmark starts don't get them, as their boot is what's
|
|
283
|
+
measured.
|
|
284
|
+
|
|
285
|
+
Their workers are big JVMs, and their compilers keep other CPUs busy,
|
|
286
|
+
so Duel starts eight workers of each version rather than twelve, and
|
|
287
|
+
measures one benchmark at a time for every four CPUs (`--jobs`), not
|
|
288
|
+
every two. More at once is no faster there, and takes more memory: some
|
|
289
|
+
500 MB a worker.
|
|
290
|
+
|
|
291
|
+
JRuby counts no objects allocated, but the JVM counts the bytes each
|
|
292
|
+
thread allocates, and Duel judges those instead (`allocated_bytes`):
|
|
293
|
+
more than 32 bytes more an iteration fails, about a small object's, and
|
|
294
|
+
a change that makes objects bigger counts too. Where every worker counts
|
|
295
|
+
the same, Duel finds one more small object an iteration. But the JIT can
|
|
296
|
+
do away with an allocation that never leaves a method (escape analysis),
|
|
297
|
+
and each JVM's JIT does away with different ones, so on some code,
|
|
298
|
+
workers' counts differ by thousands of bytes, and an interval over the
|
|
299
|
+
workers (see How Duel tells a regression from noise) is too wide to find
|
|
300
|
+
a few more. And an allocation the JIT does away with in every worker
|
|
301
|
+
isn't counted at all, as it isn't made. Turning escape analysis off
|
|
302
|
+
would count every allocation, but makes CPU time some three times
|
|
303
|
+
noisier, so Duel leaves it on. The count falls as the JIT compiles the
|
|
304
|
+
code, so it's taken after the warm-up. With fewer than five workers of
|
|
305
|
+
each version, it's each version's fewest in 10 samples, as a sample now
|
|
306
|
+
and then is some 31 bytes an iteration more than the rest. TruffleRuby
|
|
307
|
+
has no allocation count. Neither has an instruction count or profiles.
|
|
234
308
|
TruffleRuby has peak memory only on Linux. JRuby measures peak memory in
|
|
235
309
|
the JVM's heap in use, on any operating system, because the JVM takes
|
|
236
310
|
memory for its heap ahead of time, so resident memory barely moves as
|