duel 0.1.5 → 0.1.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +135 -1
- data/README.md +5 -5
- data/docs/ci.md +145 -42
- data/docs/configuration.md +8 -4
- data/docs/how-it-works.md +123 -35
- data/docs/json.md +18 -12
- data/docs/plugins.md +3 -1
- data/docs/troubleshooting.md +6 -6
- data/lib/duel/across.rb +100 -9
- data/lib/duel/baseline.rb +26 -2
- data/lib/duel/cli.rb +25 -7
- data/lib/duel/config.rb +4 -1
- data/lib/duel/init.rb +19 -3
- data/lib/duel/json_report.rb +2 -2
- data/lib/duel/memory.rb +54 -0
- data/lib/duel/metric.rb +32 -1
- data/lib/duel/rubies.rb +39 -0
- data/lib/duel/runner.rb +383 -49
- data/lib/duel/stats.rb +76 -0
- data/lib/duel/summary.rb +2 -2
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +3 -0
- data/lib/duel/templates/perf-comment.yml.erb +53 -5
- data/lib/duel/templates/perf.yml.erb +85 -42
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +5 -4
- data/sig/duel/across.rbs +24 -2
- data/sig/duel/baseline.rbs +12 -0
- data/sig/duel/cli.rbs +4 -0
- data/sig/duel/init.rbs +6 -0
- data/sig/duel/memory.rbs +30 -0
- data/sig/duel/metric.rbs +19 -1
- data/sig/duel/rubies.rbs +13 -0
- data/sig/duel/runner.rbs +129 -6
- data/sig/duel/stats.rbs +25 -0
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2c2e7b0e6e07fe45ff7a77955035a06be6850e761c9ff2f0b10c0ab4b5130eed
|
|
4
|
+
data.tar.gz: fe086be96d5238e8befcffd0bc7adfb8ef94d0a83790f1ab74a82f881cc3786e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 731c24cb25017b1f1fba0546523c8c895400eb2750d260e58072c9537afc952fdf2b29bbc90d6242ef5b9463004203782c637cb21c5ea4bf075291866a44a939
|
|
7
|
+
data.tar.gz: 6b98fb68a77b594457ea5b0f9b3bb88fab03c55a606ce7741ae7c970a4cab829000521d62effce598ed64ca362fca1e0b7e1ee50b209fa9b800fb4e6e16e2839
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,138 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.7] - 2026-11-10
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- Duel measures fewer benchmarks at once, and where even one
|
|
15
|
+
benchmark's workers wouldn't fit, starts fewer workers, so that they
|
|
16
|
+
fit in the memory available. Since 0.1.6 starts twelve workers of
|
|
17
|
+
each version for each benchmark it measures at once, a plugin whose
|
|
18
|
+
workers each start a server could run a machine out of memory: on a
|
|
19
|
+
GitHub runner, duel-mysql's 48 MySQL servers took more than its
|
|
20
|
+
16 GB, and the runner stopped answering until its job was lost. Duel
|
|
21
|
+
starts one worker of each version first to find what a pair takes
|
|
22
|
+
(with its resources), and allows each pair a quarter more, and 1 GB
|
|
23
|
+
for everything else. It reads what's available from Linux's
|
|
24
|
+
MemAvailable (or a container's cgroup limit, if lower) or macOS's
|
|
25
|
+
`vm_stat`, and its first line says when it lowered either. A
|
|
26
|
+
`--jobs` or `--processes` you give is kept as it is.
|
|
27
|
+
- The badge (in the workflow that `duel init --badge` writes) says
|
|
28
|
+
"not measured", in grey, where the Duel job failed
|
|
29
|
+
before Duel judged the push (its runner lost, say, or a step before
|
|
30
|
+
Duel's failed), instead of "perf regressed". It's red only where
|
|
31
|
+
Duel failed the push or found a regression, and a job that fails
|
|
32
|
+
after Duel passed the push leaves it green.
|
|
33
|
+
|
|
34
|
+
## [0.1.6] - 2026-11-10
|
|
35
|
+
|
|
36
|
+
### Added
|
|
37
|
+
|
|
38
|
+
- The comment workflow that `duel init` writes runs the Duel job again
|
|
39
|
+
when the label that accepts a regression (`c.accept_label`) is added
|
|
40
|
+
to a pull request whose run failed, or taken off one whose run
|
|
41
|
+
passed. Before, the label did nothing until someone ran the job again
|
|
42
|
+
by hand. The job runs on `pull_request_target`, checks nothing out,
|
|
43
|
+
and has only `actions: write`.
|
|
44
|
+
- On a schedule, the workflow compares with the last commit since the
|
|
45
|
+
last release that merged a pull request with the label, instead of
|
|
46
|
+
the release, so a slowdown accepted in its pull request doesn't fail
|
|
47
|
+
every nightly run until the next release. Duel reads that commit from
|
|
48
|
+
`DUEL_ACCEPTED`, and the report says what it compared with.
|
|
49
|
+
- The JSON report says, for each timing and across the board, how many
|
|
50
|
+
worker processes of each version the interval is over (`processes`),
|
|
51
|
+
or `null` where it's over pairs or benchmarks.
|
|
52
|
+
- The nightly bias check (`tasks/bias_check.rb`) also fails where more
|
|
53
|
+
of a metric's intervals of identical code leave out 1 than their
|
|
54
|
+
confidence allows, and runs on JRuby too, and on a pull request that
|
|
55
|
+
changes how Duel measures or judges.
|
|
56
|
+
|
|
57
|
+
- On JRuby, Duel counts the bytes each benchmark allocates
|
|
58
|
+
(`allocated_bytes`), as the JVM counts them for each thread (or every
|
|
59
|
+
thread, for a benchmark with threads of its own), and fails one that
|
|
60
|
+
allocates more than 32 bytes more an iteration. Before, JRuby had no
|
|
61
|
+
allocation count. Where each JVM's JIT does away with different
|
|
62
|
+
allocations (escape analysis), workers' counts differ, and Duel finds
|
|
63
|
+
only as many more as they can tell apart.
|
|
64
|
+
- A count is over workers too: each worker's fewest in three rounds of
|
|
65
|
+
a sample from every worker, and it fails when an interval for how many
|
|
66
|
+
more the candidate's workers count than the baseline's lies above the
|
|
67
|
+
tolerance. Before, it was each version's fewest of 3 samples, and on
|
|
68
|
+
JRuby, where one worker's JIT can do away with an allocation another's
|
|
69
|
+
doesn't, one worker's luck could fail identical code. Where every
|
|
70
|
+
worker counts the same, as on CRuby, it's as exact as before. With
|
|
71
|
+
fewer than 5 workers of each version, it's the fewest of 3 samples (10
|
|
72
|
+
on JRuby), as before.
|
|
73
|
+
- On JRuby and TruffleRuby, a worker may take 5 minutes to load or to
|
|
74
|
+
answer (`--timeout`), not 1: with `-Xbatch`, 16 JVMs starting at once
|
|
75
|
+
could take longer than a minute.
|
|
76
|
+
- Counts of 9999.5 or more an iteration show as whole numbers (25736, not
|
|
77
|
+
2.574e+04).
|
|
78
|
+
|
|
79
|
+
### Changed
|
|
80
|
+
|
|
81
|
+
- A timing's interval is over worker processes, not pairs of samples.
|
|
82
|
+
Where things land in a process's memory (or on JRuby, what its JIT
|
|
83
|
+
compiles) can make the same code faster or slower for the process's
|
|
84
|
+
whole life, so the pairs of 3 workers of each version were far from
|
|
85
|
+
independent, and with identical code on both sides, their intervals
|
|
86
|
+
left out 1 several times as often as 99% intervals should. Each
|
|
87
|
+
worker's median is now a sample, and the interval is the Mann-Whitney
|
|
88
|
+
interval for the ratio of the candidate's workers to the baseline's.
|
|
89
|
+
Duel takes rounds of a sample from every worker, up to some 120
|
|
90
|
+
pairs, and starts 12 workers of each version by default (8 on JRuby
|
|
91
|
+
and TruffleRuby). CPU-time intervals are about as wide as before, or
|
|
92
|
+
narrower, and instruction counts' (which a process repeats almost
|
|
93
|
+
exactly) a little wider. On JRuby, they're some 5% to 8% wide, as
|
|
94
|
+
wide as each JVM's JIT makes them, so Duel leaves smaller timing
|
|
95
|
+
regressions there unsettled. With fewer than 5 workers of each version
|
|
96
|
+
(`--processes`, and `Duel::Testing.compare`'s 2), and for a process
|
|
97
|
+
benchmark, the interval is over the pairs, as before, and the first
|
|
98
|
+
line of the output says which.
|
|
99
|
+
- Across the board, the interval is over the workers too: each worker's
|
|
100
|
+
score is how much faster or slower it was than its lane's other
|
|
101
|
+
workers, at the median of the benchmarks it measured. A worker whose
|
|
102
|
+
luck slowed it at every benchmark could fail identical code across
|
|
103
|
+
the board.
|
|
104
|
+
- Without `--warm`, each worker warms a benchmark up until it's steady
|
|
105
|
+
(the median of its samples over each of three half seconds in a row
|
|
106
|
+
is no more than 1% faster than over the last half second that was),
|
|
107
|
+
for up to a minute on JRuby and TruffleRuby, or 10 seconds with YJIT
|
|
108
|
+
on. Before, it warmed up for 5 seconds, or 2, every time, and on
|
|
109
|
+
JRuby, some code was still several times faster after that, so it
|
|
110
|
+
was measured while it got faster. `--warm` still warms up for the time
|
|
111
|
+
it says.
|
|
112
|
+
- On JRuby, each worker's JVM compiles in the foreground (`-Xbatch`) and
|
|
113
|
+
collects garbage with the serial collector, after any options in
|
|
114
|
+
`JRUBY_OPTS` (and without the serial collector where they choose
|
|
115
|
+
another). Each JVM's JIT compiles the same code a little differently,
|
|
116
|
+
and that made JRuby's intervals over workers twice as wide.
|
|
117
|
+
- A run measures for two hours at most by default (`--time-limit`), not
|
|
118
|
+
half an hour, as on JRuby a benchmark can take half a minute to warm
|
|
119
|
+
up.
|
|
120
|
+
- On JRuby and TruffleRuby, Duel measures one benchmark at a time for
|
|
121
|
+
every four CPUs (`--jobs`), not every two: their JIT compilers keep the
|
|
122
|
+
other CPUs busy, so more were no faster, and each worker is a JVM of
|
|
123
|
+
some 500 MB.
|
|
124
|
+
|
|
125
|
+
### Fixed
|
|
126
|
+
|
|
127
|
+
- The comment workflow fences a fork's report with more backticks than
|
|
128
|
+
any run of them in it even when the report has a NUL byte. GNU grep
|
|
129
|
+
took such a report for binary and found no backticks, so the fence
|
|
130
|
+
was three, and a line of three backticks in the report closed it. The
|
|
131
|
+
workflow now drops NUL bytes, which GitHub doesn't take in a comment,
|
|
132
|
+
and reads the report as text whatever's in it.
|
|
133
|
+
- The Mark the commit passed job runs one at a time for each branch
|
|
134
|
+
(and on GitLab CI, one at a time). Before the branch that marks the
|
|
135
|
+
last commit Duel passed existed, two pushes that passed at once could
|
|
136
|
+
both try to make it, and the second failed, saying another branch
|
|
137
|
+
kept it from being made. Where making the branch fails, the job now
|
|
138
|
+
also looks for it again before it says so. Update the job in
|
|
139
|
+
`.github/workflows/perf.yml` (or `.gitlab/duel.gitlab-ci.yml`) to pick
|
|
140
|
+
this up.
|
|
141
|
+
|
|
10
142
|
## [0.1.5] - 2026-11-08
|
|
11
143
|
|
|
12
144
|
### Changed
|
|
@@ -574,7 +706,9 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
574
706
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
575
707
|
release.
|
|
576
708
|
|
|
577
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.
|
|
709
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.7...HEAD
|
|
710
|
+
[0.1.7]: https://github.com/sferik/duel-ruby/compare/v0.1.6...v0.1.7
|
|
711
|
+
[0.1.6]: https://github.com/sferik/duel-ruby/compare/v0.1.5...v0.1.6
|
|
578
712
|
[0.1.5]: https://github.com/sferik/duel-ruby/compare/v0.1.4...v0.1.5
|
|
579
713
|
[0.1.4]: https://github.com/sferik/duel-ruby/compare/v0.1.3...v0.1.4
|
|
580
714
|
[0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
|
data/README.md
CHANGED
|
@@ -14,7 +14,7 @@ code from becoming slower.
|
|
|
14
14
|
|
|
15
15
|
```sh
|
|
16
16
|
$ duel
|
|
17
|
-
Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02)
|
|
17
|
+
Comparing 12 benchmarks, 2 at a time: baseline /src/.3f9c2a1b against candidate /src/my_parser (slower: a 99.0% interval above 1.02, over 12 worker processes of each version)
|
|
18
18
|
parse: a small document 0.6s n=412 cpu 0.998 [0.991, 1.006] same allocations 20
|
|
19
19
|
! parse: a large document 1.1s n=9 cpu 1.081 [1.062, 1.097] slower allocations 5790
|
|
20
20
|
...
|
|
@@ -193,11 +193,11 @@ and how it compares a project in a subdirectory of its repository, see
|
|
|
193
193
|
### Timeouts and exit status
|
|
194
194
|
|
|
195
195
|
A benchmark that hangs, or that waits on a server that died, doesn't
|
|
196
|
-
hang the run. If a worker takes more than a minute
|
|
197
|
-
Duel stops the worker and reports the
|
|
198
|
-
the limit, use `--timeout`.
|
|
196
|
+
hang the run. If a worker takes more than a minute (five on JRuby and
|
|
197
|
+
TruffleRuby) to load or to answer, Duel stops the worker and reports the
|
|
198
|
+
benchmark as an error. To change the limit, use `--timeout`.
|
|
199
199
|
|
|
200
|
-
A run measures for
|
|
200
|
+
A run measures for two hours at most, not counting the builds. Past
|
|
201
201
|
that, Duel starts no more measurements. A benchmark that it hasn't
|
|
202
202
|
measured yet fails as an error, and a benchmark that looked worse, but
|
|
203
203
|
that Duel hasn't measured again as often as it usually does, fails if it
|
data/docs/ci.md
CHANGED
|
@@ -61,23 +61,33 @@ jobs:
|
|
|
61
61
|
with:
|
|
62
62
|
fetch-depth: 0 # the baseline comes from history, tags included
|
|
63
63
|
persist-credentials: false
|
|
64
|
-
# (And whatever else your tests need set up: packages, services.)
|
|
65
|
-
- uses: ruby/setup-ruby@v1
|
|
66
|
-
with:
|
|
67
|
-
ruby-version: "3.4" # (or none, with a .ruby-version)
|
|
68
|
-
bundler-cache: true
|
|
69
64
|
# The labels of the pull request (or for a push, of the one that
|
|
70
65
|
# merged it), for c.accept_label in .duel.rb, as a JSON array. They're
|
|
71
|
-
# read as the job runs,
|
|
66
|
+
# read as the job runs, and adding the label (or taking it off) runs
|
|
67
|
+
# the job again: see the Perf comment workflow.
|
|
68
|
+
# On a schedule, the last commit since the last release that merged a
|
|
69
|
+
# pull request with the label (ACCEPT_LABEL, as .duel.rb names it):
|
|
70
|
+
# what that slowed down was accepted, so it's compared with instead.
|
|
72
71
|
- name: Find the pull request's labels
|
|
73
72
|
id: labels
|
|
74
|
-
if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push'
|
|
73
|
+
if: github.event_name == 'pull_request' || github.event_name == 'merge_group' || github.event_name == 'push' || github.event_name == 'schedule'
|
|
75
74
|
continue-on-error: true # (without them, nothing's accepted)
|
|
76
75
|
env:
|
|
77
76
|
GH_TOKEN: ${{ github.token }}
|
|
78
77
|
PR: ${{ github.event.pull_request.number }}
|
|
79
78
|
QUEUED: ${{ github.event.merge_group.head_ref }}
|
|
79
|
+
ACCEPT_LABEL: 'duel: accept'
|
|
80
|
+
RELEASE_TAGS: "v*"
|
|
80
81
|
run: |
|
|
82
|
+
if [ "$GITHUB_EVENT_NAME" = schedule ]; then
|
|
83
|
+
tag=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || exit 0
|
|
84
|
+
since=$(TZ=UTC git log -1 --date=format-local:%Y-%m-%dT%H:%M:%SZ --format=%cd "$tag")
|
|
85
|
+
merged=$(gh pr list --repo "$GITHUB_REPOSITORY" --state merged --label "$ACCEPT_LABEL" --search "merged:>=$since" --limit 1000 \
|
|
86
|
+
--json mergeCommit --jq '.[].mergeCommit.oid')
|
|
87
|
+
accepted=$(git rev-list "$tag..HEAD" | grep -Fx -m 1 -f <(echo "$merged")) || accepted=
|
|
88
|
+
echo "accepted=$accepted" >> "$GITHUB_OUTPUT"
|
|
89
|
+
exit 0
|
|
90
|
+
fi
|
|
81
91
|
# (In a merge queue, the last pull request queued, which the queue's
|
|
82
92
|
# branch names: gh-readonly-queue/main/pr-123-<sha>.)
|
|
83
93
|
if [ -n "$QUEUED" ]; then
|
|
@@ -90,6 +100,11 @@ jobs:
|
|
|
90
100
|
labels=$(gh api "repos/$GITHUB_REPOSITORY/commits/$GITHUB_SHA/pulls" --jq '[.[0].labels[]?.name]')
|
|
91
101
|
fi
|
|
92
102
|
echo "labels=$labels" >> "$GITHUB_OUTPUT"
|
|
103
|
+
# (And whatever else your tests need set up: packages, services.)
|
|
104
|
+
- uses: ruby/setup-ruby@v1
|
|
105
|
+
with:
|
|
106
|
+
ruby-version: "3.4" # (or none, with a .ruby-version)
|
|
107
|
+
bundler-cache: true
|
|
93
108
|
# Against the branch it targets for a pull request, the queue's base
|
|
94
109
|
# for a merge queue, what was there before for a push (or as
|
|
95
110
|
# c.push_base in .duel.rb says, the last commit Duel passed), and the
|
|
@@ -105,6 +120,7 @@ jobs:
|
|
|
105
120
|
# with itself, which checks the setup.)
|
|
106
121
|
DUEL_FORCE: ${{ github.event_name == 'workflow_dispatch' && '1' || '' }}
|
|
107
122
|
DUEL_LABELS: ${{ steps.labels.outputs.labels }}
|
|
123
|
+
DUEL_ACCEPTED: ${{ steps.labels.outputs.accepted }}
|
|
108
124
|
run: bundle exec duel --markdown "$RUNNER_TEMP/duel/duel.md"
|
|
109
125
|
# For a pull request, what Duel said, for the Perf comment workflow,
|
|
110
126
|
# which can't see this job's outputs.
|
|
@@ -149,8 +165,11 @@ The workflow compares the following:
|
|
|
149
165
|
the last commit that Duel passed. If that commit isn't in the history,
|
|
150
166
|
such as after a force push or for a branch's first push, Duel reports
|
|
151
167
|
that there's nothing to compare against, and passes.
|
|
152
|
-
- On a schedule, the last release
|
|
153
|
-
|
|
168
|
+
- On a schedule, the last release, or with
|
|
169
|
+
[`c.accept_label`](#accept-a-regression-with-a-label), the last commit
|
|
170
|
+
since then that merged a pull request with the label. Until you tag
|
|
171
|
+
your first release, Duel reports that there's nothing to compare
|
|
172
|
+
against, and passes.
|
|
154
173
|
- When you run the workflow manually, your code against itself. This
|
|
155
174
|
checks that the benchmarks run on the CI machines.
|
|
156
175
|
|
|
@@ -237,12 +256,23 @@ name: Perf comment
|
|
|
237
256
|
# ran), it trusts only what that job could have said: the pull request's
|
|
238
257
|
# number, if it names the pull request the run compared, and the comment
|
|
239
258
|
# and the status, if they're ones Duel says. A fork's report, which the
|
|
240
|
-
# fork's code wrote, is shown as text.
|
|
241
|
-
#
|
|
259
|
+
# fork's code wrote, is shown as text.
|
|
260
|
+
#
|
|
261
|
+
# And when the label that accepts a regression (c.accept_label in
|
|
262
|
+
# .duel.rb) is added to a pull request, or taken off, it runs the Duel
|
|
263
|
+
# job again, which reads the labels as it runs: a run that failed, once
|
|
264
|
+
# the label's added, or one that passed, once it's taken off.
|
|
265
|
+
#
|
|
266
|
+
# (The workflow must be on the default branch before it runs at all.)
|
|
242
267
|
on:
|
|
243
268
|
workflow_run:
|
|
244
269
|
workflows: ["Perf"]
|
|
245
270
|
types: [completed]
|
|
271
|
+
# (This repository's own workflow, as for the comment, with a token
|
|
272
|
+
# that can run a workflow again: it checks nothing out, and runs none
|
|
273
|
+
# of the pull request's code.)
|
|
274
|
+
pull_request_target:
|
|
275
|
+
types: [labeled, unlabeled]
|
|
246
276
|
|
|
247
277
|
permissions: {}
|
|
248
278
|
|
|
@@ -283,12 +313,16 @@ jobs:
|
|
|
283
313
|
# ran the fork's code, which could have written anything in it. In
|
|
284
314
|
# a code block, fenced with more backticks than any run of them in
|
|
285
315
|
# the report, it can't mention anyone, link anywhere or show an
|
|
286
|
-
# image.
|
|
316
|
+
# image. It's cut short where a comment couldn't hold it, and loses
|
|
317
|
+
# any NUL bytes, which GitHub wouldn't take, and grep reads it as
|
|
318
|
+
# text whatever's in it, so it finds every run of backticks.
|
|
287
319
|
report() {
|
|
288
320
|
if [ "${head#* }" = "$GITHUB_REPOSITORY" ]; then cat "$RUNNER_TEMP/report/duel.md" && return; fi
|
|
289
|
-
|
|
321
|
+
text="$RUNNER_TEMP/report.txt"
|
|
322
|
+
LC_ALL=C tr -d '\000' < "$RUNNER_TEMP/report/duel.md" | head -c 60000 > "$text"
|
|
323
|
+
fence=$(LC_ALL=C grep -ao '``*' "$text" | awk '{ if (length > n) n = length } END { n = (n < 3) ? 3 : n + 1; while (n-- > 0) printf "`" }')
|
|
290
324
|
printf 'From a fork, so shown as text:\n\n%s\n' "$fence"
|
|
291
|
-
|
|
325
|
+
cat "$text"
|
|
292
326
|
printf '\n%s\n' "$fence"
|
|
293
327
|
}
|
|
294
328
|
marker="<!-- duel"
|
|
@@ -300,6 +334,33 @@ jobs:
|
|
|
300
334
|
{ echo "$marker status=$STATUS -->"; report; } > "$RUNNER_TEMP/comment.md"
|
|
301
335
|
gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@"$RUNNER_TEMP/comment.md" > /dev/null
|
|
302
336
|
fi
|
|
337
|
+
|
|
338
|
+
again:
|
|
339
|
+
name: Run Duel again
|
|
340
|
+
if: >-
|
|
341
|
+
github.event_name == 'pull_request_target' && github.event.label.name == 'duel: accept'
|
|
342
|
+
runs-on: ubuntu-latest
|
|
343
|
+
permissions:
|
|
344
|
+
actions: write # to run the Perf workflow's jobs again
|
|
345
|
+
steps:
|
|
346
|
+
- name: Run Duel again
|
|
347
|
+
env:
|
|
348
|
+
GH_TOKEN: ${{ github.token }}
|
|
349
|
+
GH_REPO: ${{ github.repository }}
|
|
350
|
+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
|
351
|
+
ACTION: ${{ github.event.action }}
|
|
352
|
+
run: |
|
|
353
|
+
# The newest Perf run of the pull request's head commit, once it's
|
|
354
|
+
# done (one that's running may have read the labels already).
|
|
355
|
+
id=$(gh run list --workflow "Perf" --event pull_request --commit "$HEAD_SHA" --limit 1 \
|
|
356
|
+
--json databaseId --jq '.[0].databaseId // empty')
|
|
357
|
+
[ -n "$id" ] || exit 0
|
|
358
|
+
gh run watch "$id" --interval 30 > /dev/null || true
|
|
359
|
+
conclusion=$(gh run view "$id" --json conclusion --jq .conclusion)
|
|
360
|
+
case "$ACTION:$conclusion" in
|
|
361
|
+
labeled:failure) gh run rerun "$id" --failed ;;
|
|
362
|
+
unlabeled:success) gh run rerun "$id" ;;
|
|
363
|
+
esac
|
|
303
364
|
```
|
|
304
365
|
|
|
305
366
|
## Comment on a pushed commit
|
|
@@ -374,6 +435,23 @@ or for a push, of the pull request that merged it, and passes them to
|
|
|
374
435
|
`pull-requests: read` permission. Without the labels, such as when
|
|
375
436
|
GitHub's API fails, nothing is accepted.
|
|
376
437
|
|
|
438
|
+
The labels are read as the Duel job runs, so `perf-comment.yml` runs
|
|
439
|
+
the job again when the label changes. Its Run Duel again job runs on
|
|
440
|
+
`pull_request_target`, with this repository's own workflow, when a
|
|
441
|
+
label is added to a pull request or taken off. For the label `.duel.rb`
|
|
442
|
+
names, it waits for the Perf run of the pull request's head commit to
|
|
443
|
+
finish. Then it runs the failed jobs again where the label was added
|
|
444
|
+
to a run that failed, or the whole run again where the label was
|
|
445
|
+
taken off one that passed. It checks nothing out and runs none of the
|
|
446
|
+
pull request's code, and its only permission is `actions: write`. The
|
|
447
|
+
workflows name the label as `.duel.rb` did when `duel init` wrote them,
|
|
448
|
+
so if you rename it, rename it in both workflows too.
|
|
449
|
+
|
|
450
|
+
On a schedule, the same step looks for the pull requests merged since
|
|
451
|
+
the last release with the label, and passes `duel` the last commit
|
|
452
|
+
that merged one, in `DUEL_ACCEPTED`. Duel compares with that commit
|
|
453
|
+
instead of the release, as what it slowed down was accepted.
|
|
454
|
+
|
|
377
455
|
## Compare a push with the last commit that passed
|
|
378
456
|
|
|
379
457
|
By default, the workflow compares each push with the commit before it.
|
|
@@ -390,8 +468,10 @@ branch that the workflow guards has its own (see
|
|
|
390
468
|
nothing to compare, Duel sets the `passed` output to `true`, and the
|
|
391
469
|
Mark the commit passed job moves the branch to the push, or creates the
|
|
392
470
|
branch. It never moves the branch back: when two pushes run at once and
|
|
393
|
-
the older one finishes last, the branch stays at the newer push.
|
|
394
|
-
a
|
|
471
|
+
the older one finishes last, the branch stays at the newer push. The
|
|
472
|
+
job runs one at a time for each branch, so two runs can't race to
|
|
473
|
+
create the branch. After a force push rewrites the branch's history,
|
|
474
|
+
it moves the branch anyway.
|
|
395
475
|
That job checks nothing out either, and it gets the `contents: write`
|
|
396
476
|
permission that moving a branch takes. Until the branch exists, and
|
|
397
477
|
when the push's history doesn't have its commit, Duel compares the push
|
|
@@ -438,6 +518,13 @@ yourself:
|
|
|
438
518
|
passed:
|
|
439
519
|
name: Mark the commit passed
|
|
440
520
|
needs: duel
|
|
521
|
+
# One at a time for each branch, so that two runs can't both find no
|
|
522
|
+
# branch and race to make it (a push's run doesn't wait for the one
|
|
523
|
+
# before it). One that's running is never cancelled, and one that's
|
|
524
|
+
# waiting gives way to a newer push's, which moves it further anyway.
|
|
525
|
+
concurrency:
|
|
526
|
+
group: ${{ github.workflow }}-passed-${{ github.ref_name }}
|
|
527
|
+
cancel-in-progress: false
|
|
441
528
|
if: ${{ !cancelled() && github.event_name == 'push' && needs.duel.outputs.passed == 'true' }}
|
|
442
529
|
runs-on: ubuntu-latest
|
|
443
530
|
permissions:
|
|
@@ -461,13 +548,16 @@ yourself:
|
|
|
461
548
|
done
|
|
462
549
|
exit 1
|
|
463
550
|
}
|
|
464
|
-
# (Forward: GitHub moves it only where it's behind the push.
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
gh api --method
|
|
469
|
-
|
|
470
|
-
|
|
551
|
+
# (Forward: GitHub moves it only where it's behind the push. Where
|
|
552
|
+
# there's no branch, and making it fails, it looks once more, as
|
|
553
|
+
# something else may have made it: a push of it, say.)
|
|
554
|
+
for _ in 1 2; do
|
|
555
|
+
gh api --method PATCH "repos/$GITHUB_REPOSITORY/git/refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null 2>&1 && exit 0
|
|
556
|
+
at=$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/$BRANCH" --jq .object.sha 2>/dev/null) || at=
|
|
557
|
+
[ -n "$at" ] && break
|
|
558
|
+
gh api --method POST "repos/$GITHUB_REPOSITORY/git/refs" -f ref="refs/heads/$BRANCH" -f sha="$GITHUB_SHA" > /dev/null && exit 0
|
|
559
|
+
done
|
|
560
|
+
[ -n "$at" ] || blocked
|
|
471
561
|
compared=$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$at" --jq .status 2>&1) || case "$compared" in
|
|
472
562
|
# (GitHub can't compare commits with no history in common.)
|
|
473
563
|
*"No common ancestor"*) compared=unrelated ;;
|
|
@@ -494,16 +584,19 @@ your README can show it:
|
|
|
494
584
|
```
|
|
495
585
|
|
|
496
586
|
The badge says "perf guarded", in green, when Duel passed the push. It
|
|
497
|
-
says "perf regressed", in red, when
|
|
587
|
+
says "perf regressed", in red, when Duel failed the push or found a
|
|
498
588
|
regression, even one that passed, with
|
|
499
589
|
[`c.fail_on = :pull_requests`](#comment-on-a-pushed-commit), or that a
|
|
500
|
-
label accepted.
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
590
|
+
label accepted. It says "not measured", in grey, when the Duel job
|
|
591
|
+
failed before Duel could judge the push, as when its runner ran out of
|
|
592
|
+
memory or a step before Duel's failed. A job that fails after Duel
|
|
593
|
+
passed the push leaves it green. A push with nothing to compare leaves
|
|
594
|
+
the badge as it was, and so does a run that finishes after a newer
|
|
595
|
+
push's, so the badge is always the newest push's that Duel compared.
|
|
596
|
+
The branch holds one commit, with the badge alone, whose message names
|
|
597
|
+
that push. The job checks nothing out, and it gets the `contents: write`
|
|
598
|
+
permission that moving a branch takes. For a project in a subdirectory
|
|
599
|
+
of its repository, the branch names the directory, such as
|
|
507
600
|
`duel/badge-gems-parser`.
|
|
508
601
|
|
|
509
602
|
Until the first push to your default branch after you add the job, the
|
|
@@ -517,13 +610,15 @@ keeps `.duel.rb`), or add the job yourself:
|
|
|
517
610
|
|
|
518
611
|
```yaml
|
|
519
612
|
# .github/workflows/perf.yml, in jobs: (with duel init --badge)
|
|
520
|
-
# Duel's badge, for the README: green (perf guarded)
|
|
521
|
-
# regressed), as the last push to the default
|
|
522
|
-
# left it. It's red where
|
|
523
|
-
# even one that passed (with
|
|
524
|
-
#
|
|
525
|
-
#
|
|
526
|
-
# to
|
|
613
|
+
# Duel's badge, for the README: green (perf guarded), red (perf
|
|
614
|
+
# regressed) or grey (not measured), as the last push to the default
|
|
615
|
+
# branch that Duel compared left it. It's red where Duel failed the push
|
|
616
|
+
# or found a regression, even one that passed (with
|
|
617
|
+
# c.fail_on = :pull_requests in .duel.rb), and grey where the Duel job
|
|
618
|
+
# failed before Duel could judge the push (its runner ran out of memory,
|
|
619
|
+
# say). A push with nothing to compare leaves it as it was, and so does
|
|
620
|
+
# a run for an older push than the badge's. This job writes the badge,
|
|
621
|
+
# alone, to a branch of its own, which the README shows:
|
|
527
622
|
#
|
|
528
623
|
# [](https://github.com/sferik/duel-ruby)
|
|
529
624
|
#
|
|
@@ -569,10 +664,13 @@ keeps `.duel.rb`), or add the job yourself:
|
|
|
569
664
|
last=$(gh api "repos/$GITHUB_REPOSITORY/commits/$BRANCH" --jq .commit.message 2>/dev/null | sed -n 's/^Duel.s badge, for \([0-9a-f]*\)$/\1/p') || true
|
|
570
665
|
if [ -n "$last" ] && [ "$(gh api "repos/$GITHUB_REPOSITORY/compare/$GITHUB_SHA...$last" --jq .status 2>/dev/null)" = ahead ]; then exit 0; fi
|
|
571
666
|
# (Its image, written here rather than fetched from anywhere: green,
|
|
572
|
-
#
|
|
667
|
+
# red where Duel failed the push or found a regression, or grey
|
|
668
|
+
# where the Duel job failed before Duel said either.)
|
|
573
669
|
state="perf guarded" width=133 top="#34D058" bottom="#28A745" right="76.939C78.629 0 80 1.343 80"
|
|
574
|
-
if [ "$
|
|
670
|
+
if [ "$STATUS" = failed ]; then
|
|
575
671
|
state="perf regressed" width=142 top="#EA4A5A" bottom="#D73A49" right="85.939C87.629 0 89 1.343 89"
|
|
672
|
+
elif [ "$RESULT" = failure ] && [ "$STATUS" != passed ]; then
|
|
673
|
+
state="not measured" width=138 top="#959DA5" bottom="#6A737D" right="81.939C83.629 0 85 1.343 85"
|
|
576
674
|
fi
|
|
577
675
|
cat > "$RUNNER_TEMP/badge.svg" <<SVG
|
|
578
676
|
<svg xmlns="http://www.w3.org/2000/svg" width="$width" height="20">
|
|
@@ -825,6 +923,9 @@ duel:passed:
|
|
|
825
923
|
image: alpine:3
|
|
826
924
|
stage: .post
|
|
827
925
|
needs: [duel]
|
|
926
|
+
# One at a time, as each deletes the branch and makes it again: two at
|
|
927
|
+
# once could both delete it, and the second to make it would fail.
|
|
928
|
+
resource_group: duel/passed
|
|
828
929
|
rules:
|
|
829
930
|
- if: $DUEL_TOKEN && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
|
|
830
931
|
variables:
|
|
@@ -894,7 +995,8 @@ match. `duel init` adds these steps for a project that has a
|
|
|
894
995
|
# request targets (or run by hand, the main branch), a merge queue's
|
|
895
996
|
# base, the commit before a push (or the last commit Duel passed, PASSED, if there is
|
|
896
997
|
# one: see the Mark the commit passed job), or on a schedule the last
|
|
897
|
-
# release, the newest tag before HEAD that RELEASE_TAGS matches
|
|
998
|
+
# release, the newest tag before HEAD that RELEASE_TAGS matches
|
|
999
|
+
# (or ACCEPTED, the last change accepted since then). (If
|
|
898
1000
|
# .duel.rb sets c.main_branch or c.release_tags, set MAIN_BRANCH or
|
|
899
1001
|
# RELEASE_TAGS to match.)
|
|
900
1002
|
- name: Find the baseline's Ruby
|
|
@@ -905,13 +1007,14 @@ match. `duel init` adds these steps for a project that has a
|
|
|
905
1007
|
PASSED: origin/duel/passed/${{ github.ref_name }}
|
|
906
1008
|
MAIN_BRANCH: origin/${{ github.base_ref || github.event.repository.default_branch }}
|
|
907
1009
|
RELEASE_TAGS: "v*"
|
|
1010
|
+
ACCEPTED: ${{ steps.labels.outputs.accepted }}
|
|
908
1011
|
run: |
|
|
909
1012
|
case "$GITHUB_EVENT_NAME" in
|
|
910
1013
|
push)
|
|
911
1014
|
base=$BEFORE
|
|
912
1015
|
if git merge-base --is-ancestor "$PASSED" HEAD 2>/dev/null; then base=$PASSED; fi ;;
|
|
913
1016
|
merge_group) base=$QUEUE_BASE ;;
|
|
914
|
-
schedule) base=$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null) || base= ;;
|
|
1017
|
+
schedule) base=${ACCEPTED:-$(git describe --tags --abbrev=0 --match "$RELEASE_TAGS" HEAD^ 2>/dev/null)} || base= ;;
|
|
915
1018
|
*) base=$(git merge-base HEAD "$MAIN_BRANCH" 2>/dev/null) || base= ;;
|
|
916
1019
|
esac
|
|
917
1020
|
ruby_of() { git show "$1:./.ruby-version" 2>/dev/null | awk 'NF { print $1; exit }'; }
|
data/docs/configuration.md
CHANGED
|
@@ -140,12 +140,16 @@ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
|
|
|
140
140
|
that `duel init` writes passes Duel the pull request's labels. With
|
|
141
141
|
that label, a regression passes, with a warning, and the report says
|
|
142
142
|
that the label accepted it. The labels are read when the workflow runs,
|
|
143
|
-
so
|
|
144
|
-
pull request
|
|
145
|
-
|
|
143
|
+
so the comment workflow runs the Duel job again when you add the label
|
|
144
|
+
to a pull request whose run failed, or take it off one whose run
|
|
145
|
+
passed. For a push, Duel reads the labels of the pull request that
|
|
146
|
+
merged it, so with
|
|
146
147
|
[`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
|
|
147
148
|
the merged regression becomes the commit that later pushes are compared
|
|
148
|
-
with. Duel
|
|
149
|
+
with. On a schedule, Duel compares with the last commit since your last
|
|
150
|
+
release that merged a pull request with the label, instead of the
|
|
151
|
+
release, so a slowdown you accepted doesn't fail every night until
|
|
152
|
+
your next release. Duel's default is no such label, but the `.duel.rb` that
|
|
149
153
|
`duel init` writes (and duel-rails's generator) names `duel: accept`.
|
|
150
154
|
Create the label in your repository to use it, or set
|
|
151
155
|
`c.accept_label = nil` so that nothing can let a regression through.
|