duel 0.1.1 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +134 -1
- data/README.md +42 -7
- data/docs/benchmarks.md +2 -1
- data/docs/ci.md +630 -61
- data/docs/configuration.md +91 -4
- data/docs/how-it-works.md +34 -5
- data/docs/json.md +2 -0
- data/docs/plugins.md +2 -2
- data/docs/troubleshooting.md +14 -0
- data/lib/duel/baseline.rb +34 -8
- data/lib/duel/cli.rb +174 -29
- data/lib/duel/config.rb +84 -6
- data/lib/duel/context.rb +6 -5
- data/lib/duel/init.rb +45 -7
- data/lib/duel/json_report.rb +4 -3
- data/lib/duel/lock.rb +52 -8
- data/lib/duel/native.rb +20 -0
- data/lib/duel/resource.rb +37 -2
- data/lib/duel/runner.rb +98 -9
- data/lib/duel/summary.rb +8 -3
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +167 -0
- data/lib/duel/templates/duel.rb.erb +13 -0
- data/lib/duel/templates/perf-comment.yml.erb +59 -0
- data/lib/duel/templates/perf.yml.erb +196 -46
- data/lib/duel/testing.rb +2 -2
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +1 -1
- data/sig/duel/baseline.rbs +14 -1
- data/sig/duel/cli.rbs +64 -2
- data/sig/duel/config.rbs +39 -0
- data/sig/duel/init.rbs +16 -0
- data/sig/duel/lock.rbs +11 -0
- data/sig/duel/native.rbs +2 -0
- data/sig/duel/resource.rbs +10 -1
- data/sig/duel/runner.rbs +50 -3
- data/sig/duel/summary.rbs +6 -1
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3827f9485e70bb95c50b32ff9438042bd60cb26885d92f01fe6c1dc6de33ee72
|
|
4
|
+
data.tar.gz: 53e410c89687f6bcdec3c46be4d370409110ac134d90ef74ca8aaaad62394a7e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7710b0ab9cc84821d6e66af1d66e1005b98aa0e97a1f047e2a76d6c5d3a83defb413895e11476d067265b97d589dcbc2fb5bdf78550616da74cf92df0da2bd19
|
|
7
|
+
data.tar.gz: 6d9d0df5accaf420f0a4e554f21eb22e79e77f8865986af856759272c5358b6cacd22af27800827d6faf7360daeded8d58481a6ef3c22669279d13a85fe4ff83
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,137 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.1.3] - 2026-11-04
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- The workflow that `duel init` writes keeps a badge for the README
|
|
15
|
+
that turns red when Duel fails a push to the default branch. Its
|
|
16
|
+
Update the badge job writes `badge.svg` to a branch of its own,
|
|
17
|
+
`duel/badge`, green ("perf guarded") when Duel passed the push, and
|
|
18
|
+
red ("perf regressed") when the Duel job failed or found a
|
|
19
|
+
regression, even one that passed.
|
|
20
|
+
- A run measures for 30 minutes at most (`--time-limit SECONDS`), not
|
|
21
|
+
counting the builds. Past that, it starts no more measurements: a
|
|
22
|
+
benchmark it hasn't measured fails as an error, and one that looked
|
|
23
|
+
worse is judged on the measurements it has, so a run cut short never
|
|
24
|
+
passes a regression. The report (and the JSON report's `cut_short`)
|
|
25
|
+
says what it left undone. A benchmark that starts processes of its own
|
|
26
|
+
and sat near its limit could be measured again for hours before.
|
|
27
|
+
|
|
28
|
+
### Changed
|
|
29
|
+
|
|
30
|
+
- The workers, their directories and sockets, and the resources that
|
|
31
|
+
prepare blocks get are named `base` and `head` (`base-1`, `head-1`),
|
|
32
|
+
not `baseline` and `candidate`: `Resource#name` (`w.name`, `v.name`)
|
|
33
|
+
says so, and `Duel::Testing.prepare` names its worker `head-1` by
|
|
34
|
+
default. The report still says baseline and candidate.
|
|
35
|
+
- A run removes its lock (`tmp/duel/lock`) when it ends, and
|
|
36
|
+
`tmp/duel` (and `tmp`, if Duel made it) when that leaves them empty, so
|
|
37
|
+
a project whose plugins build nothing to keep has nothing left there.
|
|
38
|
+
- When a run ends, the builds that prepare blocks made once
|
|
39
|
+
(`Resource#cached`) and the run used replace the others of their name:
|
|
40
|
+
a key's name is what's before its version and the hash it ends with
|
|
41
|
+
(`mysql`, of `mysql-0.1.2-3f9c2a1b7e5d4c6a`). A database server or a
|
|
42
|
+
plugin upgraded left its old build in `tmp/duel/cache` before, tens
|
|
43
|
+
of megabytes (82 MB for MySQL's) each time.
|
|
44
|
+
|
|
45
|
+
### Fixed
|
|
46
|
+
|
|
47
|
+
- The candidate's workers could measure slower in every pair, with the
|
|
48
|
+
same instruction counts, on code that hadn't changed. Their names
|
|
49
|
+
(`candidate-1`, against `baseline-1`) were a byte longer, and so were
|
|
50
|
+
the paths that a plugin's servers put in their environment
|
|
51
|
+
(duel-postgresql's `DATABASE_URL`, `PGHOST` and
|
|
52
|
+
`DUEL_POSTGRESQL_URL`, say). Where a process's stack starts depends on
|
|
53
|
+
how long its arguments and environment are, and at some lengths, a
|
|
54
|
+
tight loop runs the same instructions in more cycles: on macOS,
|
|
55
|
+
server-free benchmarks of pg came out 1.2 to 1.55 times slower on the
|
|
56
|
+
candidate with the plugin, and the same as the baseline without it.
|
|
57
|
+
The two versions' names are now as long as each other.
|
|
58
|
+
|
|
59
|
+
- The workflow that `duel init` writes (and its GitLab CI jobs) no
|
|
60
|
+
longer moves the branch that marks the last commit Duel passed
|
|
61
|
+
backward. When two pushes ran at once and the older one finished
|
|
62
|
+
last, the branch went back to the older push, so the next push was
|
|
63
|
+
compared with it, and a regression already accepted counted again.
|
|
64
|
+
Now the branch only moves forward, or after a force push rewrites
|
|
65
|
+
its history.
|
|
66
|
+
- Where a branch named `duel` (or `duel/passed`) keeps the workflow that
|
|
67
|
+
`duel init` writes from making `duel/passed/main` or `duel/badge`, as
|
|
68
|
+
Git can't have both, the job's error now names that branch and says to
|
|
69
|
+
delete it, instead of GitHub's bare `Reference update failed`.
|
|
70
|
+
- On JRuby, peak memory (`peak_memory: true`) is the most of the JVM's
|
|
71
|
+
heap that a benchmark had in use, not resident memory, which barely
|
|
72
|
+
moves there, as the JVM takes memory for its heap ahead of time: a
|
|
73
|
+
benchmark that needed 24 MB more could pass. JRuby now has peak
|
|
74
|
+
memory on macOS too.
|
|
75
|
+
|
|
76
|
+
## [0.1.2] - 2026-11-02
|
|
77
|
+
|
|
78
|
+
### Added
|
|
79
|
+
|
|
80
|
+
- `c.fail_on = :pull_requests` (or `duel --fail-on pull_requests`) fails
|
|
81
|
+
a regression only in a pull request (or a merge queue), and outside
|
|
82
|
+
GitHub Actions. On GitHub Actions, a regression in a push, the
|
|
83
|
+
schedule, or a run by hand passes, with a warning, and the workflow
|
|
84
|
+
that `duel init` writes comments on a pushed commit with the report,
|
|
85
|
+
as `c.comment` says, which notifies the commit's author. The default,
|
|
86
|
+
`:every_run`, fails every run, as before, and comments on no commit.
|
|
87
|
+
- `c.push_base = :last_passed` compares a push with the last commit
|
|
88
|
+
that Duel passed, which a branch marks (`duel/passed/main` for
|
|
89
|
+
`main`, and each branch the workflow guards has its own), instead of
|
|
90
|
+
the commit before the push, so slowdowns too small for any one push to
|
|
91
|
+
fail add up until they fail one. The workflow that `duel init` writes
|
|
92
|
+
moves the branch when a push passes. The default, `:before`, compares
|
|
93
|
+
a push with the commit before it, as before.
|
|
94
|
+
- `c.accept_label` names a pull request label that accepts a regression:
|
|
95
|
+
with it, the regression passes, with a warning, and the report says
|
|
96
|
+
so. The workflow that `duel init` writes passes the pull request's
|
|
97
|
+
labels to `duel` (`DUEL_LABELS`), or for a push, the labels of the
|
|
98
|
+
pull request that merged it. By default, there's no such label.
|
|
99
|
+
- `duel init --gitlab` writes GitLab CI jobs to include from
|
|
100
|
+
`.gitlab-ci.yml`, instead of GitHub Actions workflows. They compare a
|
|
101
|
+
merge request with the branch it targets, comment on it, and with
|
|
102
|
+
project access tokens, comment on a pushed commit and mark a push
|
|
103
|
+
passed. On GitLab CI, `duel` reads a merge request's target and labels,
|
|
104
|
+
writes its decisions to `DUEL_OUTPUT` as variables, and where only a
|
|
105
|
+
merge request fails (`c.fail_on`), exits with status 3 for a
|
|
106
|
+
regression outside one, which the job lets fail, as a warning.
|
|
107
|
+
- The workflow that `duel init` writes runs for a merge queue
|
|
108
|
+
(`merge_group`), and compares its group with the queue's base, so the
|
|
109
|
+
Duel check can be required with a merge queue.
|
|
110
|
+
|
|
111
|
+
### Changed
|
|
112
|
+
|
|
113
|
+
- To upgrade a project that ran `duel init` before 0.1.2, delete
|
|
114
|
+
`.github/workflows/perf.yml` (or `perf-<its path>.yml`, for a project
|
|
115
|
+
in a subdirectory) and run `duel init` again, then carry over any
|
|
116
|
+
edits of your own. It writes both workflows, and skips the files that
|
|
117
|
+
exist, such as `.duel.rb`. Kept, the old workflow comments on a pull
|
|
118
|
+
request itself, and with the new one, every comment comes twice.
|
|
119
|
+
- A pull request gets its comment from a second workflow that `duel init`
|
|
120
|
+
writes, `.github/workflows/perf-comment.yml`, which runs once the Perf
|
|
121
|
+
workflow has (`workflow_run`), with a token that can write, so a pull
|
|
122
|
+
request from a fork gets a comment too. It runs none of the pull
|
|
123
|
+
request's code, and trusts only what Duel could have said in the
|
|
124
|
+
report's artifact. The Perf workflow no longer has a Comment job, and
|
|
125
|
+
for a project in a subdirectory, it's named for the directory, as
|
|
126
|
+
`Perf (gems/parser)`.
|
|
127
|
+
- The workflow that `duel init` writes cancels a pull request's run
|
|
128
|
+
when a newer push to the pull request starts one. Every other run,
|
|
129
|
+
such as a push's to `main`, still runs to the end.
|
|
130
|
+
- A push whose commit before it isn't in the history now reports that
|
|
131
|
+
there's no commit to compare against, and passes, from `duel` itself.
|
|
132
|
+
The workflow no longer checks this in its Compare step.
|
|
133
|
+
|
|
134
|
+
### Fixed
|
|
135
|
+
|
|
136
|
+
- On GitHub Actions, a pull request is compared with the branch that it
|
|
137
|
+
targets (`GITHUB_BASE_REF`), not the main branch. A pull request into
|
|
138
|
+
a stable branch, or into another pull request's branch, was compared
|
|
139
|
+
with `main`.
|
|
140
|
+
|
|
10
141
|
## [0.1.1] - 2026-10-26
|
|
11
142
|
|
|
12
143
|
### Added
|
|
@@ -357,6 +488,8 @@ needs Ruby 3.4 or later (or JRuby 10, or TruffleRuby 40).
|
|
|
357
488
|
Anything marked `@api private`, in lib/ and sig/, may change in any
|
|
358
489
|
release.
|
|
359
490
|
|
|
360
|
-
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.
|
|
491
|
+
[Unreleased]: https://github.com/sferik/duel-ruby/compare/v0.1.3...HEAD
|
|
492
|
+
[0.1.3]: https://github.com/sferik/duel-ruby/compare/v0.1.2...v0.1.3
|
|
493
|
+
[0.1.2]: https://github.com/sferik/duel-ruby/compare/v0.1.1...v0.1.2
|
|
361
494
|
[0.1.1]: https://github.com/sferik/duel-ruby/compare/v0.1.0...v0.1.1
|
|
362
495
|
[0.1.0]: https://github.com/sferik/duel-ruby/releases/tag/v0.1.0
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# ⚔️ Duel: Performance regression guards for Ruby
|
|
2
2
|
|
|
3
|
-
[](https://github.com/sferik/duel-ruby/actions/workflows/perf.yml)
|
|
4
4
|
[](https://github.com/sferik/duel-ruby/actions/workflows/test.yml)
|
|
5
5
|
[](https://github.com/sferik/duel-ruby/actions/workflows/lint.yml)
|
|
6
6
|
[](https://github.com/sferik/duel-ruby/actions/workflows/mutant.yml)
|
|
@@ -194,6 +194,16 @@ hang the run. If a worker takes more than a minute to load or to answer,
|
|
|
194
194
|
Duel stops the worker and reports the benchmark as an error. To change
|
|
195
195
|
the limit, use `--timeout`.
|
|
196
196
|
|
|
197
|
+
A run measures for 30 minutes at most, not counting the builds. Past
|
|
198
|
+
that, Duel starts no more measurements. A benchmark that it hasn't
|
|
199
|
+
measured yet fails as an error, and a benchmark that looked worse, but
|
|
200
|
+
that Duel hasn't measured again as often as it usually does, fails if it
|
|
201
|
+
was worse in every measurement so far. So a run that's cut short never
|
|
202
|
+
passes a regression. The report says what was left undone. To change
|
|
203
|
+
the limit, use `--time-limit SECONDS`. A run usually takes this long only
|
|
204
|
+
when some benchmark is slow to settle, such as one that starts processes
|
|
205
|
+
of its own, so making it quicker is the better fix.
|
|
206
|
+
|
|
197
207
|
`duel` exits with one of the following statuses:
|
|
198
208
|
|
|
199
209
|
| Status | Meaning |
|
|
@@ -201,6 +211,7 @@ the limit, use `--timeout`.
|
|
|
201
211
|
| `0` | Nothing got slower, or there was nothing to compare. |
|
|
202
212
|
| `1` | Something got slower, or a benchmark failed. |
|
|
203
213
|
| `2` | Duel couldn't compare. |
|
|
214
|
+
| `3` | Something got slower outside a merge request on GitLab CI, where only a merge request fails ([`c.fail_on`](docs/configuration.md#fail-only-a-pull-request)). The job that `duel init --gitlab` writes lets it fail, as a warning. |
|
|
204
215
|
|
|
205
216
|
## Run Duel in CI
|
|
206
217
|
|
|
@@ -215,6 +226,10 @@ duel:
|
|
|
215
226
|
GIT_DEPTH: 0 # the baseline comes from history
|
|
216
227
|
```
|
|
217
228
|
|
|
229
|
+
`duel init --gitlab` writes GitLab CI jobs for you, which compare a
|
|
230
|
+
merge request with the branch it targets and comment on it. See
|
|
231
|
+
[Run Duel on GitLab CI](docs/ci.md#run-duel-on-gitlab-ci).
|
|
232
|
+
|
|
218
233
|
To run Duel from Rake, define a task with `Duel::RakeTask`. Any
|
|
219
234
|
arguments after the task's name are passed to `duel`:
|
|
220
235
|
|
|
@@ -225,8 +240,10 @@ Duel::RakeTask.new(:perf_since_release, "--base", "last-release")
|
|
|
225
240
|
```
|
|
226
241
|
|
|
227
242
|
`duel init` writes a GitHub Actions workflow,
|
|
228
|
-
`.github/workflows/perf.yml
|
|
229
|
-
|
|
243
|
+
`.github/workflows/perf.yml`, and a second one,
|
|
244
|
+
`.github/workflows/perf-comment.yml`, that comments on a pull request,
|
|
245
|
+
a fork's too. It compares each pull request with the
|
|
246
|
+
branch that it targets, each push to `main` with the commit before it, and, on a
|
|
230
247
|
schedule, your last release, to catch slow creep. It also comments on a
|
|
231
248
|
pull request with Duel's report when a benchmark fails. For the
|
|
232
249
|
workflow, and for a branch that upgrades Ruby, see
|
|
@@ -304,17 +321,35 @@ but nothing to compare, or never, set
|
|
|
304
321
|
in `.duel.rb`. For more information, see
|
|
305
322
|
[Comment on a pull request](docs/ci.md#comment-on-a-pull-request).
|
|
306
323
|
|
|
324
|
+
In a project where people push to `main` directly, set
|
|
325
|
+
[`c.fail_on = :pull_requests`](docs/configuration.md#fail-only-a-pull-request)
|
|
326
|
+
to pass a regression in a push, with a warning, and comment on the
|
|
327
|
+
pushed commit with the report instead. To compare each push with the
|
|
328
|
+
last commit that passed, instead of the commit before it, so that small
|
|
329
|
+
slowdowns add up until they fail a push, set
|
|
330
|
+
[`c.push_base = :last_passed`](docs/configuration.md#compare-a-push-with-the-last-commit-that-passed).
|
|
331
|
+
To accept a regression that's on purpose, label the pull request, as
|
|
332
|
+
[`c.accept_label`](docs/configuration.md#accept-a-regression-with-a-label)
|
|
333
|
+
says.
|
|
334
|
+
|
|
307
335
|
### Show that a project is guarded
|
|
308
336
|
|
|
309
337
|
To show that Duel guards your project, add its badge to your README,
|
|
310
|
-
next to your other badges
|
|
338
|
+
next to your other badges. With the workflow that `duel init` writes,
|
|
339
|
+
the badge turns red when Duel fails a push to your default branch.
|
|
340
|
+
Replace `OWNER/REPO` with your repository's:
|
|
311
341
|
|
|
312
342
|
```markdown
|
|
313
|
-
[](https://github.com/sferik/duel-ruby)
|
|
314
344
|
```
|
|
315
345
|
|
|
316
|
-
|
|
317
|
-
|
|
346
|
+
See [Show whether the last push passed](docs/ci.md#show-whether-the-last-push-passed)
|
|
347
|
+
for how the workflow keeps it. Without the workflow, such as on GitLab
|
|
348
|
+
CI, use Duel's own badge, which is always green:
|
|
349
|
+
|
|
350
|
+
```markdown
|
|
351
|
+
[](https://github.com/sferik/duel-ruby)
|
|
352
|
+
```
|
|
318
353
|
|
|
319
354
|
## Plugins
|
|
320
355
|
|
data/docs/benchmarks.md
CHANGED
|
@@ -217,7 +217,8 @@ To change that, pass these options to `duel`:
|
|
|
217
217
|
benchmark thread's.
|
|
218
218
|
- `peak_memory: true`: also judges the benchmark's peak memory, which is
|
|
219
219
|
the most resident memory that its measured work reached, above where
|
|
220
|
-
it started
|
|
220
|
+
it started, or on JRuby, the most of the JVM's heap that it had in
|
|
221
|
+
use. Duel takes each version's lowest peak from three fresh
|
|
221
222
|
workers. Like every other measure, peak memory leaves out what
|
|
222
223
|
`once` and `untimed` blocks leave behind.
|
|
223
224
|
|