duel 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENT_PROMPT.md +51 -27
- data/CHANGELOG.md +247 -1
- data/README.md +30 -14
- data/docs/benchmarks.md +28 -24
- data/docs/ci.md +328 -362
- data/docs/configuration.md +40 -37
- data/docs/how-it-works.md +23 -10
- data/docs/json.md +21 -25
- data/docs/plugins.md +40 -22
- data/docs/troubleshooting.md +3 -2
- data/lib/duel/across.rb +18 -18
- data/lib/duel/badge.rb +174 -0
- data/lib/duel/baseline.rb +38 -15
- data/lib/duel/cli.rb +175 -107
- data/lib/duel/comment.rb +80 -71
- data/lib/duel/config.rb +145 -63
- data/lib/duel/context.rb +29 -24
- data/lib/duel/forge.rb +87 -0
- data/lib/duel/init.rb +44 -87
- data/lib/duel/json_report.rb +34 -25
- data/lib/duel/lane.rb +107 -61
- data/lib/duel/lock.rb +1 -1
- data/lib/duel/mark_passed.rb +132 -0
- data/lib/duel/metric.rb +28 -27
- data/lib/duel/native.rb +7 -5
- data/lib/duel/platform.rb +4 -0
- data/lib/duel/process_sample.rb +9 -6
- data/lib/duel/rake_task.rb +4 -5
- data/lib/duel/resource.rb +10 -11
- data/lib/duel/result.rb +22 -14
- data/lib/duel/rubies.rb +6 -2
- data/lib/duel/runner.rb +27 -15
- data/lib/duel/spawn.rb +29 -4
- data/lib/duel/stats.rb +14 -12
- data/lib/duel/summary.rb +31 -29
- data/lib/duel/templates/benchmark_duel.rb.erb +2 -2
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +18 -54
- data/lib/duel/templates/duel.rb.erb +7 -7
- data/lib/duel/templates/perf-caller.yml.erb +22 -5
- data/lib/duel/templates/perf-comment.yml.erb +11 -11
- data/lib/duel/templates/perf-push-jobs.yml.erb +127 -0
- data/lib/duel/templates/perf-push.yml.erb +50 -0
- data/lib/duel/templates/perf.yml.erb +113 -335
- data/lib/duel/testing.rb +20 -25
- data/lib/duel/version.rb +1 -1
- data/lib/duel/worker.rb +16 -9
- data/lib/duel/worker_process.rb +6 -6
- data/lib/duel.rb +88 -33
- data/sig/duel/across.rbs +4 -0
- data/sig/duel/badge.rbs +51 -0
- data/sig/duel/baseline.rbs +7 -2
- data/sig/duel/cli.rbs +31 -20
- data/sig/duel/comment.rbs +14 -15
- data/sig/duel/config.rbs +71 -29
- data/sig/duel/context.rbs +4 -4
- data/sig/duel/forge.rbs +42 -0
- data/sig/duel/init.rbs +8 -26
- data/sig/duel/json_report.rbs +7 -6
- data/sig/duel/lane.rbs +29 -6
- data/sig/duel/mark_passed.rbs +39 -0
- data/sig/duel/metric.rbs +10 -10
- data/sig/duel/native.rbs +3 -0
- data/sig/duel/platform.rbs +1 -0
- data/sig/duel/process_sample.rbs +3 -0
- data/sig/duel/resource.rbs +3 -3
- data/sig/duel/result.rbs +12 -5
- data/sig/duel/rubies.rbs +3 -0
- data/sig/duel/runner.rbs +23 -16
- data/sig/duel/spawn.rbs +7 -0
- data/sig/duel/stats.rbs +4 -1
- data/sig/duel/summary.rbs +8 -0
- data/sig/duel/testing.rbs +2 -2
- data/sig/duel/worker.rbs +6 -5
- data/sig/duel/worker_process.rbs +3 -3
- data/sig/duel.rbs +50 -31
- metadata +9 -1
data/docs/configuration.md
CHANGED
|
@@ -9,22 +9,23 @@ runner and every worker load it:
|
|
|
9
9
|
|
|
10
10
|
```ruby
|
|
11
11
|
Duel.configure do |c|
|
|
12
|
-
c.benchmarks = "perf/**/*_bench.rb" # default: duel/**/*_duel.rb
|
|
13
|
-
c.watch = %w[lib ext] # the code a change must touch to be compared
|
|
12
|
+
c.benchmarks = "perf/**/*_bench.rb" # or a list of globs (default: duel/**/*_duel.rb)
|
|
13
|
+
c.watch = %w[lib ext] # the code a change must touch to be compared (a path, or a list)
|
|
14
14
|
c.library = "my_gem" # required from each version (default: the gemspec's)
|
|
15
15
|
c.build = "make" # for both versions (default: rake compile, with ext/)
|
|
16
16
|
c.main_branch = "origin/trunk"
|
|
17
|
-
c.release_tags =
|
|
17
|
+
c.release_tags = %w[v* release-*] # a glob, or a list, any of which a tag matches (default: v*)
|
|
18
18
|
c.ruby = :version_file # each version's own Ruby (default: the runner's)
|
|
19
19
|
c.comment = :results # when CI comments on a pull request (default: :failures)
|
|
20
20
|
c.fail_on = :pull_requests # which runs a regression fails (default: :every_run)
|
|
21
21
|
c.push_base = :last_passed # what CI compares a push with (default: :before)
|
|
22
|
-
c.accept_label = "
|
|
22
|
+
c.accept_label = "perf: accept" # a pull request label that accepts a regression (default: "duel: accept")
|
|
23
23
|
c.limit = 1.05 # a run's settings, as their options (see Set a run's settings)
|
|
24
24
|
c.use :redis # a plugin (see Use plugins)
|
|
25
25
|
|
|
26
|
-
#
|
|
27
|
-
|
|
26
|
+
# Or, instead of a library to require, how a worker loads one version
|
|
27
|
+
# of your code, from the root of its tree:
|
|
28
|
+
# c.library = ->(root) { require File.join(root, "config", "boot") }
|
|
28
29
|
end
|
|
29
30
|
```
|
|
30
31
|
|
|
@@ -42,7 +43,8 @@ nothing to compare, and Duel passes without running the benchmarks.
|
|
|
42
43
|
watches what decides how your code is measured:
|
|
43
44
|
|
|
44
45
|
- The benchmarks' directory: the part of `c.benchmarks` before its first
|
|
45
|
-
wildcard, `duel/` by default, with its
|
|
46
|
+
wildcard (each glob's, for a list), `duel/` by default, with its
|
|
47
|
+
helpers and fixtures.
|
|
46
48
|
- The configuration's file, `.duel.rb`.
|
|
47
49
|
- With [`c.ruby = :version_file`](#run-each-version-on-its-own-ruby),
|
|
48
50
|
the version files at your project's root (`.ruby-version`,
|
|
@@ -64,7 +66,7 @@ end
|
|
|
64
66
|
|
|
65
67
|
The [GitHub Actions workflow](ci.md#comment-on-a-pull-request) that
|
|
66
68
|
`duel init` writes comments on a pull request with Duel's report.
|
|
67
|
-
`c.comment` chooses when
|
|
69
|
+
`c.comment` chooses when:
|
|
68
70
|
|
|
69
71
|
| Value | When the workflow comments |
|
|
70
72
|
|---|---|
|
|
@@ -99,8 +101,7 @@ says which values it takes.
|
|
|
99
101
|
By default, a regression fails every run: a pull request's, a push's,
|
|
100
102
|
and the schedule's. In a project where people push to `main` directly,
|
|
101
103
|
a failed push only gets GitHub's email that the run failed. To report a
|
|
102
|
-
regression in a push without failing it, set `c.fail_on
|
|
103
|
-
`duel --fail-on` overrides it:
|
|
104
|
+
regression in a push without failing it, set `c.fail_on`:
|
|
104
105
|
|
|
105
106
|
| Value | Which runs a regression fails |
|
|
106
107
|
|---|---|
|
|
@@ -128,12 +129,13 @@ always fails.
|
|
|
128
129
|
## Accept a regression with a label
|
|
129
130
|
|
|
130
131
|
Sometimes a change is slower on purpose: it's correct where the old
|
|
131
|
-
code wasn't, say. To let a pull request through anyway,
|
|
132
|
-
that accepts its regression
|
|
132
|
+
code wasn't, say. To let a pull request through anyway, label it
|
|
133
|
+
`duel: accept`, the label that accepts its regression by default. To
|
|
134
|
+
name another, or none:
|
|
133
135
|
|
|
134
136
|
```ruby
|
|
135
137
|
Duel.configure do |c|
|
|
136
|
-
c.accept_label = "
|
|
138
|
+
c.accept_label = "perf: accept" # or nil, so that nothing can let a regression through
|
|
137
139
|
end
|
|
138
140
|
```
|
|
139
141
|
|
|
@@ -152,10 +154,8 @@ the merged regression becomes the commit that later pushes are compared
|
|
|
152
154
|
with. On a schedule, Duel compares with the last commit since your last
|
|
153
155
|
release that merged a pull request with the label, instead of the
|
|
154
156
|
release, so a slowdown you accepted doesn't fail every night until
|
|
155
|
-
your next release.
|
|
156
|
-
`duel init`
|
|
157
|
-
Create the label in your repository to use it, or set
|
|
158
|
-
`c.accept_label = nil` so that nothing can let a regression through.
|
|
157
|
+
your next release. Create the label in your repository to use it
|
|
158
|
+
(`duel init` does, where `gh` can).
|
|
159
159
|
|
|
160
160
|
## Compare a push with the last commit that passed
|
|
161
161
|
|
|
@@ -184,13 +184,15 @@ move the branch to accept the regression:
|
|
|
184
184
|
|
|
185
185
|
The settings that `duel`'s options set can be set in `.duel.rb` too, so
|
|
186
186
|
that a run in CI, where the workflow runs `duel` with no options, uses
|
|
187
|
-
them. An option given still wins
|
|
187
|
+
them. An option given still wins, and so does its variable (`DUEL_` and
|
|
188
|
+
its name): `duel --limit 1.1`, or `DUEL_LIMIT=1.1 duel`, runs with 1.1,
|
|
188
189
|
whatever `c.limit` says.
|
|
189
190
|
|
|
190
191
|
```ruby
|
|
191
192
|
Duel.configure do |c|
|
|
192
193
|
c.limit = 1.05 # --limit: the ratio above which a timing is slower (default: 1.02)
|
|
193
194
|
c.confidence = 0.995 # --confidence: of every interval (default: 0.99)
|
|
195
|
+
c.jobs = 2 # --jobs: benchmarks measured at once (default: one for every two CPUs, as many as fit)
|
|
194
196
|
c.processes = 6 # --processes: workers of each version (default: 12, 8 on JRuby, as many as fit)
|
|
195
197
|
c.timeout = 120 # --timeout: seconds a worker may take to load or answer (default: 60, 300 on JRuby)
|
|
196
198
|
c.time_limit = 3600 # --time-limit: seconds a run measures for (default: 7200)
|
|
@@ -239,12 +241,12 @@ A prepare block can use the following methods:
|
|
|
239
241
|
| `w.spawn(*command)` | Starts a process. Its output goes to a log in the worker's directory, named after the program. For a program like `ruby` that says little about itself, give the log a name with `log:`. To stop it with a signal other than `TERM`, pass `stop_signal:`. |
|
|
240
242
|
| `w.wait_until { ... }` | Waits until the block returns true. Fails if a spawned process exits first, or if the wait takes longer than `--timeout`. |
|
|
241
243
|
| `w.run(*command)` | Runs a command to completion, for example to set something up. Fails, and shows the command's log, if the command fails. |
|
|
242
|
-
| `w.
|
|
244
|
+
| `w.fail!("message")` | Fails the same way, for any other reason. |
|
|
243
245
|
| `w.at_stop { ... }` | Runs the block when the worker stops. |
|
|
244
246
|
| `w.cached(key) { \|dir\| ... }` | Makes something once, for every worker. See the following example. |
|
|
245
247
|
| `w.name` | The worker's name, for example `"base-1"`. |
|
|
246
248
|
| `w.dir` | The worker's directory. `w.path("name")` returns the path of a file in it. |
|
|
247
|
-
| `w.
|
|
249
|
+
| `w.free_port` | A free port of 127.0.0.1's, for a server to listen on, rather than on a Unix socket in `w.dir`: another each call, so keep the one you give the server. |
|
|
248
250
|
| `w.root` | The tree of the version that the worker runs. |
|
|
249
251
|
| `w.ruby` | The path of the Ruby that the version runs on. See [Run each version on its own Ruby](#run-each-version-on-its-own-ruby). |
|
|
250
252
|
| `w.env` | The version's environment variables. See the following paragraph. |
|
|
@@ -270,7 +272,7 @@ names what's cached, so put a version in it. If the key ends with a hash
|
|
|
270
272
|
of what the build depends on, such as `postgres-16.2-3f9c2a1b`, then
|
|
271
273
|
when a run ends, Duel removes the other builds of that name, the part
|
|
272
274
|
before the version and the hash, so an upgrade doesn't leave the old
|
|
273
|
-
build behind. Each build that the run used stays. `duel
|
|
275
|
+
build behind. Each build that the run used stays. `duel clean` removes
|
|
274
276
|
every cached directory.
|
|
275
277
|
|
|
276
278
|
```ruby
|
|
@@ -439,7 +441,7 @@ Duel.configure do |c|
|
|
|
439
441
|
end
|
|
440
442
|
c.metric(
|
|
441
443
|
"redis_cpu" => {
|
|
442
|
-
kind: :
|
|
444
|
+
kind: :timed,
|
|
443
445
|
unit: "µs", # shown as time (and sizing samples as CPU time does)
|
|
444
446
|
floor: 300, # µs: below this on both versions, a sample isn't judged on it
|
|
445
447
|
limit: 1.05 # its own limit (default: the run's, 1.02)
|
|
@@ -515,16 +517,16 @@ Every setting is optional:
|
|
|
515
517
|
|
|
516
518
|
| Setting | Applies to | Description |
|
|
517
519
|
|---|---|---|
|
|
518
|
-
| `kind:` | All | `:counted` (the default), which is exact and fails on any increase, or `:
|
|
520
|
+
| `kind:` | All | `:counted` (the default), which is exact and fails on any increase, or `:timed`, which is judged by paired ratios with a confidence interval, the same way as CPU time. |
|
|
519
521
|
| `unit:` | All | What the metric is in, as reports show it. `"ns"`, `"µs"`, `"ms"`, and `"s"` show it as time, and set `unit_ns`. Any other unit is either a word with its plural, such as `%w[row rows]` (1 row, 2 rows), or a symbol that's the same for any number, such as `"B"` or `"KB"`. A word without its plural fails. |
|
|
520
|
-
| `floor:` |
|
|
521
|
-
| `cpu_floor:` |
|
|
522
|
-
| `limit:` |
|
|
523
|
-
| `unit_ns:` |
|
|
524
|
-
| `
|
|
525
|
-
| `across:` |
|
|
526
|
-
| `
|
|
527
|
-
| `
|
|
522
|
+
| `floor:` | Timed | The least that a sample must measure, on either version, to be judged. |
|
|
523
|
+
| `cpu_floor:` | Timed | The least share of the sample's CPU time that the metric must be (in time, by `unit_ns`), on either version, to be judged. For example, `0.05` for 5%. |
|
|
524
|
+
| `limit:` | Timed | The ratio above which the metric is slower. The default is the run's limit. |
|
|
525
|
+
| `unit_ns:` | Timed | Nanoseconds per unit, to size samples by the metric. A time `unit:` sets it. |
|
|
526
|
+
| `size_samples:` | Timed | `false` for a metric that's too noisy to size samples by. |
|
|
527
|
+
| `across:` | Timed | `false` for a metric that's too noisy to judge across the board, where the margin is a quarter of its limit. |
|
|
528
|
+
| `per_iteration:` | Counted | How much more per iteration still doesn't count as more. |
|
|
529
|
+
| `per_sample:` | Counted | How much more per sample still doesn't count as more, for noise that doesn't grow with the iterations. |
|
|
528
530
|
| `calibrate:` | Counted | `false` to keep what a reading costs in the total, for a block that subtracts it itself. A reading that costs nothing needs no setting. |
|
|
529
531
|
|
|
530
532
|
A setting that doesn't exist fails, such as `toleranse:`, and so does a
|
|
@@ -560,11 +562,11 @@ worker. A subclass defines the following methods:
|
|
|
560
562
|
|
|
561
563
|
| Method | Description |
|
|
562
564
|
|---|---|
|
|
563
|
-
| `self.measures` | Names what the metric measures, with the settings in [Metric settings](#metric-settings), for example `{"wait" => {kind: :
|
|
565
|
+
| `self.measures` | Names what the metric measures, with the settings in [Metric settings](#metric-settings), for example `{"wait" => {kind: :timed, unit: "ns"}}`. By default, it's one counted measure named `self.metric_name`, the class's name in snake case: `QueryCount` is `query_count`. |
|
|
564
566
|
| `self.available?` | Returns `false` where the metric can't measure, to leave it out. |
|
|
565
567
|
| `self.costly?` | Returns `true` for a metric that's costly to read, such as one that queries a server, so that Duel doesn't pause it around each iteration inside an `around_iteration` hook. For more information, see [Account for the cost of reading](#account-for-the-cost-of-reading). |
|
|
566
568
|
| `open` | Called once in each worker, after its `c.in_worker` blocks, to set up what the metric reads, such as a subscription. |
|
|
567
|
-
| `start_sample(sample)` | Called before every sample. `sample.n` is the sample's number of iterations, and `sample.benchmark` is the benchmark, with its `name`, `
|
|
569
|
+
| `start_sample(sample)` | Called before every sample. `sample.n` is the sample's number of iterations, and `sample.benchmark` is the benchmark, with its `name`, and what it's judged on: `judge` (`[:timed, :counted]`, say), and `timed?`, `counted?` and `peak_memory?`. |
|
|
568
570
|
| `stop_sample` | Called after the sample. Returns what was measured, or a Hash of measurements by name. |
|
|
569
571
|
| `pause_sample` and `resume_sample(sample)` | Called around an `untimed` block, and around what an `around_iteration` hook does outside each iteration (unless the metric is costly). By default, they call `stop_sample` and `start_sample`. |
|
|
570
572
|
| `details` | Explains a regression. See the following paragraphs. |
|
|
@@ -575,11 +577,12 @@ A metric also has the following methods to call:
|
|
|
575
577
|
- `settings`: its settings by measure name, as the configuration has
|
|
576
578
|
them after `c.measure`'s changes, with every setting filled in. A
|
|
577
579
|
metric that measures one thing gets one setting with
|
|
578
|
-
`setting(:
|
|
579
|
-
`setting(:
|
|
580
|
+
`setting(:per_sample)`. Otherwise, name the measure:
|
|
581
|
+
`setting(:per_sample, "rows")`.
|
|
580
582
|
- `within?(more, n)`: whether a count that grew by `more` over `n`
|
|
581
|
-
iterations is still within its
|
|
582
|
-
|
|
583
|
+
iterations is still within what its `per_iteration:` and `per_sample:`
|
|
584
|
+
allow, as Duel judges
|
|
585
|
+
it. `Duel::Metric.within?(more, n, per_iteration:, per_sample:)` does the
|
|
583
586
|
same for any count.
|
|
584
587
|
|
|
585
588
|
To explain a regression, define `details`. It returns text, or text by
|
data/docs/how-it-works.md
CHANGED
|
@@ -88,8 +88,9 @@ never on noise, so that it can gate every pull request:
|
|
|
88
88
|
your own, fail on any increase: one more allocation per two
|
|
89
89
|
iterations. Each worker's count is its fewest in three rounds of a
|
|
90
90
|
sample from every worker, and a count fails when an interval over the
|
|
91
|
-
workers, for how many more the candidate's are, lies above the
|
|
92
|
-
|
|
91
|
+
workers, for how many more the candidate's are, lies above what the
|
|
92
|
+
measure allows (`per_iteration:` an iteration and `per_sample:` a
|
|
93
|
+
sample). Where every worker counts the same, as CRuby's workers
|
|
93
94
|
do, that's exactly how many more. Where workers differ, it's as few
|
|
94
95
|
as they can tell apart, so one worker's luck can't fail a benchmark.
|
|
95
96
|
With fewer than five workers of each version, a count is each
|
|
@@ -201,7 +202,7 @@ next run. If your project has an `ext/` directory, Duel builds the
|
|
|
201
202
|
baseline with `bundle exec rake compile`. To remove the baseline, git's
|
|
202
203
|
record of it, and the rest of `tmp/duel` (see
|
|
203
204
|
[What Duel keeps in tmp/duel](#what-duel-keeps-in-tmpduel)), run
|
|
204
|
-
`duel
|
|
205
|
+
`duel clean`.
|
|
205
206
|
|
|
206
207
|
If your project is in a subdirectory of its repository, as in a
|
|
207
208
|
monorepo, run `duel` in that subdirectory. Duel checks out the whole
|
|
@@ -240,7 +241,7 @@ Duel keeps what it needs between runs in your project's `tmp/duel`:
|
|
|
240
241
|
|
|
241
242
|
When a run ends, Duel removes `tmp/duel` if nothing is left in it, and
|
|
242
243
|
`tmp` too if Duel made it. A project without plugins that build
|
|
243
|
-
something keeps nothing there. `duel
|
|
244
|
+
something keeps nothing there. `duel clean` removes the rest. Each
|
|
244
245
|
run's sockets and server directories are in a temporary directory,
|
|
245
246
|
which the run removes when it ends.
|
|
246
247
|
|
|
@@ -265,12 +266,24 @@ when the gems are installed in the project's directory, as
|
|
|
265
266
|
|
|
266
267
|
## JIT compilers
|
|
267
268
|
|
|
268
|
-
If
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
compiles a method
|
|
272
|
-
|
|
273
|
-
|
|
269
|
+
If one of CRuby's JIT compilers is on in a worker, YJIT or ZJIT (Ruby
|
|
270
|
+
4.0's newer one, experimental there), Duel runs each benchmark until
|
|
271
|
+
it's steady before it measures it, for up to ten seconds. Each JIT
|
|
272
|
+
compiles a method after 30 calls (YJIT, in a big app, after 120), so a
|
|
273
|
+
small library's methods are compiled within Duel's first samples, but a
|
|
274
|
+
big app's take longer. A JIT can be on because you turned it on
|
|
275
|
+
(`RUBY_YJIT_ENABLE=1` or `RUBY_ZJIT_ENABLE=1`, for example), or because
|
|
276
|
+
your code did as it loaded, as Rails 7.2 and later do YJIT. Ruby runs
|
|
277
|
+
one JIT at most. Under ZJIT, a Rails app still boots: its call to turn
|
|
278
|
+
YJIT on warns that only one JIT can be on, and ZJIT stays on.
|
|
279
|
+
|
|
280
|
+
A Ruby without the JIT a variable turns on ignores it: one before 4.0,
|
|
281
|
+
or one built without ZJIT, ignores `RUBY_ZJIT_ENABLE`, and JRuby and
|
|
282
|
+
TruffleRuby, whose JIT is their own, ignore both. So where a worker's
|
|
283
|
+
environment turns a JIT on (as Ruby reads the variable: `1`, `true` or
|
|
284
|
+
`yes`) and the worker doesn't run it, Duel fails the run, saying which
|
|
285
|
+
version's workers, which JIT, and on what Ruby, rather than measure
|
|
286
|
+
without it.
|
|
274
287
|
|
|
275
288
|
A worker is steady once the median of its samples over each of three
|
|
276
289
|
half seconds in a row is no more than 1% faster than over the last half
|
data/docs/json.md
CHANGED
|
@@ -18,13 +18,13 @@ run replaces the file.
|
|
|
18
18
|
Duel writes the report whatever the outcome: a comparison that passed or
|
|
19
19
|
failed, nothing to compare, or a comparison that couldn't run. It writes
|
|
20
20
|
nothing when the command line isn't valid, such as an unknown option,
|
|
21
|
-
and nothing for
|
|
21
|
+
and nothing for `duel clean`, `--version`, or `duel init`, which aren't
|
|
22
22
|
comparisons. So a missing report means that `duel` didn't get as far as
|
|
23
23
|
comparing, or crashed.
|
|
24
24
|
|
|
25
25
|
## Versions of the schema
|
|
26
26
|
|
|
27
|
-
The report's `schema` key is the version of its schema, now `
|
|
27
|
+
The report's `schema` key is the version of its schema, now `2`. A new
|
|
28
28
|
key can appear in any release without a new version, so ignore the keys
|
|
29
29
|
that you don't know. Any other change, such as a key removed or renamed,
|
|
30
30
|
or a value that means something else, comes with a new version. Check
|
|
@@ -42,12 +42,12 @@ A run that found one regression, abridged to two benchmarks:
|
|
|
42
42
|
|
|
43
43
|
```json
|
|
44
44
|
{
|
|
45
|
-
"schema":
|
|
46
|
-
"duel_version": "0.
|
|
45
|
+
"schema": 2,
|
|
46
|
+
"duel_version": "0.3.0",
|
|
47
47
|
"status": "failed",
|
|
48
48
|
"exit": 1,
|
|
49
|
+
"passed_anyway": null,
|
|
49
50
|
"message": null,
|
|
50
|
-
"comment": "post",
|
|
51
51
|
"settings": {
|
|
52
52
|
"confidence": 0.99,
|
|
53
53
|
"limit": 1.02,
|
|
@@ -78,7 +78,6 @@ A run that found one regression, abridged to two benchmarks:
|
|
|
78
78
|
"verdict": "regressed",
|
|
79
79
|
"passed": false,
|
|
80
80
|
"regressed": ["instructions", "cpu", "allocations"],
|
|
81
|
-
"unsettled": false,
|
|
82
81
|
"iterations": 9,
|
|
83
82
|
"seconds": 3.1,
|
|
84
83
|
"confirmations": 3,
|
|
@@ -116,7 +115,7 @@ A run that found one regression, abridged to two benchmarks:
|
|
|
116
115
|
"candidate": 5791.0,
|
|
117
116
|
"unit": null,
|
|
118
117
|
"plural": null,
|
|
119
|
-
"
|
|
118
|
+
"verdict": "more"
|
|
120
119
|
}
|
|
121
120
|
},
|
|
122
121
|
"peak_memory": null,
|
|
@@ -136,7 +135,6 @@ A run that found one regression, abridged to two benchmarks:
|
|
|
136
135
|
"verdict": "same",
|
|
137
136
|
"passed": true,
|
|
138
137
|
"regressed": [],
|
|
139
|
-
"unsettled": false,
|
|
140
138
|
"iterations": 2400,
|
|
141
139
|
"seconds": 0.8,
|
|
142
140
|
"confirmations": 0,
|
|
@@ -161,7 +159,7 @@ A run that found one regression, abridged to two benchmarks:
|
|
|
161
159
|
"candidate": 3.0,
|
|
162
160
|
"unit": null,
|
|
163
161
|
"plural": null,
|
|
164
|
-
"
|
|
162
|
+
"verdict": "same"
|
|
165
163
|
}
|
|
166
164
|
},
|
|
167
165
|
"peak_memory": null,
|
|
@@ -178,11 +176,11 @@ A run that found one regression, abridged to two benchmarks:
|
|
|
178
176
|
"limit": 1.005,
|
|
179
177
|
"benchmarks": 12,
|
|
180
178
|
"processes": 8,
|
|
181
|
-
"verdict": "
|
|
179
|
+
"verdict": "unsettled",
|
|
182
180
|
"confirmed": false
|
|
183
181
|
}
|
|
184
182
|
},
|
|
185
|
-
"
|
|
183
|
+
"cut_short": null
|
|
186
184
|
}
|
|
187
185
|
```
|
|
188
186
|
|
|
@@ -190,12 +188,12 @@ A run with nothing to compare has the same keys, with nothing compared:
|
|
|
190
188
|
|
|
191
189
|
```json
|
|
192
190
|
{
|
|
193
|
-
"schema":
|
|
194
|
-
"duel_version": "0.
|
|
191
|
+
"schema": 2,
|
|
192
|
+
"duel_version": "0.3.0",
|
|
195
193
|
"status": "nothing_to_compare",
|
|
196
194
|
"exit": 0,
|
|
195
|
+
"passed_anyway": null,
|
|
197
196
|
"message": "duel: lib, ext, duel, .duel.rb unchanged since the baseline, so there's nothing to compare (--force to anyway)",
|
|
198
|
-
"comment": "change",
|
|
199
197
|
"settings": {"confidence": 0.99, "limit": 1.02, "filter": "", "jobs": 4, "processes": 8, "timeout": 60.0, "warm": null, "force": false},
|
|
200
198
|
"seconds": null,
|
|
201
199
|
"baseline": null,
|
|
@@ -203,7 +201,7 @@ A run with nothing to compare has the same keys, with nothing compared:
|
|
|
203
201
|
"plugins": [],
|
|
204
202
|
"benchmarks": [],
|
|
205
203
|
"across": {},
|
|
206
|
-
"
|
|
204
|
+
"cut_short": null
|
|
207
205
|
}
|
|
208
206
|
```
|
|
209
207
|
|
|
@@ -218,19 +216,18 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
|
|
|
218
216
|
|
|
219
217
|
| Key | What it is |
|
|
220
218
|
|---|---|
|
|
221
|
-
| `schema` | The schema's version: `
|
|
219
|
+
| `schema` | The schema's version: `2`. |
|
|
222
220
|
| `duel_version` | The version of Duel that wrote the report. |
|
|
223
|
-
| `status` | `"passed"`, `"failed"`, `"nothing_to_compare"`, or `"error"` (Duel couldn't compare). |
|
|
224
|
-
| `exit` | The exit status of `duel`: `0` for passed
|
|
221
|
+
| `status` | What the comparison found: `"passed"`, `"failed"`, `"nothing_to_compare"`, or `"error"` (Duel couldn't compare). |
|
|
222
|
+
| `exit` | The exit status of `duel`: `0` for passed, nothing to compare, or failed but passed anyway (see `passed_anyway`), `1` for failed, `2` for an error, and `3` for failed but only warned (on GitLab CI, where only a merge request fails). |
|
|
223
|
+
| `passed_anyway` | Why a run that failed passed anyway: `"accept_label"`, where the pull request's label accepted the regression (see [`c.accept_label`](configuration.md#accept-a-regression-with-a-label)), or `"fail_on"`, where only a pull request fails (see `c.fail_on`). Otherwise `null`. |
|
|
225
224
|
| `message` | What Duel said, for nothing to compare or an error, such as `"duel: no release tag (v*) before HEAD to compare against"`. Otherwise `null`. |
|
|
226
|
-
| `comment` | Whether the workflow that `duel init` writes comments on a pull request with the report, as [`c.comment`](configuration.md#choose-when-to-comment-on-a-pull-request) (or `--comment`) decides for this run: `"post"` a new comment, `"change"`, which posts one only if the workflow commented on the pull request before and its last comment's status differs from this run's, or `"none"`. See [Comment on a pull request](ci.md#comment-on-a-pull-request). |
|
|
227
225
|
| `settings` | The settings that the run used (see below). |
|
|
228
226
|
| `seconds` | How long the comparison took. `null` without a comparison. |
|
|
229
227
|
| `baseline`, `candidate` | The two versions (see below). `null` without a comparison. |
|
|
230
228
|
| `plugins` | The names of the plugins in use, such as `["rails", "activerecord"]`. |
|
|
231
229
|
| `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
|
|
232
230
|
| `across` | The verdicts across the board, by metric (see below). |
|
|
233
|
-
| `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
|
|
234
231
|
| `cut_short` | What the run left undone when it reached its time limit (`--time-limit`): `{"unmeasured": [names], "unconfirmed": [names], "across": true or false}`. `unmeasured` are the benchmarks that it didn't measure, which are errors. `unconfirmed` are those that it judged without measuring them again as often as it usually does, and `across` is whether it judged across the board without measuring every benchmark again. `null` for a run that finished within its limit. |
|
|
235
232
|
|
|
236
233
|
A run fails when a benchmark regressed, a benchmark failed, or a metric
|
|
@@ -268,13 +265,12 @@ was slower across the board and confirmed (`"confirmed": true`).
|
|
|
268
265
|
| `verdict` | `"regressed"`, `"error"`, `"unsettled"` (a timing too close to its limit to settle), `"faster"`, `"changed"` (a count changed), `"skipped"`, or `"same"`, as in the Markdown report's table. |
|
|
269
266
|
| `passed` | Whether it regressed on nothing and didn't fail. |
|
|
270
267
|
| `regressed` | The metrics that it regressed on, in every measurement, such as `["cpu", "allocations"]`. Peak memory is `"peak_memory"`. |
|
|
271
|
-
| `unsettled` | Whether a timing was too close to its limit to settle. |
|
|
272
268
|
| `iterations` | The iterations in each sample. `null` for a benchmark that failed or was skipped. |
|
|
273
269
|
| `seconds` | How long its first measurement took. `null` for a benchmark that failed or was skipped. |
|
|
274
270
|
| `confirmations` | How many times it was measured again, from scratch, because it looked worse. |
|
|
275
271
|
| `timed` | Its timings, by metric (see below). |
|
|
276
272
|
| `counted` | Its counts, by metric (see below). |
|
|
277
|
-
| `peak_memory` | Each version's peak memory, as `{"baseline": bytes, "candidate": bytes, "unit": "B", "
|
|
273
|
+
| `peak_memory` | Each version's peak memory, as `{"baseline": bytes, "candidate": bytes, "unit": "B", "verdict": "more" or "same"}`, for a benchmark that measures it: `"more"` by more than Duel tolerates. Otherwise `null`. |
|
|
278
274
|
| `details` | What metrics said of the last samples, by metric and then by version, such as `{"allocations": {"baseline": "2 String", "candidate": "3 String"}}`. |
|
|
279
275
|
| `explanation` | For a regression, the lines that explain it, as the terminal report and the Markdown report show them. Otherwise `[]`. |
|
|
280
276
|
| `error` | Why it failed, with any backtrace. Otherwise `null`. |
|
|
@@ -289,7 +285,7 @@ Each timing in `timed`, such as `cpu`:
|
|
|
289
285
|
| `limit` | The ratio above which it's slower. |
|
|
290
286
|
| `pairs` | The pairs of samples measured. |
|
|
291
287
|
| `processes` | The worker processes of each version that the interval is over, or `null` where it's over the pairs: for a process benchmark, or with fewer workers than the interval takes (see [How it works](how-it-works.md)). |
|
|
292
|
-
| `verdict` | `"slower"`, `"faster"`, `"same"`, or `"
|
|
288
|
+
| `verdict` | `"slower"`, `"faster"`, `"same"`, or `"unsettled"` (too close to the limit to settle). |
|
|
293
289
|
| `baseline`, `candidate` | Each version's median for one iteration, in `unit`. |
|
|
294
290
|
| `unit`, `plural` | The metric's unit, singular and plural, such as `"ns"`, or `null` for a plain number. |
|
|
295
291
|
| `across` | Whether it counts across the board. |
|
|
@@ -300,7 +296,7 @@ Each count in `counted`, such as `allocations`:
|
|
|
300
296
|
|---|---|
|
|
301
297
|
| `baseline`, `candidate` | Each version's fewest for one iteration, in `unit`. |
|
|
302
298
|
| `unit`, `plural` | The metric's unit, singular and plural, such as `"query"` and `"queries"`, or `null` for a plain number. |
|
|
303
|
-
| `
|
|
299
|
+
| `verdict` | `"more"` (worse: more than the metric tolerates), `"fewer"`, or `"same"` (or more, within what the metric tolerates). |
|
|
304
300
|
|
|
305
301
|
### Across the board
|
|
306
302
|
|
|
@@ -313,5 +309,5 @@ Each metric in `across`, such as `cpu`:
|
|
|
313
309
|
| `limit` | The ratio above which it's slower across the board. |
|
|
314
310
|
| `benchmarks` | The benchmarks that it's from. |
|
|
315
311
|
| `processes` | The worker processes of each version that the interval is over, or `null` where it's over the benchmarks. |
|
|
316
|
-
| `verdict` | `"slower"`, `"faster"`, `"same"`, or `"
|
|
312
|
+
| `verdict` | `"slower"`, `"faster"`, `"same"`, or `"unsettled"`. |
|
|
317
313
|
| `confirmed` | Whether it was slower again when every benchmark was measured a second time, which fails the run. |
|
data/docs/plugins.md
CHANGED
|
@@ -85,7 +85,7 @@ When you write a plugin, follow these guidelines:
|
|
|
85
85
|
a server takes more as it's used, so Duel counts at least this much
|
|
86
86
|
for each worker. A version of Duel before 0.1.8 ignores it.
|
|
87
87
|
- If a worker's server listens on a Unix socket in `w.dir`, give the
|
|
88
|
-
plugin an option to listen on `w.
|
|
88
|
+
plugin an option to listen on `w.free_port` instead, and default it to
|
|
89
89
|
where `Duel.unix_sockets?` is false, for example
|
|
90
90
|
`def self.configure(config, tcp: !Duel.unix_sockets?)`. It's false on
|
|
91
91
|
Windows, where a server's own build often has Unix sockets that Ruby
|
|
@@ -97,6 +97,23 @@ When you write a plugin, follow these guidelines:
|
|
|
97
97
|
`Duel::Platform` or `RbConfig` yourself: Duel's internals can change
|
|
98
98
|
in any release, and `Duel.windows?` won't. A version of Duel before
|
|
99
99
|
0.2.0 doesn't have it.
|
|
100
|
+
- If the plugin starts a server program, find it with
|
|
101
|
+
`Duel.find_program(server)`, which takes a path (anything with a `/`
|
|
102
|
+
or a `\` in it, expanded) as it is, and looks for a name on the
|
|
103
|
+
`PATH`, adding `.exe` on Windows. It returns `nil` if there's none,
|
|
104
|
+
and `Duel.path?(server)` tells the plugin's error whether to say it
|
|
105
|
+
isn't on the `PATH`, for example
|
|
106
|
+
`Duel.find_program(server) || w.fail!("no #{server}#{" on the PATH" unless Duel.path?(server)}")`.
|
|
107
|
+
A version of Duel before 0.3.0 doesn't have them.
|
|
108
|
+
- Give a plugin that starts a server the options that Duel's own server
|
|
109
|
+
plugins share, by the same names:
|
|
110
|
+
- `server:`, its program (by default `DUEL_<NAME>_SERVER`, or the
|
|
111
|
+
program's name), found with `Duel.find_program`.
|
|
112
|
+
- `settings:`, a Hash of more of the server's settings, by the
|
|
113
|
+
server's own names, passed after the plugin's own so that they win.
|
|
114
|
+
- `env:`, the variables given its address besides the plugin's own
|
|
115
|
+
`DUEL_<NAME>_*` one.
|
|
116
|
+
- `tcp:` (see above).
|
|
100
117
|
- To read a project's settings outside a run, as a generator that writes
|
|
101
118
|
its files might, load its `.duel.rb` with `Duel.load_config(path)`,
|
|
102
119
|
which returns its configuration. Before 0.2.0, it was private.
|
|
@@ -104,7 +121,7 @@ When you write a plugin, follow these guidelines:
|
|
|
104
121
|
> [!IMPORTANT]
|
|
105
122
|
> A helper that's meant for setup, such as one that runs a query, must
|
|
106
123
|
> call `Duel.unmeasured!` first, for example
|
|
107
|
-
> `Duel.unmeasured!("Duel::Redis.
|
|
124
|
+
> `Duel.unmeasured!("Duel::Redis.reset")`. It raises an error during a
|
|
108
125
|
> timed stretch (when `Duel.measuring?` is true), where what the helper
|
|
109
126
|
> does would be counted. That includes the block of
|
|
110
127
|
> `Duel::Testing.measure`, but not the block's `untimed` blocks.
|
|
@@ -190,11 +207,10 @@ A plugin's hooks are the configuration's, so they run before a file's
|
|
|
190
207
|
and a benchmark's own, in the order that they were given, and they run
|
|
191
208
|
where they're written, with the plugin's module as `self`:
|
|
192
209
|
|
|
193
|
-
- `c.before_sample {
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
with its `name`, `only`, and `peak_memory`, if it takes it.
|
|
210
|
+
- `c.before_sample { ... }` runs before every sample, warm-up samples
|
|
211
|
+
included, and isn't measured. Use it to put back what earlier samples
|
|
212
|
+
changed, such as reloading a database's fixtures, so that every
|
|
213
|
+
sample starts alike.
|
|
198
214
|
- `c.after_iterations { ... }` runs after each sample's iterations, and
|
|
199
215
|
is the one hook that Duel measures, with them. Use it to do the work
|
|
200
216
|
that the iterations started and left to be done later, such as jobs
|
|
@@ -274,8 +290,9 @@ returns. The block receives the following:
|
|
|
274
290
|
it has one stretch, and one more for each `untimed` block, as any
|
|
275
291
|
other sample does.
|
|
276
292
|
- `sample`: the sample. `sample.n` is its number of iterations, and
|
|
277
|
-
`sample.benchmark` is the benchmark, with its `name`,
|
|
278
|
-
`
|
|
293
|
+
`sample.benchmark` is the benchmark, with its `name`, and what it's
|
|
294
|
+
judged on: `judge` (`[:timed, :counted]`, say), and `timed?`,
|
|
295
|
+
`counted?` and `peak_memory?`.
|
|
279
296
|
|
|
280
297
|
A reading has the following:
|
|
281
298
|
|
|
@@ -339,16 +356,17 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
|
|
|
339
356
|
way that a worker does, and which ends after the sample. It runs the
|
|
340
357
|
configuration's hooks as a worker runs a benchmark's. Its block
|
|
341
358
|
receives the benchmark's context, with `untimed`, as a benchmark's
|
|
342
|
-
does: `measure(config) { |s| s.untimed { ... } }`.
|
|
343
|
-
|
|
344
|
-
benchmark's that
|
|
359
|
+
does: `measure(config) { |s| s.untimed { ... } }`. The block is one
|
|
360
|
+
iteration, and `measure` runs it `n:` times (once by default), as a
|
|
361
|
+
worker runs a benchmark's block that takes no `n`. With an
|
|
362
|
+
`around_iteration` hook, the configuration's or its own, each
|
|
363
|
+
iteration runs inside the hook, the way that a worker runs a
|
|
364
|
+
benchmark with one, including what it learns first to subtract the
|
|
365
|
+
hook's work from a costly metric. `judge:` makes the sample a
|
|
366
|
+
benchmark's that has, for example, `judge: :timed`. It also takes the
|
|
345
367
|
benchmark's own hooks, as `duel` does, such as `before_sample:`. It
|
|
346
368
|
returns a Hash of what was measured, by name, whose `details` are
|
|
347
|
-
what the metrics said of the sample, by measure.
|
|
348
|
-
`around_iteration` hook, the configuration's or its own, the block is
|
|
349
|
-
one iteration: `measure` runs it `n` times, each inside the hook, the
|
|
350
|
-
way that a worker runs a benchmark with one, including what it learns
|
|
351
|
-
first to subtract the hook's work from a costly metric.
|
|
369
|
+
what the metrics said of the sample, by measure.
|
|
352
370
|
- `in_worker(config) { ... }`: runs the block as a worker, and ends it
|
|
353
371
|
the way that a worker ends, so what its `c.in_worker` blocks gave
|
|
354
372
|
`at_stop` runs after the block. Every `measure` in the block is that
|
|
@@ -361,8 +379,8 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
|
|
|
361
379
|
- `compare(base, candidate)`: compares two projects in workers. See the
|
|
362
380
|
following paragraphs.
|
|
363
381
|
- `sample(n: 1)`: a stand-in sample of `n` iterations, which measures
|
|
364
|
-
nothing, for a metric's `start_sample` or a `details` block. `name
|
|
365
|
-
|
|
382
|
+
nothing, for a metric's `start_sample` or a `details` block. `name:`
|
|
383
|
+
and `judge:` describe its benchmark.
|
|
366
384
|
- `reading(totals, context)`: a stand-in reading of running totals, by
|
|
367
385
|
name, with the context that was read with them. A sample's stretches
|
|
368
386
|
are pairs of readings.
|
|
@@ -389,7 +407,7 @@ rolled_back = lambda do |iteration|
|
|
|
389
407
|
pg.exec("ROLLBACK")
|
|
390
408
|
end
|
|
391
409
|
Duel::Testing.measure(config, n: 20, around_iteration: rolled_back) { pg.exec("INSERT INTO items VALUES (1)") }
|
|
392
|
-
# => {"
|
|
410
|
+
# => {"postgresql_queries" => 20, "postgresql_rows" => 20, ...}
|
|
393
411
|
```
|
|
394
412
|
|
|
395
413
|
To test or benchmark what a metric says of a sample, without a worker,
|
|
@@ -405,7 +423,7 @@ A plugin's tests find what the plugin added in `config.metrics`, the
|
|
|
405
423
|
metrics in order. Each metric has its `measures` and their `settings`,
|
|
406
424
|
with every setting filled in. For example,
|
|
407
425
|
`config.metrics.first.settings.fetch("redis_cpu")` is
|
|
408
|
-
`{kind: :
|
|
426
|
+
`{kind: :timed, unit: "µs", floor: 300, ...}`.
|
|
409
427
|
|
|
410
428
|
`compare` takes `duel`'s options as keywords: `processes:` (two unless
|
|
411
429
|
given, too few for an interval over workers, so each timing is judged
|
|
@@ -424,7 +442,7 @@ printed), and its `results`, one for each benchmark.
|
|
|
424
442
|
version's median (`"baseline"` and `"candidate"`), by name.
|
|
425
443
|
- Each `counted` measure's fewest on each version (`"baseline"` and
|
|
426
444
|
`"candidate"`) over a sample's `"n"` iterations, with its
|
|
427
|
-
`"
|
|
445
|
+
`"per_iteration"` and `"per_sample"` (both with their `"unit"` and
|
|
428
446
|
`"plural"`), by name.
|
|
429
447
|
- What metrics said of it (`details`), by measure and by version.
|
|
430
448
|
- Its peak `memory`, by version, when it was asked for.
|
data/docs/troubleshooting.md
CHANGED
|
@@ -14,7 +14,8 @@ what a change must touch to be compared, set `c.watch`. See
|
|
|
14
14
|
|
|
15
15
|
Against `--base last-release`, Duel also reports that there's nothing to
|
|
16
16
|
compare against until you tag your first release. Releases are tagged
|
|
17
|
-
`v*` by default. To match other tags, set `c.release_tags
|
|
17
|
+
`v*` by default. To match other tags, set `c.release_tags`, to a glob or
|
|
18
|
+
a list of them (`%w[v* my_gem-v*]`, for a gem that renamed its tags).
|
|
18
19
|
|
|
19
20
|
## Duel finds no main branch, or no commits
|
|
20
21
|
|
|
@@ -125,7 +126,7 @@ Duel reports a timing that's still too close to the limit to call, and
|
|
|
125
126
|
doesn't fail on it. A benchmark that's reported unsettled run after run
|
|
126
127
|
is noisy. Make every iteration do the same work, or, for work that's
|
|
127
128
|
mostly waiting, judge the benchmark on its counts alone with
|
|
128
|
-
`
|
|
129
|
+
`judge: :counted`. See
|
|
129
130
|
[Choose what a benchmark is judged on](benchmarks.md#choose-what-a-benchmark-is-judged-on)
|
|
130
131
|
and [How Duel tells a regression from noise](how-it-works.md#how-duel-tells-a-regression-from-noise).
|
|
131
132
|
|