duel 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. checksums.yaml +4 -4
  2. data/AGENT_PROMPT.md +51 -27
  3. data/CHANGELOG.md +247 -1
  4. data/README.md +30 -14
  5. data/docs/benchmarks.md +28 -24
  6. data/docs/ci.md +328 -362
  7. data/docs/configuration.md +40 -37
  8. data/docs/how-it-works.md +23 -10
  9. data/docs/json.md +21 -25
  10. data/docs/plugins.md +40 -22
  11. data/docs/troubleshooting.md +3 -2
  12. data/lib/duel/across.rb +18 -18
  13. data/lib/duel/badge.rb +174 -0
  14. data/lib/duel/baseline.rb +38 -15
  15. data/lib/duel/cli.rb +175 -107
  16. data/lib/duel/comment.rb +80 -71
  17. data/lib/duel/config.rb +145 -63
  18. data/lib/duel/context.rb +29 -24
  19. data/lib/duel/forge.rb +87 -0
  20. data/lib/duel/init.rb +44 -87
  21. data/lib/duel/json_report.rb +34 -25
  22. data/lib/duel/lane.rb +107 -61
  23. data/lib/duel/lock.rb +1 -1
  24. data/lib/duel/mark_passed.rb +132 -0
  25. data/lib/duel/metric.rb +28 -27
  26. data/lib/duel/native.rb +7 -5
  27. data/lib/duel/platform.rb +4 -0
  28. data/lib/duel/process_sample.rb +9 -6
  29. data/lib/duel/rake_task.rb +4 -5
  30. data/lib/duel/resource.rb +10 -11
  31. data/lib/duel/result.rb +22 -14
  32. data/lib/duel/rubies.rb +6 -2
  33. data/lib/duel/runner.rb +27 -15
  34. data/lib/duel/spawn.rb +29 -4
  35. data/lib/duel/stats.rb +14 -12
  36. data/lib/duel/summary.rb +31 -29
  37. data/lib/duel/templates/benchmark_duel.rb.erb +2 -2
  38. data/lib/duel/templates/duel.gitlab-ci.yml.erb +18 -54
  39. data/lib/duel/templates/duel.rb.erb +7 -7
  40. data/lib/duel/templates/perf-caller.yml.erb +22 -5
  41. data/lib/duel/templates/perf-comment.yml.erb +11 -11
  42. data/lib/duel/templates/perf-push-jobs.yml.erb +127 -0
  43. data/lib/duel/templates/perf-push.yml.erb +50 -0
  44. data/lib/duel/templates/perf.yml.erb +113 -335
  45. data/lib/duel/testing.rb +20 -25
  46. data/lib/duel/version.rb +1 -1
  47. data/lib/duel/worker.rb +16 -9
  48. data/lib/duel/worker_process.rb +6 -6
  49. data/lib/duel.rb +88 -33
  50. data/sig/duel/across.rbs +4 -0
  51. data/sig/duel/badge.rbs +51 -0
  52. data/sig/duel/baseline.rbs +7 -2
  53. data/sig/duel/cli.rbs +31 -20
  54. data/sig/duel/comment.rbs +14 -15
  55. data/sig/duel/config.rbs +71 -29
  56. data/sig/duel/context.rbs +4 -4
  57. data/sig/duel/forge.rbs +42 -0
  58. data/sig/duel/init.rbs +8 -26
  59. data/sig/duel/json_report.rbs +7 -6
  60. data/sig/duel/lane.rbs +29 -6
  61. data/sig/duel/mark_passed.rbs +39 -0
  62. data/sig/duel/metric.rbs +10 -10
  63. data/sig/duel/native.rbs +3 -0
  64. data/sig/duel/platform.rbs +1 -0
  65. data/sig/duel/process_sample.rbs +3 -0
  66. data/sig/duel/resource.rbs +3 -3
  67. data/sig/duel/result.rbs +12 -5
  68. data/sig/duel/rubies.rbs +3 -0
  69. data/sig/duel/runner.rbs +23 -16
  70. data/sig/duel/spawn.rbs +7 -0
  71. data/sig/duel/stats.rbs +4 -1
  72. data/sig/duel/summary.rbs +8 -0
  73. data/sig/duel/testing.rbs +2 -2
  74. data/sig/duel/worker.rbs +6 -5
  75. data/sig/duel/worker_process.rbs +3 -3
  76. data/sig/duel.rbs +50 -31
  77. metadata +9 -1
@@ -9,22 +9,23 @@ runner and every worker load it:
9
9
 
10
10
  ```ruby
11
11
  Duel.configure do |c|
12
- c.benchmarks = "perf/**/*_bench.rb" # default: duel/**/*_duel.rb
13
- c.watch = %w[lib ext] # the code a change must touch to be compared
12
+ c.benchmarks = "perf/**/*_bench.rb" # or a list of globs (default: duel/**/*_duel.rb)
13
+ c.watch = %w[lib ext] # the code a change must touch to be compared (a path, or a list)
14
14
  c.library = "my_gem" # required from each version (default: the gemspec's)
15
15
  c.build = "make" # for both versions (default: rake compile, with ext/)
16
16
  c.main_branch = "origin/trunk"
17
- c.release_tags = "release-*" # default: v*
17
+ c.release_tags = %w[v* release-*] # a glob, or a list, any of which a tag matches (default: v*)
18
18
  c.ruby = :version_file # each version's own Ruby (default: the runner's)
19
19
  c.comment = :results # when CI comments on a pull request (default: :failures)
20
20
  c.fail_on = :pull_requests # which runs a regression fails (default: :every_run)
21
21
  c.push_base = :last_passed # what CI compares a push with (default: :before)
22
- c.accept_label = "duel: accept" # a pull request label that accepts a regression
22
+ c.accept_label = "perf: accept" # a pull request label that accepts a regression (default: "duel: accept")
23
23
  c.limit = 1.05 # a run's settings, as their options (see Set a run's settings)
24
24
  c.use :redis # a plugin (see Use plugins)
25
25
 
26
- # How a worker loads one version of your code, instead of the library.
27
- c.load { |root| require File.join(root, "config", "boot") }
26
+ # Or, instead of a library to require, how a worker loads one version
27
+ # of your code, from the root of its tree:
28
+ # c.library = ->(root) { require File.join(root, "config", "boot") }
28
29
  end
29
30
  ```
30
31
 
@@ -42,7 +43,8 @@ nothing to compare, and Duel passes without running the benchmarks.
42
43
  watches what decides how your code is measured:
43
44
 
44
45
  - The benchmarks' directory: the part of `c.benchmarks` before its first
45
- wildcard, `duel/` by default, with its helpers and fixtures.
46
+ wildcard (each glob's, for a list), `duel/` by default, with its
47
+ helpers and fixtures.
46
48
  - The configuration's file, `.duel.rb`.
47
49
  - With [`c.ruby = :version_file`](#run-each-version-on-its-own-ruby),
48
50
  the version files at your project's root (`.ruby-version`,
@@ -64,7 +66,7 @@ end
64
66
 
65
67
  The [GitHub Actions workflow](ci.md#comment-on-a-pull-request) that
66
68
  `duel init` writes comments on a pull request with Duel's report.
67
- `c.comment` chooses when, and `duel --comment` overrides it:
69
+ `c.comment` chooses when:
68
70
 
69
71
  | Value | When the workflow comments |
70
72
  |---|---|
@@ -99,8 +101,7 @@ says which values it takes.
99
101
  By default, a regression fails every run: a pull request's, a push's,
100
102
  and the schedule's. In a project where people push to `main` directly,
101
103
  a failed push only gets GitHub's email that the run failed. To report a
102
- regression in a push without failing it, set `c.fail_on`, and
103
- `duel --fail-on` overrides it:
104
+ regression in a push without failing it, set `c.fail_on`:
104
105
 
105
106
  | Value | Which runs a regression fails |
106
107
  |---|---|
@@ -128,12 +129,13 @@ always fails.
128
129
  ## Accept a regression with a label
129
130
 
130
131
  Sometimes a change is slower on purpose: it's correct where the old
131
- code wasn't, say. To let a pull request through anyway, name a label
132
- that accepts its regression:
132
+ code wasn't, say. To let a pull request through anyway, label it
133
+ `duel: accept`, the label that accepts its regression by default. To
134
+ name another, or none:
133
135
 
134
136
  ```ruby
135
137
  Duel.configure do |c|
136
- c.accept_label = "duel: accept"
138
+ c.accept_label = "perf: accept" # or nil, so that nothing can let a regression through
137
139
  end
138
140
  ```
139
141
 
@@ -152,10 +154,8 @@ the merged regression becomes the commit that later pushes are compared
152
154
  with. On a schedule, Duel compares with the last commit since your last
153
155
  release that merged a pull request with the label, instead of the
154
156
  release, so a slowdown you accepted doesn't fail every night until
155
- your next release. Duel's default is no such label, but the `.duel.rb` that
156
- `duel init` writes (and duel-rails's generator) names `duel: accept`.
157
- Create the label in your repository to use it, or set
158
- `c.accept_label = nil` so that nothing can let a regression through.
157
+ your next release. Create the label in your repository to use it
158
+ (`duel init` does, where `gh` can).
159
159
 
160
160
  ## Compare a push with the last commit that passed
161
161
 
@@ -184,13 +184,15 @@ move the branch to accept the regression:
184
184
 
185
185
  The settings that `duel`'s options set can be set in `.duel.rb` too, so
186
186
  that a run in CI, where the workflow runs `duel` with no options, uses
187
- them. An option given still wins: `duel --limit 1.1` runs with 1.1,
187
+ them. An option given still wins, and so does its variable (`DUEL_` and
188
+ its name): `duel --limit 1.1`, or `DUEL_LIMIT=1.1 duel`, runs with 1.1,
188
189
  whatever `c.limit` says.
189
190
 
190
191
  ```ruby
191
192
  Duel.configure do |c|
192
193
  c.limit = 1.05 # --limit: the ratio above which a timing is slower (default: 1.02)
193
194
  c.confidence = 0.995 # --confidence: of every interval (default: 0.99)
195
+ c.jobs = 2 # --jobs: benchmarks measured at once (default: one for every two CPUs, as many as fit)
194
196
  c.processes = 6 # --processes: workers of each version (default: 12, 8 on JRuby, as many as fit)
195
197
  c.timeout = 120 # --timeout: seconds a worker may take to load or answer (default: 60, 300 on JRuby)
196
198
  c.time_limit = 3600 # --time-limit: seconds a run measures for (default: 7200)
@@ -239,12 +241,12 @@ A prepare block can use the following methods:
239
241
  | `w.spawn(*command)` | Starts a process. Its output goes to a log in the worker's directory, named after the program. For a program like `ruby` that says little about itself, give the log a name with `log:`. To stop it with a signal other than `TERM`, pass `stop_signal:`. |
240
242
  | `w.wait_until { ... }` | Waits until the block returns true. Fails if a spawned process exits first, or if the wait takes longer than `--timeout`. |
241
243
  | `w.run(*command)` | Runs a command to completion, for example to set something up. Fails, and shows the command's log, if the command fails. |
242
- | `w.failed("message")` | Fails the same way, for any other reason. |
244
+ | `w.fail!("message")` | Fails the same way, for any other reason. |
243
245
  | `w.at_stop { ... }` | Runs the block when the worker stops. |
244
246
  | `w.cached(key) { \|dir\| ... }` | Makes something once, for every worker. See the following example. |
245
247
  | `w.name` | The worker's name, for example `"base-1"`. |
246
248
  | `w.dir` | The worker's directory. `w.path("name")` returns the path of a file in it. |
247
- | `w.port` | A free port of 127.0.0.1's, for a server to listen on, rather than on a Unix socket in `w.dir`. |
249
+ | `w.free_port` | A free port of 127.0.0.1's, for a server to listen on, rather than on a Unix socket in `w.dir`: another each call, so keep the one you give the server. |
248
250
  | `w.root` | The tree of the version that the worker runs. |
249
251
  | `w.ruby` | The path of the Ruby that the version runs on. See [Run each version on its own Ruby](#run-each-version-on-its-own-ruby). |
250
252
  | `w.env` | The version's environment variables. See the following paragraph. |
@@ -270,7 +272,7 @@ names what's cached, so put a version in it. If the key ends with a hash
270
272
  of what the build depends on, such as `postgres-16.2-3f9c2a1b`, then
271
273
  when a run ends, Duel removes the other builds of that name, the part
272
274
  before the version and the hash, so an upgrade doesn't leave the old
273
- build behind. Each build that the run used stays. `duel --clean` removes
275
+ build behind. Each build that the run used stays. `duel clean` removes
274
276
  every cached directory.
275
277
 
276
278
  ```ruby
@@ -439,7 +441,7 @@ Duel.configure do |c|
439
441
  end
440
442
  c.metric(
441
443
  "redis_cpu" => {
442
- kind: :ratio,
444
+ kind: :timed,
443
445
  unit: "µs", # shown as time (and sizing samples as CPU time does)
444
446
  floor: 300, # µs: below this on both versions, a sample isn't judged on it
445
447
  limit: 1.05 # its own limit (default: the run's, 1.02)
@@ -515,16 +517,16 @@ Every setting is optional:
515
517
 
516
518
  | Setting | Applies to | Description |
517
519
  |---|---|---|
518
- | `kind:` | All | `:counted` (the default), which is exact and fails on any increase, or `:ratio`, which is judged by paired ratios with a confidence interval, the same way as CPU time. |
520
+ | `kind:` | All | `:counted` (the default), which is exact and fails on any increase, or `:timed`, which is judged by paired ratios with a confidence interval, the same way as CPU time. |
519
521
  | `unit:` | All | What the metric is in, as reports show it. `"ns"`, `"µs"`, `"ms"`, and `"s"` show it as time, and set `unit_ns`. Any other unit is either a word with its plural, such as `%w[row rows]` (1 row, 2 rows), or a symbol that's the same for any number, such as `"B"` or `"KB"`. A word without its plural fails. |
520
- | `floor:` | Ratio | The least that a sample must measure, on either version, to be judged. |
521
- | `cpu_floor:` | Ratio | The least share of the sample's CPU time that the metric must be (in time, by `unit_ns`), on either version, to be judged. For example, `0.05` for 5%. |
522
- | `limit:` | Ratio | The ratio above which the metric is slower. The default is the run's limit. |
523
- | `unit_ns:` | Ratio | Nanoseconds per unit, to size samples by the metric. A time `unit:` sets it. |
524
- | `sizes_samples:` | Ratio | `false` for a metric that's too noisy to size samples by. |
525
- | `across:` | Ratio | `false` for a metric that's too noisy to judge across the board, where the margin is a quarter of its limit. |
526
- | `tolerance:` | Counted | How much more per iteration still doesn't count as more. |
527
- | `allowance:` | Counted | How much more per sample still doesn't count as more, for noise that doesn't grow with the iterations. |
522
+ | `floor:` | Timed | The least that a sample must measure, on either version, to be judged. |
523
+ | `cpu_floor:` | Timed | The least share of the sample's CPU time that the metric must be (in time, by `unit_ns`), on either version, to be judged. For example, `0.05` for 5%. |
524
+ | `limit:` | Timed | The ratio above which the metric is slower. The default is the run's limit. |
525
+ | `unit_ns:` | Timed | Nanoseconds per unit, to size samples by the metric. A time `unit:` sets it. |
526
+ | `size_samples:` | Timed | `false` for a metric that's too noisy to size samples by. |
527
+ | `across:` | Timed | `false` for a metric that's too noisy to judge across the board, where the margin is a quarter of its limit. |
528
+ | `per_iteration:` | Counted | How much more per iteration still doesn't count as more. |
529
+ | `per_sample:` | Counted | How much more per sample still doesn't count as more, for noise that doesn't grow with the iterations. |
528
530
  | `calibrate:` | Counted | `false` to keep what a reading costs in the total, for a block that subtracts it itself. A reading that costs nothing needs no setting. |
529
531
 
530
532
  A setting that doesn't exist fails, such as `toleranse:`, and so does a
@@ -560,11 +562,11 @@ worker. A subclass defines the following methods:
560
562
 
561
563
  | Method | Description |
562
564
  |---|---|
563
- | `self.measures` | Names what the metric measures, with the settings in [Metric settings](#metric-settings), for example `{"wait" => {kind: :ratio, unit: "ns"}}`. By default, it's one counted measure named `self.metric_name`, the class's name in snake case: `QueryCount` is `query_count`. |
565
+ | `self.measures` | Names what the metric measures, with the settings in [Metric settings](#metric-settings), for example `{"wait" => {kind: :timed, unit: "ns"}}`. By default, it's one counted measure named `self.metric_name`, the class's name in snake case: `QueryCount` is `query_count`. |
564
566
  | `self.available?` | Returns `false` where the metric can't measure, to leave it out. |
565
567
  | `self.costly?` | Returns `true` for a metric that's costly to read, such as one that queries a server, so that Duel doesn't pause it around each iteration inside an `around_iteration` hook. For more information, see [Account for the cost of reading](#account-for-the-cost-of-reading). |
566
568
  | `open` | Called once in each worker, after its `c.in_worker` blocks, to set up what the metric reads, such as a subscription. |
567
- | `start_sample(sample)` | Called before every sample. `sample.n` is the sample's number of iterations, and `sample.benchmark` is the benchmark, with its `name`, `only`, and `peak_memory`. |
569
+ | `start_sample(sample)` | Called before every sample. `sample.n` is the sample's number of iterations, and `sample.benchmark` is the benchmark, with its `name`, and what it's judged on: `judge` (`[:timed, :counted]`, say), and `timed?`, `counted?` and `peak_memory?`. |
568
570
  | `stop_sample` | Called after the sample. Returns what was measured, or a Hash of measurements by name. |
569
571
  | `pause_sample` and `resume_sample(sample)` | Called around an `untimed` block, and around what an `around_iteration` hook does outside each iteration (unless the metric is costly). By default, they call `stop_sample` and `start_sample`. |
570
572
  | `details` | Explains a regression. See the following paragraphs. |
@@ -575,11 +577,12 @@ A metric also has the following methods to call:
575
577
  - `settings`: its settings by measure name, as the configuration has
576
578
  them after `c.measure`'s changes, with every setting filled in. A
577
579
  metric that measures one thing gets one setting with
578
- `setting(:allowance)`. Otherwise, name the measure:
579
- `setting(:allowance, "rows")`.
580
+ `setting(:per_sample)`. Otherwise, name the measure:
581
+ `setting(:per_sample, "rows")`.
580
582
  - `within?(more, n)`: whether a count that grew by `more` over `n`
581
- iterations is still within its tolerance and allowance, as Duel judges
582
- it. `Duel::Metric.within?(more, n, tolerance:, allowance:)` does the
583
+ iterations is still within what its `per_iteration:` and `per_sample:`
584
+ allow, as Duel judges
585
+ it. `Duel::Metric.within?(more, n, per_iteration:, per_sample:)` does the
583
586
  same for any count.
584
587
 
585
588
  To explain a regression, define `details`. It returns text, or text by
data/docs/how-it-works.md CHANGED
@@ -88,8 +88,9 @@ never on noise, so that it can gate every pull request:
88
88
  your own, fail on any increase: one more allocation per two
89
89
  iterations. Each worker's count is its fewest in three rounds of a
90
90
  sample from every worker, and a count fails when an interval over the
91
- workers, for how many more the candidate's are, lies above the
92
- tolerance. Where every worker counts the same, as CRuby's workers
91
+ workers, for how many more the candidate's are, lies above what the
92
+ measure allows (`per_iteration:` an iteration and `per_sample:` a
93
+ sample). Where every worker counts the same, as CRuby's workers
93
94
  do, that's exactly how many more. Where workers differ, it's as few
94
95
  as they can tell apart, so one worker's luck can't fail a benchmark.
95
96
  With fewer than five workers of each version, a count is each
@@ -201,7 +202,7 @@ next run. If your project has an `ext/` directory, Duel builds the
201
202
  baseline with `bundle exec rake compile`. To remove the baseline, git's
202
203
  record of it, and the rest of `tmp/duel` (see
203
204
  [What Duel keeps in tmp/duel](#what-duel-keeps-in-tmpduel)), run
204
- `duel --clean`.
205
+ `duel clean`.
205
206
 
206
207
  If your project is in a subdirectory of its repository, as in a
207
208
  monorepo, run `duel` in that subdirectory. Duel checks out the whole
@@ -240,7 +241,7 @@ Duel keeps what it needs between runs in your project's `tmp/duel`:
240
241
 
241
242
  When a run ends, Duel removes `tmp/duel` if nothing is left in it, and
242
243
  `tmp` too if Duel made it. A project without plugins that build
243
- something keeps nothing there. `duel --clean` removes the rest. Each
244
+ something keeps nothing there. `duel clean` removes the rest. Each
244
245
  run's sockets and server directories are in a temporary directory,
245
246
  which the run removes when it ends.
246
247
 
@@ -265,12 +266,24 @@ when the gems are installed in the project's directory, as
265
266
 
266
267
  ## JIT compilers
267
268
 
268
- If YJIT is on in a worker, Duel runs each benchmark until it's steady
269
- before it measures it, for up to ten seconds. A small library's methods
270
- are compiled within Duel's first samples, but in a big app, YJIT
271
- compiles a method only after 120 calls. YJIT can be on because you
272
- turned it on (`RUBY_YJIT_ENABLE=1`, for example), or because your code
273
- did as it loaded, as Rails 7.2 and later do.
269
+ If one of CRuby's JIT compilers is on in a worker, YJIT or ZJIT (Ruby
270
+ 4.0's newer one, experimental there), Duel runs each benchmark until
271
+ it's steady before it measures it, for up to ten seconds. Each JIT
272
+ compiles a method after 30 calls (YJIT, in a big app, after 120), so a
273
+ small library's methods are compiled within Duel's first samples, but a
274
+ big app's take longer. A JIT can be on because you turned it on
275
+ (`RUBY_YJIT_ENABLE=1` or `RUBY_ZJIT_ENABLE=1`, for example), or because
276
+ your code did as it loaded, as Rails 7.2 and later do YJIT. Ruby runs
277
+ one JIT at most. Under ZJIT, a Rails app still boots: its call to turn
278
+ YJIT on warns that only one JIT can be on, and ZJIT stays on.
279
+
280
+ A Ruby without the JIT a variable turns on ignores it: one before 4.0,
281
+ or one built without ZJIT, ignores `RUBY_ZJIT_ENABLE`, and JRuby and
282
+ TruffleRuby, whose JIT is their own, ignore both. So where a worker's
283
+ environment turns a JIT on (as Ruby reads the variable: `1`, `true` or
284
+ `yes`) and the worker doesn't run it, Duel fails the run, saying which
285
+ version's workers, which JIT, and on what Ruby, rather than measure
286
+ without it.
274
287
 
275
288
  A worker is steady once the median of its samples over each of three
276
289
  half seconds in a row is no more than 1% faster than over the last half
data/docs/json.md CHANGED
@@ -18,13 +18,13 @@ run replaces the file.
18
18
  Duel writes the report whatever the outcome: a comparison that passed or
19
19
  failed, nothing to compare, or a comparison that couldn't run. It writes
20
20
  nothing when the command line isn't valid, such as an unknown option,
21
- and nothing for `--clean`, `--version`, or `duel init`, which aren't
21
+ and nothing for `duel clean`, `--version`, or `duel init`, which aren't
22
22
  comparisons. So a missing report means that `duel` didn't get as far as
23
23
  comparing, or crashed.
24
24
 
25
25
  ## Versions of the schema
26
26
 
27
- The report's `schema` key is the version of its schema, now `1`. A new
27
+ The report's `schema` key is the version of its schema, now `2`. A new
28
28
  key can appear in any release without a new version, so ignore the keys
29
29
  that you don't know. Any other change, such as a key removed or renamed,
30
30
  or a value that means something else, comes with a new version. Check
@@ -42,12 +42,12 @@ A run that found one regression, abridged to two benchmarks:
42
42
 
43
43
  ```json
44
44
  {
45
- "schema": 1,
46
- "duel_version": "0.1.0",
45
+ "schema": 2,
46
+ "duel_version": "0.3.0",
47
47
  "status": "failed",
48
48
  "exit": 1,
49
+ "passed_anyway": null,
49
50
  "message": null,
50
- "comment": "post",
51
51
  "settings": {
52
52
  "confidence": 0.99,
53
53
  "limit": 1.02,
@@ -78,7 +78,6 @@ A run that found one regression, abridged to two benchmarks:
78
78
  "verdict": "regressed",
79
79
  "passed": false,
80
80
  "regressed": ["instructions", "cpu", "allocations"],
81
- "unsettled": false,
82
81
  "iterations": 9,
83
82
  "seconds": 3.1,
84
83
  "confirmations": 3,
@@ -116,7 +115,7 @@ A run that found one regression, abridged to two benchmarks:
116
115
  "candidate": 5791.0,
117
116
  "unit": null,
118
117
  "plural": null,
119
- "worse": true
118
+ "verdict": "more"
120
119
  }
121
120
  },
122
121
  "peak_memory": null,
@@ -136,7 +135,6 @@ A run that found one regression, abridged to two benchmarks:
136
135
  "verdict": "same",
137
136
  "passed": true,
138
137
  "regressed": [],
139
- "unsettled": false,
140
138
  "iterations": 2400,
141
139
  "seconds": 0.8,
142
140
  "confirmations": 0,
@@ -161,7 +159,7 @@ A run that found one regression, abridged to two benchmarks:
161
159
  "candidate": 3.0,
162
160
  "unit": null,
163
161
  "plural": null,
164
- "worse": false
162
+ "verdict": "same"
165
163
  }
166
164
  },
167
165
  "peak_memory": null,
@@ -178,11 +176,11 @@ A run that found one regression, abridged to two benchmarks:
178
176
  "limit": 1.005,
179
177
  "benchmarks": 12,
180
178
  "processes": 8,
181
- "verdict": "undecided",
179
+ "verdict": "unsettled",
182
180
  "confirmed": false
183
181
  }
184
182
  },
185
- "errors": []
183
+ "cut_short": null
186
184
  }
187
185
  ```
188
186
 
@@ -190,12 +188,12 @@ A run with nothing to compare has the same keys, with nothing compared:
190
188
 
191
189
  ```json
192
190
  {
193
- "schema": 1,
194
- "duel_version": "0.1.0",
191
+ "schema": 2,
192
+ "duel_version": "0.3.0",
195
193
  "status": "nothing_to_compare",
196
194
  "exit": 0,
195
+ "passed_anyway": null,
197
196
  "message": "duel: lib, ext, duel, .duel.rb unchanged since the baseline, so there's nothing to compare (--force to anyway)",
198
- "comment": "change",
199
197
  "settings": {"confidence": 0.99, "limit": 1.02, "filter": "", "jobs": 4, "processes": 8, "timeout": 60.0, "warm": null, "force": false},
200
198
  "seconds": null,
201
199
  "baseline": null,
@@ -203,7 +201,7 @@ A run with nothing to compare has the same keys, with nothing compared:
203
201
  "plugins": [],
204
202
  "benchmarks": [],
205
203
  "across": {},
206
- "errors": []
204
+ "cut_short": null
207
205
  }
208
206
  ```
209
207
 
@@ -218,19 +216,18 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
218
216
 
219
217
  | Key | What it is |
220
218
  |---|---|
221
- | `schema` | The schema's version: `1`. |
219
+ | `schema` | The schema's version: `2`. |
222
220
  | `duel_version` | The version of Duel that wrote the report. |
223
- | `status` | `"passed"`, `"failed"`, `"nothing_to_compare"`, or `"error"` (Duel couldn't compare). |
224
- | `exit` | The exit status of `duel`: `0` for passed or nothing to compare, `1` for failed, `2` for an error. |
221
+ | `status` | What the comparison found: `"passed"`, `"failed"`, `"nothing_to_compare"`, or `"error"` (Duel couldn't compare). |
222
+ | `exit` | The exit status of `duel`: `0` for passed, nothing to compare, or failed but passed anyway (see `passed_anyway`), `1` for failed, `2` for an error, and `3` for failed but only warned (on GitLab CI, where only a merge request fails). |
223
+ | `passed_anyway` | Why a run that failed passed anyway: `"accept_label"`, where the pull request's label accepted the regression (see [`c.accept_label`](configuration.md#accept-a-regression-with-a-label)), or `"fail_on"`, where only a pull request fails (see `c.fail_on`). Otherwise `null`. |
225
224
  | `message` | What Duel said, for nothing to compare or an error, such as `"duel: no release tag (v*) before HEAD to compare against"`. Otherwise `null`. |
226
- | `comment` | Whether the workflow that `duel init` writes comments on a pull request with the report, as [`c.comment`](configuration.md#choose-when-to-comment-on-a-pull-request) (or `--comment`) decides for this run: `"post"` a new comment, `"change"`, which posts one only if the workflow commented on the pull request before and its last comment's status differs from this run's, or `"none"`. See [Comment on a pull request](ci.md#comment-on-a-pull-request). |
227
225
  | `settings` | The settings that the run used (see below). |
228
226
  | `seconds` | How long the comparison took. `null` without a comparison. |
229
227
  | `baseline`, `candidate` | The two versions (see below). `null` without a comparison. |
230
228
  | `plugins` | The names of the plugins in use, such as `["rails", "activerecord"]`. |
231
229
  | `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
232
230
  | `across` | The verdicts across the board, by metric (see below). |
233
- | `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
234
231
  | `cut_short` | What the run left undone when it reached its time limit (`--time-limit`): `{"unmeasured": [names], "unconfirmed": [names], "across": true or false}`. `unmeasured` are the benchmarks that it didn't measure, which are errors. `unconfirmed` are those that it judged without measuring them again as often as it usually does, and `across` is whether it judged across the board without measuring every benchmark again. `null` for a run that finished within its limit. |
235
232
 
236
233
  A run fails when a benchmark regressed, a benchmark failed, or a metric
@@ -268,13 +265,12 @@ was slower across the board and confirmed (`"confirmed": true`).
268
265
  | `verdict` | `"regressed"`, `"error"`, `"unsettled"` (a timing too close to its limit to settle), `"faster"`, `"changed"` (a count changed), `"skipped"`, or `"same"`, as in the Markdown report's table. |
269
266
  | `passed` | Whether it regressed on nothing and didn't fail. |
270
267
  | `regressed` | The metrics that it regressed on, in every measurement, such as `["cpu", "allocations"]`. Peak memory is `"peak_memory"`. |
271
- | `unsettled` | Whether a timing was too close to its limit to settle. |
272
268
  | `iterations` | The iterations in each sample. `null` for a benchmark that failed or was skipped. |
273
269
  | `seconds` | How long its first measurement took. `null` for a benchmark that failed or was skipped. |
274
270
  | `confirmations` | How many times it was measured again, from scratch, because it looked worse. |
275
271
  | `timed` | Its timings, by metric (see below). |
276
272
  | `counted` | Its counts, by metric (see below). |
277
- | `peak_memory` | Each version's peak memory, as `{"baseline": bytes, "candidate": bytes, "unit": "B", "worse": true or false}`, for a benchmark that measures it. Otherwise `null`. |
273
+ | `peak_memory` | Each version's peak memory, as `{"baseline": bytes, "candidate": bytes, "unit": "B", "verdict": "more" or "same"}`, for a benchmark that measures it: `"more"` by more than Duel tolerates. Otherwise `null`. |
278
274
  | `details` | What metrics said of the last samples, by metric and then by version, such as `{"allocations": {"baseline": "2 String", "candidate": "3 String"}}`. |
279
275
  | `explanation` | For a regression, the lines that explain it, as the terminal report and the Markdown report show them. Otherwise `[]`. |
280
276
  | `error` | Why it failed, with any backtrace. Otherwise `null`. |
@@ -289,7 +285,7 @@ Each timing in `timed`, such as `cpu`:
289
285
  | `limit` | The ratio above which it's slower. |
290
286
  | `pairs` | The pairs of samples measured. |
291
287
  | `processes` | The worker processes of each version that the interval is over, or `null` where it's over the pairs: for a process benchmark, or with fewer workers than the interval takes (see [How it works](how-it-works.md)). |
292
- | `verdict` | `"slower"`, `"faster"`, `"same"`, or `"undecided"` (too close to the limit to settle). |
288
+ | `verdict` | `"slower"`, `"faster"`, `"same"`, or `"unsettled"` (too close to the limit to settle). |
293
289
  | `baseline`, `candidate` | Each version's median for one iteration, in `unit`. |
294
290
  | `unit`, `plural` | The metric's unit, singular and plural, such as `"ns"`, or `null` for a plain number. |
295
291
  | `across` | Whether it counts across the board. |
@@ -300,7 +296,7 @@ Each count in `counted`, such as `allocations`:
300
296
  |---|---|
301
297
  | `baseline`, `candidate` | Each version's fewest for one iteration, in `unit`. |
302
298
  | `unit`, `plural` | The metric's unit, singular and plural, such as `"query"` and `"queries"`, or `null` for a plain number. |
303
- | `worse` | Whether the candidate's count is worse, by more than the metric tolerates. |
299
+ | `verdict` | `"more"` (worse: more than the metric tolerates), `"fewer"`, or `"same"` (or more, within what the metric tolerates). |
304
300
 
305
301
  ### Across the board
306
302
 
@@ -313,5 +309,5 @@ Each metric in `across`, such as `cpu`:
313
309
  | `limit` | The ratio above which it's slower across the board. |
314
310
  | `benchmarks` | The benchmarks that it's from. |
315
311
  | `processes` | The worker processes of each version that the interval is over, or `null` where it's over the benchmarks. |
316
- | `verdict` | `"slower"`, `"faster"`, `"same"`, or `"undecided"`. |
312
+ | `verdict` | `"slower"`, `"faster"`, `"same"`, or `"unsettled"`. |
317
313
  | `confirmed` | Whether it was slower again when every benchmark was measured a second time, which fails the run. |
data/docs/plugins.md CHANGED
@@ -85,7 +85,7 @@ When you write a plugin, follow these guidelines:
85
85
  a server takes more as it's used, so Duel counts at least this much
86
86
  for each worker. A version of Duel before 0.1.8 ignores it.
87
87
  - If a worker's server listens on a Unix socket in `w.dir`, give the
88
- plugin an option to listen on `w.port` instead, and default it to
88
+ plugin an option to listen on `w.free_port` instead, and default it to
89
89
  where `Duel.unix_sockets?` is false, for example
90
90
  `def self.configure(config, tcp: !Duel.unix_sockets?)`. It's false on
91
91
  Windows, where a server's own build often has Unix sockets that Ruby
@@ -97,6 +97,23 @@ When you write a plugin, follow these guidelines:
97
97
  `Duel::Platform` or `RbConfig` yourself: Duel's internals can change
98
98
  in any release, and `Duel.windows?` won't. A version of Duel before
99
99
  0.2.0 doesn't have it.
100
+ - If the plugin starts a server program, find it with
101
+ `Duel.find_program(server)`, which takes a path (anything with a `/`
102
+ or a `\` in it, expanded) as it is, and looks for a name on the
103
+ `PATH`, adding `.exe` on Windows. It returns `nil` if there's none,
104
+ and `Duel.path?(server)` tells the plugin's error whether to say it
105
+ isn't on the `PATH`, for example
106
+ `Duel.find_program(server) || w.fail!("no #{server}#{" on the PATH" unless Duel.path?(server)}")`.
107
+ A version of Duel before 0.3.0 doesn't have them.
108
+ - Give a plugin that starts a server the options that Duel's own server
109
+ plugins share, by the same names:
110
+ - `server:`, its program (by default `DUEL_<NAME>_SERVER`, or the
111
+ program's name), found with `Duel.find_program`.
112
+ - `settings:`, a Hash of more of the server's settings, by the
113
+ server's own names, passed after the plugin's own so that they win.
114
+ - `env:`, the variables given its address besides the plugin's own
115
+ `DUEL_<NAME>_*` one.
116
+ - `tcp:` (see above).
100
117
  - To read a project's settings outside a run, as a generator that writes
101
118
  its files might, load its `.duel.rb` with `Duel.load_config(path)`,
102
119
  which returns its configuration. Before 0.2.0, it was private.
@@ -104,7 +121,7 @@ When you write a plugin, follow these guidelines:
104
121
  > [!IMPORTANT]
105
122
  > A helper that's meant for setup, such as one that runs a query, must
106
123
  > call `Duel.unmeasured!` first, for example
107
- > `Duel.unmeasured!("Duel::Redis.flush")`. It raises an error during a
124
+ > `Duel.unmeasured!("Duel::Redis.reset")`. It raises an error during a
108
125
  > timed stretch (when `Duel.measuring?` is true), where what the helper
109
126
  > does would be counted. That includes the block of
110
127
  > `Duel::Testing.measure`, but not the block's `untimed` blocks.
@@ -190,11 +207,10 @@ A plugin's hooks are the configuration's, so they run before a file's
190
207
  and a benchmark's own, in the order that they were given, and they run
191
208
  where they're written, with the plugin's module as `self`:
192
209
 
193
- - `c.before_sample { |benchmark| ... }` runs before every sample,
194
- warm-up samples included, and isn't measured. Use it to put back what
195
- earlier samples changed, such as reloading a database's fixtures, so
196
- that every sample starts alike. The block receives the benchmark,
197
- with its `name`, `only`, and `peak_memory`, if it takes it.
210
+ - `c.before_sample { ... }` runs before every sample, warm-up samples
211
+ included, and isn't measured. Use it to put back what earlier samples
212
+ changed, such as reloading a database's fixtures, so that every
213
+ sample starts alike.
198
214
  - `c.after_iterations { ... }` runs after each sample's iterations, and
199
215
  is the one hook that Duel measures, with them. Use it to do the work
200
216
  that the iterations started and left to be done later, such as jobs
@@ -274,8 +290,9 @@ returns. The block receives the following:
274
290
  it has one stretch, and one more for each `untimed` block, as any
275
291
  other sample does.
276
292
  - `sample`: the sample. `sample.n` is its number of iterations, and
277
- `sample.benchmark` is the benchmark, with its `name`, `only`, and
278
- `peak_memory`.
293
+ `sample.benchmark` is the benchmark, with its `name`, and what it's
294
+ judged on: `judge` (`[:timed, :counted]`, say), and `timed?`,
295
+ `counted?` and `peak_memory?`.
279
296
 
280
297
  A reading has the following:
281
298
 
@@ -339,16 +356,17 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
339
356
  way that a worker does, and which ends after the sample. It runs the
340
357
  configuration's hooks as a worker runs a benchmark's. Its block
341
358
  receives the benchmark's context, with `untimed`, as a benchmark's
342
- does: `measure(config) { |s| s.untimed { ... } }`. `n:` is the
343
- sample's number of iterations, and `only:` makes the sample a
344
- benchmark's that has, for example, `only: :timing`. It also takes the
359
+ does: `measure(config) { |s| s.untimed { ... } }`. The block is one
360
+ iteration, and `measure` runs it `n:` times (once by default), as a
361
+ worker runs a benchmark's block that takes no `n`. With an
362
+ `around_iteration` hook, the configuration's or its own, each
363
+ iteration runs inside the hook, the way that a worker runs a
364
+ benchmark with one, including what it learns first to subtract the
365
+ hook's work from a costly metric. `judge:` makes the sample a
366
+ benchmark's that has, for example, `judge: :timed`. It also takes the
345
367
  benchmark's own hooks, as `duel` does, such as `before_sample:`. It
346
368
  returns a Hash of what was measured, by name, whose `details` are
347
- what the metrics said of the sample, by measure. With an
348
- `around_iteration` hook, the configuration's or its own, the block is
349
- one iteration: `measure` runs it `n` times, each inside the hook, the
350
- way that a worker runs a benchmark with one, including what it learns
351
- first to subtract the hook's work from a costly metric.
369
+ what the metrics said of the sample, by measure.
352
370
  - `in_worker(config) { ... }`: runs the block as a worker, and ends it
353
371
  the way that a worker ends, so what its `c.in_worker` blocks gave
354
372
  `at_stop` runs after the block. Every `measure` in the block is that
@@ -361,8 +379,8 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
361
379
  - `compare(base, candidate)`: compares two projects in workers. See the
362
380
  following paragraphs.
363
381
  - `sample(n: 1)`: a stand-in sample of `n` iterations, which measures
364
- nothing, for a metric's `start_sample` or a `details` block. `name:`,
365
- `only:`, and `peak_memory:` describe its benchmark.
382
+ nothing, for a metric's `start_sample` or a `details` block. `name:`
383
+ and `judge:` describe its benchmark.
366
384
  - `reading(totals, context)`: a stand-in reading of running totals, by
367
385
  name, with the context that was read with them. A sample's stretches
368
386
  are pairs of readings.
@@ -389,7 +407,7 @@ rolled_back = lambda do |iteration|
389
407
  pg.exec("ROLLBACK")
390
408
  end
391
409
  Duel::Testing.measure(config, n: 20, around_iteration: rolled_back) { pg.exec("INSERT INTO items VALUES (1)") }
392
- # => {"postgres_queries" => 20, "postgres_rows" => 20, ...}
410
+ # => {"postgresql_queries" => 20, "postgresql_rows" => 20, ...}
393
411
  ```
394
412
 
395
413
  To test or benchmark what a metric says of a sample, without a worker,
@@ -405,7 +423,7 @@ A plugin's tests find what the plugin added in `config.metrics`, the
405
423
  metrics in order. Each metric has its `measures` and their `settings`,
406
424
  with every setting filled in. For example,
407
425
  `config.metrics.first.settings.fetch("redis_cpu")` is
408
- `{kind: :ratio, unit: "µs", floor: 300, ...}`.
426
+ `{kind: :timed, unit: "µs", floor: 300, ...}`.
409
427
 
410
428
  `compare` takes `duel`'s options as keywords: `processes:` (two unless
411
429
  given, too few for an interval over workers, so each timing is judged
@@ -424,7 +442,7 @@ printed), and its `results`, one for each benchmark.
424
442
  version's median (`"baseline"` and `"candidate"`), by name.
425
443
  - Each `counted` measure's fewest on each version (`"baseline"` and
426
444
  `"candidate"`) over a sample's `"n"` iterations, with its
427
- `"tolerance"` and `"allowance"` (both with their `"unit"` and
445
+ `"per_iteration"` and `"per_sample"` (both with their `"unit"` and
428
446
  `"plural"`), by name.
429
447
  - What metrics said of it (`details`), by measure and by version.
430
448
  - Its peak `memory`, by version, when it was asked for.
@@ -14,7 +14,8 @@ what a change must touch to be compared, set `c.watch`. See
14
14
 
15
15
  Against `--base last-release`, Duel also reports that there's nothing to
16
16
  compare against until you tag your first release. Releases are tagged
17
- `v*` by default. To match other tags, set `c.release_tags`.
17
+ `v*` by default. To match other tags, set `c.release_tags`, to a glob or
18
+ a list of them (`%w[v* my_gem-v*]`, for a gem that renamed its tags).
18
19
 
19
20
  ## Duel finds no main branch, or no commits
20
21
 
@@ -125,7 +126,7 @@ Duel reports a timing that's still too close to the limit to call, and
125
126
  doesn't fail on it. A benchmark that's reported unsettled run after run
126
127
  is noisy. Make every iteration do the same work, or, for work that's
127
128
  mostly waiting, judge the benchmark on its counts alone with
128
- `only: :count`. See
129
+ `judge: :counted`. See
129
130
  [Choose what a benchmark is judged on](benchmarks.md#choose-what-a-benchmark-is-judged-on)
130
131
  and [How Duel tells a regression from noise](how-it-works.md#how-duel-tells-a-regression-from-noise).
131
132