duel 0.1.1 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -17,6 +17,9 @@ Duel.configure do |c|
17
17
  c.release_tags = "release-*" # default: v*
18
18
  c.ruby = :version_file # each version's own Ruby (default: the runner's)
19
19
  c.comment = :results # when CI comments on a pull request (default: :failures)
20
+ c.fail_on = :pull_requests # which runs a regression fails (default: :every_run)
21
+ c.push_base = :last_passed # what CI compares a push with (default: :before)
22
+ c.accept_label = "duel: accept" # a pull request label that accepts a regression
20
23
  c.use :redis # a plugin (see Use plugins)
21
24
 
22
25
  # How a worker loads one version of your code, instead of the library.
@@ -90,6 +93,83 @@ with a nonzero status, and the job's summary still has the report.
90
93
  `c.comment` takes only these four values: any other fails the run, and
91
94
  says which values it takes.
92
95
 
96
+ ## Fail only a pull request
97
+
98
+ By default, a regression fails every run: a pull request's, a push's,
99
+ and the schedule's. In a project where people push to `main` directly,
100
+ a failed push only gets GitHub's email that the run failed. To report a
101
+ regression in a push without failing it, set `c.fail_on`, and
102
+ `duel --fail-on` overrides it:
103
+
104
+ | Value | Which runs a regression fails |
105
+ |---|---|
106
+ | `:every_run` | Every run. This is the default. |
107
+ | `:pull_requests` | A pull request's (or a merge queue's), and a run outside GitHub Actions. |
108
+
109
+ ```ruby
110
+ Duel.configure do |c|
111
+ c.fail_on = :pull_requests
112
+ end
113
+ ```
114
+
115
+ With `:pull_requests`, a regression on GitHub Actions outside a pull
116
+ request (a push, the schedule, or a run by hand) passes, with a warning
117
+ in the run's summary, and the job's summary still has the report. On
118
+ GitLab CI, a regression outside a merge request exits with status 3,
119
+ which the [job](ci.md#run-duel-on-gitlab-ci) that `duel init --gitlab`
120
+ writes shows as a warning. The
121
+ [GitHub Actions workflow](ci.md#comment-on-a-pushed-commit) that
122
+ `duel init` writes then comments on a pushed commit with the report,
123
+ as `c.comment` says, which notifies the commit's author. A run that
124
+ couldn't compare fails either way, and on your machine, a regression
125
+ always fails.
126
+
127
+ ## Accept a regression with a label
128
+
129
+ Sometimes a change is slower on purpose: it's correct where the old
130
+ code wasn't, say. To let a pull request through anyway, name a label
131
+ that accepts its regression:
132
+
133
+ ```ruby
134
+ Duel.configure do |c|
135
+ c.accept_label = "duel: accept"
136
+ end
137
+ ```
138
+
139
+ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
140
+ that `duel init` writes passes Duel the pull request's labels. With
141
+ that label, a regression passes, with a warning, and the report says
142
+ that the label accepted it. The labels are read when the workflow runs,
143
+ so after you add the label, run the workflow again (Re-run jobs, on the
144
+ pull request's Checks tab). For a push, Duel reads the labels of the
145
+ pull request that merged it, so with
146
+ [`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
147
+ the merged regression becomes the commit that later pushes are compared
148
+ with. By default, there's no such label.
149
+
150
+ ## Compare a push with the last commit that passed
151
+
152
+ The [GitHub Actions workflow](ci.md#compare-a-push-with-the-last-commit-that-passed)
153
+ that `duel init` writes compares a push to `main` with the commit
154
+ before it. A series of pushes that are each a little slower then each
155
+ pass. `c.push_base` chooses what a push is compared with:
156
+
157
+ | Value | What the workflow compares a push with |
158
+ |---|---|
159
+ | `:before` | The commit before the push. This is the default. |
160
+ | `:last_passed` | The last commit that Duel passed, which a branch marks: `duel/passed/main` for `main`. |
161
+
162
+ ```ruby
163
+ Duel.configure do |c|
164
+ c.push_base = :last_passed
165
+ end
166
+ ```
167
+
168
+ With `:last_passed`, the slowdowns add up until they fail a push. After
169
+ a regression, every push fails until one is no slower, or until you
170
+ move the branch to accept the regression:
171
+ `git push origin main:duel/passed/main`.
172
+
93
173
  ## Prepare each worker
94
174
 
95
175
  Each worker can get things of its own, such as a server, so that the
@@ -121,7 +201,7 @@ A prepare block can use the following methods:
121
201
  | `w.failed("message")` | Fails the same way, for any other reason. |
122
202
  | `w.at_stop { ... }` | Runs the block when the worker stops. |
123
203
  | `w.cached(key) { \|dir\| ... }` | Makes something once, for every worker. See the following example. |
124
- | `w.name` | The worker's name, for example `"baseline-1"`. |
204
+ | `w.name` | The worker's name, for example `"base-1"`. |
125
205
  | `w.dir` | The worker's directory. `w.path("name")` returns the path of a file in it. |
126
206
  | `w.root` | The tree of the version that the worker runs. |
127
207
  | `w.ruby` | The path of the Ruby that the version runs on. See [Run each version on its own Ruby](#run-each-version-on-its-own-ruby). |
@@ -143,7 +223,11 @@ Some things take long to make, and are the same for every worker, such
143
223
  as a database's initial files. `w.cached` makes them once. The first
144
224
  worker that asks builds the directory, and the other workers, in this
145
225
  run and in later ones, get its path, to copy or to start from. The key
146
- names what's cached, so put a version in it. `duel --clean` removes
226
+ names what's cached, so put a version in it. If the key ends with a hash
227
+ of what the build depends on, such as `postgres-16.2-3f9c2a1b`, then
228
+ when a run ends, Duel removes the other builds of that name, the part
229
+ before the version and the hash, so an upgrade doesn't leave the old
230
+ build behind. Each build that the run used stays. `duel --clean` removes
147
231
  every cached directory.
148
232
 
149
233
  ```ruby
@@ -187,8 +271,11 @@ Duel.configure do |c|
187
271
  end
188
272
  ```
189
273
 
190
- `v.name` is `"baseline"` or `"candidate"`, and `v.root` is the version's
191
- directory. What the block spawns stops when the run ends. A worker's
274
+ `v.name` is `"base"` (the baseline) or `"head"` (the candidate), and
275
+ `v.root` is the version's directory. The two names are the same length
276
+ on purpose: where a process's stack starts depends on how long its
277
+ arguments and environment are, so a name one byte longer can make every
278
+ worker of one version measurably slower. What the block spawns stops when the run ends. A worker's
192
279
  prepare blocks get the version's environment too, as `w.env`. If a
193
280
  worker's prepare block sets a variable that the version's environment
194
281
  sets to a different value, the run fails.
data/docs/how-it-works.md CHANGED
@@ -73,7 +73,7 @@ Duel judges every benchmark on the following metrics:
73
73
  | `cpu` | The benchmark thread's CPU time (or the process's) | More than 2% slower, with 99% confidence |
74
74
  | `instructions` | Instructions retired, from the CPU's own counters | More than 1% more, with 99% confidence |
75
75
  | `allocations` | Ruby objects allocated (on CRuby) | One more per two iterations |
76
- | `peak_memory` | The most resident memory, above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
76
+ | `peak_memory` | The most resident memory (on JRuby, the JVM's heap in use), above the start, that measured work reached (not `once` or `untimed`), with garbage collection running as usual (when asked for) | More than 10% and 1 MB more |
77
77
  | Yours | See [Add your own metrics](configuration.md#add-your-own-metrics) | |
78
78
 
79
79
  Instruction counts barely vary from run to run, so they catch slowdowns
@@ -140,14 +140,19 @@ iteration, as they are in the explanation of a regression.
140
140
  The main branch is the one that `origin/HEAD` names. Without
141
141
  `origin/HEAD`, Duel uses `origin/main`, and then `origin/master`. In a
142
142
  repository without a remote, it uses `main`, and then `master`. To name
143
- another branch, set `c.main_branch`.
143
+ another branch, set `c.main_branch`. On GitHub Actions, for a pull
144
+ request, the main branch is the one that the pull request targets
145
+ (`GITHUB_BASE_REF`), such as a stable branch, or another pull request's
146
+ branch in a stack of them.
144
147
 
145
148
  Duel checks out a git baseline as a git worktree next to your project,
146
149
  under a hidden name that's as long as your project's directory name, for
147
150
  example `.3f9c2a1b` next to `my_parser`. Duel keeps it there for the
148
151
  next run. If your project has an `ext/` directory, Duel builds the
149
152
  baseline with `bundle exec rake compile`. To remove the baseline, git's
150
- record of it, and the rest of `tmp/duel`, run `duel --clean`.
153
+ record of it, and the rest of `tmp/duel` (see
154
+ [What Duel keeps in tmp/duel](#what-duel-keeps-in-tmpduel)), run
155
+ `duel --clean`.
151
156
 
152
157
  If your project is in a subdirectory of its repository, as in a
153
158
  monorepo, run `duel` in that subdirectory. Duel checks out the whole
@@ -169,6 +174,27 @@ that small. If Duel can't write next to your project, or something else
169
174
  already has that name, it uses `tmp/duel/base` instead. Whenever the two
170
175
  versions' paths differ in length, Duel says so.
171
176
 
177
+ ### What Duel keeps in tmp/duel
178
+
179
+ Duel keeps what it needs between runs in your project's `tmp/duel`:
180
+
181
+ - `lock`, so that only one run at a time compares the project. A run
182
+ removes it when it ends.
183
+ - `cache/`, what plugins build once and reuse, such as a database's
184
+ initial files. A build is named for what it is, its version, and a
185
+ hash, such as `mysql-0.1.2-3f9c2a1b7e5d4c6a`. When a run ends, Duel
186
+ removes the builds of the same name that the ones it used replace,
187
+ such as the files of a database server that you've since upgraded, so
188
+ the cache keeps one build of each.
189
+ - `base/`, the baseline's checkout, when it can't go next to your
190
+ project.
191
+
192
+ When a run ends, Duel removes `tmp/duel` if nothing is left in it, and
193
+ `tmp` too if Duel made it. A project without plugins that build
194
+ something keeps nothing there. `duel --clean` removes the rest. Each
195
+ run's sockets and server directories are in a temporary directory,
196
+ which the run removes when it ends.
197
+
172
198
  If nothing under `lib/` or `ext/`, your benchmarks, or `.duel.rb`
173
199
  changed since the baseline, there's nothing to compare, and Duel says
174
200
  so. To compare anyway, use `--force`. See
@@ -204,6 +230,9 @@ Duel runs each benchmark for five seconds before it measures it
204
230
  (`--warm`). Their timings stay noisier than CRuby's, so Duel often
205
231
  leaves small timing regressions on them unsettled.
206
232
 
207
- Neither has an allocation count, an instruction count, or profiles, and
208
- each has peak memory only on Linux. StackProf runs on CRuby only: Duel
233
+ Neither has an allocation count, an instruction count, or profiles.
234
+ TruffleRuby has peak memory only on Linux. JRuby measures peak memory in
235
+ the JVM's heap in use, on any operating system, because the JVM takes
236
+ memory for its heap ahead of time, so resident memory barely moves as
237
+ a benchmark's objects fill it. StackProf runs on CRuby only: Duel
209
238
  doesn't use it on JRuby or TruffleRuby, even where it's installed.
data/docs/json.md CHANGED
@@ -227,6 +227,7 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
227
227
  | `benchmarks` | Every benchmark compared, in the order Duel found them (see below). |
228
228
  | `across` | The verdicts across the board, by metric (see below). |
229
229
  | `errors` | The benchmarks that failed, each as `{"benchmark": name, "message": error}`. |
230
+ | `cut_short` | What the run left undone when it reached its time limit (`--time-limit`): `{"unmeasured": [names], "unconfirmed": [names], "across": true or false}`. `unmeasured` are the benchmarks that it didn't measure, which are errors. `unconfirmed` are those that it judged without measuring them again as often as it usually does, and `across` is whether it judged across the board without measuring every benchmark again. `null` for a run that finished within its limit. |
230
231
 
231
232
  A run fails when a benchmark regressed, a benchmark failed, or a metric
232
233
  was slower across the board and confirmed (`"confirmed": true`).
@@ -241,6 +242,7 @@ was slower across the board and confirmed (`"confirmed": true`).
241
242
  | `jobs` | Benchmarks measured at once (`--jobs`). |
242
243
  | `processes` | Worker processes for each version (`--processes`). |
243
244
  | `timeout` | The seconds that a worker may take to load or to answer (`--timeout`). |
245
+ | `time_limit` | The seconds that a run measures for, at most (`--time-limit`). |
244
246
  | `warm` | The seconds of warm-up for each benchmark (`--warm`), or `null` for the default, which depends on the Ruby. |
245
247
  | `force` | Whether `--force` was given. |
246
248
 
data/docs/plugins.md CHANGED
@@ -303,10 +303,10 @@ Duel::Testing.compare(base, candidate).passed? # whole comparisons, in workers
303
303
  `Duel::Testing` has the following methods:
304
304
 
305
305
  - `prepare(config)`: runs the configuration's `c.prepare(:version)`
306
- blocks, named for the version (`"candidate"`), and then its
306
+ blocks, named for the version (`"head"`), and then its
307
307
  `c.prepare(:worker)` blocks, as Duel prepares a worker. It sets the
308
308
  environment that they return while its block runs, and stops what
309
- they started after. `name:` names the worker (`"candidate-1"` by
309
+ they started after. `name:` names the worker (`"head-1"` by
310
310
  default), and `root:` is the version's tree (the configuration's root
311
311
  by default). `cache:` is where `w.cached` builds go. By default,
312
312
  that's one directory for the whole test process. To look inside it,
@@ -48,6 +48,20 @@ worker and reports the benchmark as an error. To change the limit, use
48
48
  `--timeout`. If a benchmark's iterations are slow, split it into
49
49
  smaller benchmarks.
50
50
 
51
+ ## The run reached its time limit
52
+
53
+ A run measures for 30 minutes at most (`--time-limit`). Past that, Duel
54
+ starts no more measurements: it fails the benchmarks it hasn't measured,
55
+ and judges each benchmark that looked worse on the measurements it has.
56
+ The report starts the list of what it left undone with `TIME LIMIT`.
57
+
58
+ A run usually takes that long because a benchmark is slow to settle. A
59
+ benchmark that starts processes of its own, such as one that forks
60
+ workers, spends most of its time waiting for them, and Duel measures
61
+ each one again when it looks worse. Give it less work per call, or
62
+ fewer calls, so that each sample takes less time. If your benchmarks
63
+ need more time, raise the limit with `--time-limit SECONDS`.
64
+
51
65
  ## Duel tells you to set c.build
52
66
 
53
67
  Duel checks that each version's workers load only native extensions that
data/lib/duel/baseline.rb CHANGED
@@ -11,12 +11,15 @@ module Duel
11
11
  # git commit checked out (and built, see Config#build) where #checkout
12
12
  # says. Settings#base names the commit:
13
13
  #
14
- # - nil: where HEAD left the main branch (their merge base):
14
+ # - nil: where HEAD left the main branch (their merge base): for a
15
+ # pull request, the branch it targets (Settings#target), or else
15
16
  # Config#main_branch, or the one origin/HEAD names, or origin/main, or
16
17
  # origin/master, or (a repository without a remote) main, or master
17
18
  # - "last-release": the newest release tag (Config#release_tags) before
18
19
  # HEAD, to catch slowdowns too small for any one change to fail but
19
20
  # that add up between releases
21
+ # - a push's (Settings#pushed, the branch pushed): the commit before it, or where
22
+ # Config#push_base says, the last commit Duel passed (#passed)
20
23
  # - anything else: a git ref
21
24
  # @api private
22
25
  class Baseline
@@ -29,17 +32,29 @@ module Duel
29
32
  @settings = settings
30
33
  end
31
34
 
32
- # The commit, or nil for "last-release" with no release yet.
35
+ # The commit, or nil for "last-release" with no release yet, and for
36
+ # a push with no commit before it here (see #pushed).
33
37
  def ref
34
38
  return @ref if instance_variable_defined?(:@ref)
35
39
 
36
40
  @ref = case @settings.base
37
41
  when nil then merge_base
38
42
  when LAST_RELEASE then git("describe", "--tags", "--abbrev=0", "--match", @config.release_tags, "HEAD^", allow_failure: true)
39
- else @settings.base
43
+ else @settings.pushed ? pushed : @settings.base
40
44
  end
41
45
  end
42
46
 
47
+ # The branch that marks the last commit Duel passed in a push to the
48
+ # branch pushed (Settings#pushed), for Config#push_base:
49
+ # duel/passed/main, or for a project in a subdirectory of its
50
+ # repository, with its path (duel/passed-gems-parser/main), so each
51
+ # branch, and each of a monorepo's projects, has its own.
52
+ def passed_branch = "#{prefix.empty? ? "duel/passed" : "duel/passed-#{prefix.tr("/", "-")}"}/#{@settings.pushed}"
53
+
54
+ # The last commit Duel passed in a push, as checked out from GitHub
55
+ # (#passed_branch, on origin).
56
+ def passed = "origin/#{passed_branch}"
57
+
43
58
  # Whether nothing #watched changed since the baseline (only for a git
44
59
  # baseline compared with the working tree).
45
60
  def unchanged?
@@ -109,20 +124,20 @@ module Duel
109
124
  sibling = File.join(File.dirname(top), ".#{Digest::SHA256.hexdigest(root).ljust(length, "0")[0, length]}")
110
125
  return sibling if File.writable?(File.dirname(top)) && (!File.exist?(sibling) || ours?(sibling))
111
126
 
112
- File.join(@config.root, "tmp", "duel", "base")
127
+ File.join(@config.state, "base")
113
128
  end
114
129
 
115
- # Removes what Duel keeps: tmp/duel, and the baseline's checkout (and
116
- # git's record of it as a worktree).
130
+ # Removes what Duel keeps (Config#state: tmp/duel), and the baseline's
131
+ # checkout (and git's record of it as a worktree).
117
132
  def clean
118
- FileUtils.rm_rf(File.join(@config.root, "tmp", "duel"))
133
+ FileUtils.rm_rf(@config.state)
119
134
  FileUtils.rm_rf(checkout)
120
135
  git("worktree", "prune", allow_failure: true) # (not a git repository, say)
121
136
  end
122
137
 
123
138
  # The main branch (see #ref).
124
139
  def main_branch
125
- @config.main_branch ||
140
+ @settings.target || @config.main_branch ||
126
141
  git("symbolic-ref", "--short", "refs/remotes/origin/HEAD", allow_failure: true) ||
127
142
  BRANCHES.find { |branch| git("rev-parse", branch, allow_failure: true) } ||
128
143
  raise(Error, "no main branch to compare with (set c.main_branch in .duel.rb, or --base)")
@@ -178,6 +193,17 @@ module Duel
178
193
  raise Error, "HEAD and #{branch} have no commit in common: #{why}, or compare with --base REF"
179
194
  end
180
195
 
196
+ # A push's baseline: the last commit Duel passed (#passed), where
197
+ # Config#push_base says and HEAD has it, or else the commit before the
198
+ # push, or nil where that isn't here (a branch's first push, or one
199
+ # that rewrote history).
200
+ def pushed
201
+ return passed if @config.push_base == :last_passed && git("merge-base", "--is-ancestor", passed, "HEAD", allow_failure: true)
202
+
203
+ before = @settings.base #: String
204
+ before if git("cat-file", "-e", "#{before}^{commit}", allow_failure: true)
205
+ end
206
+
181
207
  # Whether +dir+ is a checkout of this repository (a worktree of it).
182
208
  def ours?(dir) = common_dir(dir) == common_dir(@config.root)
183
209