duel 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -2,19 +2,24 @@
2
2
 
3
3
  Most gems don't need any configuration: Duel's conventions cover them.
4
4
  Benchmarks are in `duel/**/*_duel.rb`, each version's `lib/` is first on
5
- the load path, a change is compared when it touches `lib/` or `ext/`,
6
- and releases are tagged `v*`. If your project differs, create a `.duel.rb` file
7
- at its root. Both the runner and every worker load it:
5
+ the load path, a change is compared when it touches `lib/` or `ext/`
6
+ (or the benchmarks, or `.duel.rb`), and releases are tagged `v*`. If
7
+ your project differs, create a `.duel.rb` file at its root. Both the
8
+ runner and every worker load it:
8
9
 
9
10
  ```ruby
10
11
  Duel.configure do |c|
11
12
  c.benchmarks = "perf/**/*_bench.rb" # default: duel/**/*_duel.rb
12
- c.watch = %w[lib ext] # what a change must touch to be compared
13
+ c.watch = %w[lib ext] # the code a change must touch to be compared
13
14
  c.library = "my_gem" # required from each version (default: the gemspec's)
14
15
  c.build = "make" # for both versions (default: rake compile, with ext/)
15
16
  c.main_branch = "origin/trunk"
16
17
  c.release_tags = "release-*" # default: v*
17
18
  c.ruby = :version_file # each version's own Ruby (default: the runner's)
19
+ c.comment = :results # when CI comments on a pull request (default: :failures)
20
+ c.fail_on = :pull_requests # which runs a regression fails (default: :every_run)
21
+ c.push_base = :last_passed # what CI compares a push with (default: :before)
22
+ c.accept_label = "duel: accept" # a pull request label that accepts a regression
18
23
  c.use :redis # a plugin (see Use plugins)
19
24
 
20
25
  # How a worker loads one version of your code, instead of the library.
@@ -28,6 +33,143 @@ directory. The configuration also takes hooks, such as
28
33
  `c.before_sample`. See
29
34
  [Run code around a benchmark](benchmarks.md#run-code-around-a-benchmark).
30
35
 
36
+ ## Choose what a change must touch to be compared
37
+
38
+ When nothing that Duel watches changed since the baseline, there's
39
+ nothing to compare, and Duel passes without running the benchmarks.
40
+ `c.watch` names your code: `lib/` and `ext/` by default. Duel also
41
+ watches what decides how your code is measured:
42
+
43
+ - The benchmarks' directory: the part of `c.benchmarks` before its first
44
+ wildcard, `duel/` by default, with its helpers and fixtures.
45
+ - The configuration's file, `.duel.rb`.
46
+ - With [`c.ruby = :version_file`](#run-each-version-on-its-own-ruby),
47
+ the version files at your project's root (`.ruby-version`,
48
+ `.tool-versions`, `mise.toml`, or `.mise.toml`).
49
+
50
+ Each version runs your branch's benchmarks and configuration, so a
51
+ change to only those compares your code with itself. That checks that
52
+ the changed benchmarks run, and pass on identical code, instead of
53
+ passing without running them. To compare a change to anything else, such
54
+ as your CI workflow, add it to `c.watch`:
55
+
56
+ ```ruby
57
+ Duel.configure do |c|
58
+ c.watch += %w[.github/workflows/perf.yml]
59
+ end
60
+ ```
61
+
62
+ ## Choose when to comment on a pull request
63
+
64
+ The [GitHub Actions workflow](ci.md#comment-on-a-pull-request) that
65
+ `duel init` writes comments on a pull request with Duel's report.
66
+ `c.comment` chooses when, and `duel --comment` overrides it:
67
+
68
+ | Value | When the workflow comments |
69
+ |---|---|
70
+ | `:always` | On every run, whatever the outcome. |
71
+ | `:results` | When the run passed or failed, but not when there was nothing to compare. |
72
+ | `:failures` | When a benchmark failed, or Duel couldn't compare. This is the default. |
73
+ | `:never` | Never. |
74
+
75
+ ```ruby
76
+ Duel.configure do |c|
77
+ c.comment = :results
78
+ end
79
+ ```
80
+
81
+ The workflow never edits a comment: each one is new, so the pull
82
+ request keeps them all. Once the workflow has commented on a pull
83
+ request, it also comments on a run that it otherwise wouldn't, except
84
+ with `:never`, when the run's status (passed, failed, or nothing to
85
+ compare) differs from its last comment's. So with `:failures`, after a
86
+ comment that a benchmark failed, the first run that passes posts a
87
+ comment saying so, and the runs after it post nothing. The last comment
88
+ is never stale.
89
+
90
+ With `:never`, a regression still fails the check, because `duel` exits
91
+ with a nonzero status, and the job's summary still has the report.
92
+
93
+ `c.comment` takes only these four values: any other fails the run, and
94
+ says which values it takes.
95
+
96
+ ## Fail only a pull request
97
+
98
+ By default, a regression fails every run: a pull request's, a push's,
99
+ and the schedule's. In a project where people push to `main` directly,
100
+ a failed push only gets GitHub's email that the run failed. To report a
101
+ regression in a push without failing it, set `c.fail_on`, and
102
+ `duel --fail-on` overrides it:
103
+
104
+ | Value | Which runs a regression fails |
105
+ |---|---|
106
+ | `:every_run` | Every run. This is the default. |
107
+ | `:pull_requests` | A pull request's (or a merge queue's), and a run outside GitHub Actions. |
108
+
109
+ ```ruby
110
+ Duel.configure do |c|
111
+ c.fail_on = :pull_requests
112
+ end
113
+ ```
114
+
115
+ With `:pull_requests`, a regression on GitHub Actions outside a pull
116
+ request (a push, the schedule, or a run by hand) passes, with a warning
117
+ in the run's summary, and the job's summary still has the report. On
118
+ GitLab CI, a regression outside a merge request exits with status 3,
119
+ which the [job](ci.md#run-duel-on-gitlab-ci) that `duel init --gitlab`
120
+ writes shows as a warning. The
121
+ [GitHub Actions workflow](ci.md#comment-on-a-pushed-commit) that
122
+ `duel init` writes then comments on a pushed commit with the report,
123
+ as `c.comment` says, which notifies the commit's author. A run that
124
+ couldn't compare fails either way, and on your machine, a regression
125
+ always fails.
126
+
127
+ ## Accept a regression with a label
128
+
129
+ Sometimes a change is slower on purpose: it's correct where the old
130
+ code wasn't, say. To let a pull request through anyway, name a label
131
+ that accepts its regression:
132
+
133
+ ```ruby
134
+ Duel.configure do |c|
135
+ c.accept_label = "duel: accept"
136
+ end
137
+ ```
138
+
139
+ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
140
+ that `duel init` writes passes Duel the pull request's labels. With
141
+ that label, a regression passes, with a warning, and the report says
142
+ that the label accepted it. The labels are read when the workflow runs,
143
+ so after you add the label, run the workflow again (Re-run jobs, on the
144
+ pull request's Checks tab). For a push, Duel reads the labels of the
145
+ pull request that merged it, so with
146
+ [`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
147
+ the merged regression becomes the commit that later pushes are compared
148
+ with. By default, there's no such label.
149
+
150
+ ## Compare a push with the last commit that passed
151
+
152
+ The [GitHub Actions workflow](ci.md#compare-a-push-with-the-last-commit-that-passed)
153
+ that `duel init` writes compares a push to `main` with the commit
154
+ before it. A series of pushes that are each a little slower then each
155
+ pass. `c.push_base` chooses what a push is compared with:
156
+
157
+ | Value | What the workflow compares a push with |
158
+ |---|---|
159
+ | `:before` | The commit before the push. This is the default. |
160
+ | `:last_passed` | The last commit that Duel passed, which a branch marks: `duel/passed/main` for `main`. |
161
+
162
+ ```ruby
163
+ Duel.configure do |c|
164
+ c.push_base = :last_passed
165
+ end
166
+ ```
167
+
168
+ With `:last_passed`, the slowdowns add up until they fail a push. After
169
+ a regression, every push fails until one is no slower, or until you
170
+ move the branch to accept the regression:
171
+ `git push origin main:duel/passed/main`.
172
+
31
173
  ## Prepare each worker
32
174
 
33
175
  Each worker can get things of its own, such as a server, so that the
data/docs/how-it-works.md CHANGED
@@ -140,7 +140,10 @@ iteration, as they are in the explanation of a regression.
140
140
  The main branch is the one that `origin/HEAD` names. Without
141
141
  `origin/HEAD`, Duel uses `origin/main`, and then `origin/master`. In a
142
142
  repository without a remote, it uses `main`, and then `master`. To name
143
- another branch, set `c.main_branch`.
143
+ another branch, set `c.main_branch`. On GitHub Actions, for a pull
144
+ request, the main branch is the one that the pull request targets
145
+ (`GITHUB_BASE_REF`), such as a stable branch, or another pull request's
146
+ branch in a stack of them.
144
147
 
145
148
  Duel checks out a git baseline as a git worktree next to your project,
146
149
  under a hidden name that's as long as your project's directory name, for
@@ -169,8 +172,10 @@ that small. If Duel can't write next to your project, or something else
169
172
  already has that name, it uses `tmp/duel/base` instead. Whenever the two
170
173
  versions' paths differ in length, Duel says so.
171
174
 
172
- If nothing under `lib/` or `ext/` changed since the baseline, there's
173
- nothing to compare, and Duel says so. To compare anyway, use `--force`.
175
+ If nothing under `lib/` or `ext/`, your benchmarks, or `.duel.rb`
176
+ changed since the baseline, there's nothing to compare, and Duel says
177
+ so. To compare anyway, use `--force`. See
178
+ [Choose what a change must touch to be compared](configuration.md#choose-what-a-change-must-touch-to-be-compared).
174
179
 
175
180
  Duel also builds the candidate before every run. Builds are incremental,
176
181
  so a build that's already done costs little. A stale or missing build,
data/docs/json.md CHANGED
@@ -47,6 +47,7 @@ A run that found one regression, abridged to two benchmarks:
47
47
  "status": "failed",
48
48
  "exit": 1,
49
49
  "message": null,
50
+ "comment": "post",
50
51
  "settings": {
51
52
  "confidence": 0.99,
52
53
  "limit": 1.02,
@@ -189,7 +190,8 @@ A run with nothing to compare has the same keys, with nothing compared:
189
190
  "duel_version": "0.1.0",
190
191
  "status": "nothing_to_compare",
191
192
  "exit": 0,
192
- "message": "duel: lib, ext unchanged since the baseline, so there's nothing to compare (--force to anyway)",
193
+ "message": "duel: lib, ext, duel, .duel.rb unchanged since the baseline, so there's nothing to compare (--force to anyway)",
194
+ "comment": "change",
193
195
  "settings": {"confidence": 0.99, "limit": 1.02, "filter": "", "jobs": 4, "processes": 3, "timeout": 60.0, "warm": null, "force": false},
194
196
  "seconds": null,
195
197
  "baseline": null,
@@ -217,6 +219,7 @@ jq -r '.benchmarks[] | select(.verdict == "regressed") | .name' tmp/duel.json
217
219
  | `status` | `"passed"`, `"failed"`, `"nothing_to_compare"`, or `"error"` (Duel couldn't compare). |
218
220
  | `exit` | The exit status of `duel`: `0` for passed or nothing to compare, `1` for failed, `2` for an error. |
219
221
  | `message` | What Duel said, for nothing to compare or an error, such as `"duel: no release tag (v*) before HEAD to compare against"`. Otherwise `null`. |
222
+ | `comment` | Whether the workflow that `duel init` writes comments on a pull request with the report, as [`c.comment`](configuration.md#choose-when-to-comment-on-a-pull-request) (or `--comment`) decides for this run: `"post"` a new comment, `"change"`, which posts one only if the workflow commented on the pull request before and its last comment's status differs from this run's, or `"none"`. See [Comment on a pull request](ci.md#comment-on-a-pull-request). |
220
223
  | `settings` | The settings that the run used (see below). |
221
224
  | `seconds` | How long the comparison took. `null` without a comparison. |
222
225
  | `baseline`, `candidate` | The two versions (see below). `null` without a comparison. |
@@ -247,7 +250,7 @@ was slower across the board and confirmed (`"confirmed": true`).
247
250
  |---|---|
248
251
  | `ref` | The baseline's git ref: the one `--base` named, the release tag for `--base last-release`, or the main branch, such as `"origin/main"`, by default, where the baseline is the commit where `HEAD` left that branch. `null` for a baseline directory. |
249
252
  | `commit` | The full hash of the commit compared. `null` for a baseline directory, or for a candidate outside a git repository. |
250
- | `uncommitted_changes` | The candidate only: whether what `c.watch` names has changes that aren't committed. `null` outside a git repository. |
253
+ | `uncommitted_changes` | The candidate only: whether what Duel watches (`c.watch`, the benchmarks, and the rest that a change must touch to be compared: see [Configure Duel](configuration.md)) has changes that aren't committed. `null` outside a git repository. |
251
254
  | `root` | The tree that was compared, as an absolute path. |
252
255
  | `ruby` | The version's Ruby, as its `RUBY_DESCRIPTION`. |
253
256
 
@@ -5,11 +5,12 @@ comparison doesn't go as you expect, and what to do about it.
5
5
 
6
6
  ## Duel says there's nothing to compare
7
7
 
8
- If nothing under `lib/` or `ext/` changed since the baseline, there's
9
- nothing to compare, and Duel passes. To compare anyway, use `--force`.
10
- With nothing changed, Duel compares your code with itself, which checks
11
- that your benchmarks run. To change what a change must touch to be
12
- compared, set `c.watch`. See [Configure Duel](configuration.md).
8
+ If nothing under `lib/` or `ext/`, your benchmarks, or `.duel.rb`
9
+ changed since the baseline, there's nothing to compare, and Duel passes.
10
+ To compare anyway, use `--force`. With nothing changed, Duel compares
11
+ your code with itself, which checks that your benchmarks run. To change
12
+ what a change must touch to be compared, set `c.watch`. See
13
+ [Choose what a change must touch to be compared](configuration.md#choose-what-a-change-must-touch-to-be-compared).
13
14
 
14
15
  Against `--base last-release`, Duel also reports that there's nothing to
15
16
  compare against until you tag your first release. Releases are tagged
data/lib/duel/baseline.rb CHANGED
@@ -4,18 +4,22 @@ require "digest"
4
4
  require "fileutils"
5
5
  require "open3"
6
6
  require_relative "../duel"
7
+ require_relative "rubies"
7
8
 
8
9
  module Duel
9
10
  # The version to compare against: a directory (Settings#base_dir), or a
10
11
  # git commit checked out (and built, see Config#build) where #checkout
11
12
  # says. Settings#base names the commit:
12
13
  #
13
- # - nil: where HEAD left the main branch (their merge base):
14
+ # - nil: where HEAD left the main branch (their merge base): for a
15
+ # pull request, the branch it targets (Settings#target), or else
14
16
  # Config#main_branch, or the one origin/HEAD names, or origin/main, or
15
17
  # origin/master, or (a repository without a remote) main, or master
16
18
  # - "last-release": the newest release tag (Config#release_tags) before
17
19
  # HEAD, to catch slowdowns too small for any one change to fail but
18
20
  # that add up between releases
21
+ # - a push's (Settings#pushed, the branch pushed): the commit before it, or where
22
+ # Config#push_base says, the last commit Duel passed (#passed)
19
23
  # - anything else: a git ref
20
24
  # @api private
21
25
  class Baseline
@@ -28,28 +32,53 @@ module Duel
28
32
  @settings = settings
29
33
  end
30
34
 
31
- # The commit, or nil for "last-release" with no release yet.
35
+ # The commit, or nil for "last-release" with no release yet, and for
36
+ # a push with no commit before it here (see #pushed).
32
37
  def ref
33
38
  return @ref if instance_variable_defined?(:@ref)
34
39
 
35
40
  @ref = case @settings.base
36
41
  when nil then merge_base
37
42
  when LAST_RELEASE then git("describe", "--tags", "--abbrev=0", "--match", @config.release_tags, "HEAD^", allow_failure: true)
38
- else @settings.base
43
+ else @settings.pushed ? pushed : @settings.base
39
44
  end
40
45
  end
41
46
 
42
- # Whether nothing Config#watch names changed since the baseline (only
43
- # for a git baseline compared with the working tree).
47
+ # The branch that marks the last commit Duel passed in a push to the
48
+ # branch pushed (Settings#pushed), for Config#push_base:
49
+ # duel/passed/main, or for a project in a subdirectory of its
50
+ # repository, with its path (duel/passed-gems-parser/main), so each
51
+ # branch, and each of a monorepo's projects, has its own.
52
+ def passed_branch = "#{prefix.empty? ? "duel/passed" : "duel/passed-#{prefix.tr("/", "-")}"}/#{@settings.pushed}"
53
+
54
+ # The last commit Duel passed in a push, as checked out from GitHub
55
+ # (#passed_branch, on origin).
56
+ def passed = "origin/#{passed_branch}"
57
+
58
+ # Whether nothing #watched changed since the baseline (only for a git
59
+ # baseline compared with the working tree).
44
60
  def unchanged?
45
61
  return false if @settings.base_dir || @settings.candidate_dir
46
62
 
47
63
  # (There's a ref: CLI#run checks first.)
48
64
  commit = ref #: String
49
- _, status = Open3.capture2e("git", "-C", @config.root, "diff", "--quiet", commit, "--", *@config.watch)
65
+ _, status = Open3.capture2e("git", "-C", @config.root, "diff", "--quiet", commit, "--", *watched)
50
66
  status.success?
51
67
  end
52
68
 
69
+ # What a change must touch to be compared, from the root: the code
70
+ # (Config#watch), and what decides how it's measured. Each version
71
+ # runs the candidate's benchmarks and configuration, so a change to
72
+ # only those compares the code with itself, which checks them. They
73
+ # are the benchmarks' directory (the glob's, up to its first
74
+ # wildcard), the configuration's file, and with each version on its
75
+ # own Ruby, the version files at the root.
76
+ def watched
77
+ measured = [benchmarks, config_file]
78
+ measured.concat(version_files) if (@settings.ruby || @config.ruby) == :version_file
79
+ (@config.watch + measured).uniq
80
+ end
81
+
53
82
  # The baseline's tree, checked out and built if it's a commit.
54
83
  def root
55
84
  base_dir = @settings.base_dir
@@ -108,7 +137,7 @@ module Duel
108
137
 
109
138
  # The main branch (see #ref).
110
139
  def main_branch
111
- @config.main_branch ||
140
+ @settings.target || @config.main_branch ||
112
141
  git("symbolic-ref", "--short", "refs/remotes/origin/HEAD", allow_failure: true) ||
113
142
  BRANCHES.find { |branch| git("rev-parse", branch, allow_failure: true) } ||
114
143
  raise(Error, "no main branch to compare with (set c.main_branch in .duel.rb, or --base)")
@@ -129,6 +158,20 @@ module Duel
129
158
 
130
159
  private
131
160
 
161
+ # The benchmarks' directory (see #watched), or for benchmarks at the
162
+ # root, their glob, as git matches one.
163
+ def benchmarks
164
+ glob = @config.benchmarks
165
+ dir = glob.split("/").take_while { |part| !part.match?(/[*?\[{]/) }.join("/")
166
+ dir.empty? ? ":(glob)#{glob}" : dir
167
+ end
168
+
169
+ # The configuration's file, from the root (which is its directory).
170
+ def config_file = File.basename(@settings.config_path || ".duel.rb")
171
+
172
+ # The version files at the root (see Rubies).
173
+ def version_files = Rubies::VERSION_FILES.select { |name| File.exist?(File.join(@config.root, name)) }
174
+
132
175
  # Where HEAD left the main branch: their merge base, which there isn't
133
176
  # when their histories are unrelated, or in a shallow clone (CI's,
134
177
  # fetched without the history), where git fails saying nothing.
@@ -150,6 +193,17 @@ module Duel
150
193
  raise Error, "HEAD and #{branch} have no commit in common: #{why}, or compare with --base REF"
151
194
  end
152
195
 
196
+ # A push's baseline: the last commit Duel passed (#passed), where
197
+ # Config#push_base says and HEAD has it, or else the commit before the
198
+ # push, or nil where that isn't here (a branch's first push, or one
199
+ # that rewrote history).
200
+ def pushed
201
+ return passed if @config.push_base == :last_passed && git("merge-base", "--is-ancestor", passed, "HEAD", allow_failure: true)
202
+
203
+ before = @settings.base #: String
204
+ before if git("cat-file", "-e", "#{before}^{commit}", allow_failure: true)
205
+ end
206
+
153
207
  # Whether +dir+ is a checkout of this repository (a worktree of it).
154
208
  def ours?(dir) = common_dir(dir) == common_dir(@config.root)
155
209