duel 0.1.1 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -17,6 +17,9 @@ Duel.configure do |c|
17
17
  c.release_tags = "release-*" # default: v*
18
18
  c.ruby = :version_file # each version's own Ruby (default: the runner's)
19
19
  c.comment = :results # when CI comments on a pull request (default: :failures)
20
+ c.fail_on = :pull_requests # which runs a regression fails (default: :every_run)
21
+ c.push_base = :last_passed # what CI compares a push with (default: :before)
22
+ c.accept_label = "duel: accept" # a pull request label that accepts a regression
20
23
  c.use :redis # a plugin (see Use plugins)
21
24
 
22
25
  # How a worker loads one version of your code, instead of the library.
@@ -90,6 +93,83 @@ with a nonzero status, and the job's summary still has the report.
90
93
  `c.comment` takes only these four values: any other fails the run, and
91
94
  says which values it takes.
92
95
 
96
+ ## Fail only a pull request
97
+
98
+ By default, a regression fails every run: a pull request's, a push's,
99
+ and the schedule's. In a project where people push to `main` directly,
100
+ a failed push only gets GitHub's email that the run failed. To report a
101
+ regression in a push without failing it, set `c.fail_on`, and
102
+ `duel --fail-on` overrides it:
103
+
104
+ | Value | Which runs a regression fails |
105
+ |---|---|
106
+ | `:every_run` | Every run. This is the default. |
107
+ | `:pull_requests` | A pull request's (or a merge queue's), and a run outside GitHub Actions. |
108
+
109
+ ```ruby
110
+ Duel.configure do |c|
111
+ c.fail_on = :pull_requests
112
+ end
113
+ ```
114
+
115
+ With `:pull_requests`, a regression on GitHub Actions outside a pull
116
+ request (a push, the schedule, or a run by hand) passes, with a warning
117
+ in the run's summary, and the job's summary still has the report. On
118
+ GitLab CI, a regression outside a merge request exits with status 3,
119
+ which the [job](ci.md#run-duel-on-gitlab-ci) that `duel init --gitlab`
120
+ writes shows as a warning. The
121
+ [GitHub Actions workflow](ci.md#comment-on-a-pushed-commit) that
122
+ `duel init` writes then comments on a pushed commit with the report,
123
+ as `c.comment` says, which notifies the commit's author. A run that
124
+ couldn't compare fails either way, and on your machine, a regression
125
+ always fails.
126
+
127
+ ## Accept a regression with a label
128
+
129
+ Sometimes a change is slower on purpose: it's correct where the old
130
+ code wasn't, say. To let a pull request through anyway, name a label
131
+ that accepts its regression:
132
+
133
+ ```ruby
134
+ Duel.configure do |c|
135
+ c.accept_label = "duel: accept"
136
+ end
137
+ ```
138
+
139
+ On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
140
+ that `duel init` writes passes Duel the pull request's labels. With
141
+ that label, a regression passes, with a warning, and the report says
142
+ that the label accepted it. The labels are read when the workflow runs,
143
+ so after you add the label, run the workflow again (Re-run jobs, on the
144
+ pull request's Checks tab). For a push, Duel reads the labels of the
145
+ pull request that merged it, so with
146
+ [`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
147
+ the merged regression becomes the commit that later pushes are compared
148
+ with. By default, there's no such label.
149
+
150
+ ## Compare a push with the last commit that passed
151
+
152
+ The [GitHub Actions workflow](ci.md#compare-a-push-with-the-last-commit-that-passed)
153
+ that `duel init` writes compares a push to `main` with the commit
154
+ before it. A series of pushes that are each a little slower then each
155
+ pass. `c.push_base` chooses what a push is compared with:
156
+
157
+ | Value | What the workflow compares a push with |
158
+ |---|---|
159
+ | `:before` | The commit before the push. This is the default. |
160
+ | `:last_passed` | The last commit that Duel passed, which a branch marks: `duel/passed/main` for `main`. |
161
+
162
+ ```ruby
163
+ Duel.configure do |c|
164
+ c.push_base = :last_passed
165
+ end
166
+ ```
167
+
168
+ With `:last_passed`, the slowdowns add up until they fail a push. After
169
+ a regression, every push fails until one is no slower, or until you
170
+ move the branch to accept the regression:
171
+ `git push origin main:duel/passed/main`.
172
+
93
173
  ## Prepare each worker
94
174
 
95
175
  Each worker can get things of its own, such as a server, so that the
data/docs/how-it-works.md CHANGED
@@ -140,7 +140,10 @@ iteration, as they are in the explanation of a regression.
140
140
  The main branch is the one that `origin/HEAD` names. Without
141
141
  `origin/HEAD`, Duel uses `origin/main`, and then `origin/master`. In a
142
142
  repository without a remote, it uses `main`, and then `master`. To name
143
- another branch, set `c.main_branch`.
143
+ another branch, set `c.main_branch`. On GitHub Actions, for a pull
144
+ request, the main branch is the one that the pull request targets
145
+ (`GITHUB_BASE_REF`), such as a stable branch, or another pull request's
146
+ branch in a stack of them.
144
147
 
145
148
  Duel checks out a git baseline as a git worktree next to your project,
146
149
  under a hidden name that's as long as your project's directory name, for
data/lib/duel/baseline.rb CHANGED
@@ -11,12 +11,15 @@ module Duel
11
11
  # git commit checked out (and built, see Config#build) where #checkout
12
12
  # says. Settings#base names the commit:
13
13
  #
14
- # - nil: where HEAD left the main branch (their merge base):
14
+ # - nil: where HEAD left the main branch (their merge base): for a
15
+ # pull request, the branch it targets (Settings#target), or else
15
16
  # Config#main_branch, or the one origin/HEAD names, or origin/main, or
16
17
  # origin/master, or (a repository without a remote) main, or master
17
18
  # - "last-release": the newest release tag (Config#release_tags) before
18
19
  # HEAD, to catch slowdowns too small for any one change to fail but
19
20
  # that add up between releases
21
+ # - a push's (Settings#pushed, the branch pushed): the commit before it, or where
22
+ # Config#push_base says, the last commit Duel passed (#passed)
20
23
  # - anything else: a git ref
21
24
  # @api private
22
25
  class Baseline
@@ -29,17 +32,29 @@ module Duel
29
32
  @settings = settings
30
33
  end
31
34
 
32
- # The commit, or nil for "last-release" with no release yet.
35
+ # The commit, or nil for "last-release" with no release yet, and for
36
+ # a push with no commit before it here (see #pushed).
33
37
  def ref
34
38
  return @ref if instance_variable_defined?(:@ref)
35
39
 
36
40
  @ref = case @settings.base
37
41
  when nil then merge_base
38
42
  when LAST_RELEASE then git("describe", "--tags", "--abbrev=0", "--match", @config.release_tags, "HEAD^", allow_failure: true)
39
- else @settings.base
43
+ else @settings.pushed ? pushed : @settings.base
40
44
  end
41
45
  end
42
46
 
47
+ # The branch that marks the last commit Duel passed in a push to the
48
+ # branch pushed (Settings#pushed), for Config#push_base:
49
+ # duel/passed/main, or for a project in a subdirectory of its
50
+ # repository, with its path (duel/passed-gems-parser/main), so each
51
+ # branch, and each of a monorepo's projects, has its own.
52
+ def passed_branch = "#{prefix.empty? ? "duel/passed" : "duel/passed-#{prefix.tr("/", "-")}"}/#{@settings.pushed}"
53
+
54
+ # The last commit Duel passed in a push, as checked out from GitHub
55
+ # (#passed_branch, on origin).
56
+ def passed = "origin/#{passed_branch}"
57
+
43
58
  # Whether nothing #watched changed since the baseline (only for a git
44
59
  # baseline compared with the working tree).
45
60
  def unchanged?
@@ -122,7 +137,7 @@ module Duel
122
137
 
123
138
  # The main branch (see #ref).
124
139
  def main_branch
125
- @config.main_branch ||
140
+ @settings.target || @config.main_branch ||
126
141
  git("symbolic-ref", "--short", "refs/remotes/origin/HEAD", allow_failure: true) ||
127
142
  BRANCHES.find { |branch| git("rev-parse", branch, allow_failure: true) } ||
128
143
  raise(Error, "no main branch to compare with (set c.main_branch in .duel.rb, or --base)")
@@ -178,6 +193,17 @@ module Duel
178
193
  raise Error, "HEAD and #{branch} have no commit in common: #{why}, or compare with --base REF"
179
194
  end
180
195
 
196
+ # A push's baseline: the last commit Duel passed (#passed), where
197
+ # Config#push_base says and HEAD has it, or else the commit before the
198
+ # push, or nil where that isn't here (a branch's first push, or one
199
+ # that rewrote history).
200
+ def pushed
201
+ return passed if @config.push_base == :last_passed && git("merge-base", "--is-ancestor", passed, "HEAD", allow_failure: true)
202
+
203
+ before = @settings.base #: String
204
+ before if git("cat-file", "-e", "#{before}^{commit}", allow_failure: true)
205
+ end
206
+
181
207
  # Whether +dir+ is a checkout of this repository (a worktree of it).
182
208
  def ours?(dir) = common_dir(dir) == common_dir(@config.root)
183
209
 
data/lib/duel/cli.rb CHANGED
@@ -18,7 +18,8 @@ module Duel
18
18
  # On GitHub Actions (GITHUB_STEP_SUMMARY), it adds its report to the
19
19
  # job's summary, in Markdown (see Summary), and tells the step's
20
20
  # outputs (GITHUB_OUTPUT) what to do about the comment on a pull
21
- # request (see .comment). --markdown writes the Markdown to a file, and
21
+ # request (see .comment), and for a push, whether to mark it passed
22
+ # (see Config#push_base). --markdown writes the Markdown to a file, and
22
23
  # --json the report as JSON (see JSONReport), whatever the run's
23
24
  # outcome, but not for a command line that isn't valid.
24
25
  # @api private
@@ -32,6 +33,10 @@ module Duel
32
33
  # Where GitHub Actions has a step set its outputs.
33
34
  STEP_OUTPUT = "GITHUB_OUTPUT"
34
35
 
36
+ # Where else to write them, as variables (DUEL_COMMENT, DUEL_STATUS,
37
+ # DUEL_PASSED): on GitLab CI, a dotenv report, for the jobs after.
38
+ OUTPUT = "DUEL_OUTPUT"
39
+
35
40
  # The statuses (as the JSON report has them) that the workflow
36
41
  # comments on, by when it does (Config#comment, but :never, which
37
42
  # comments on none).
@@ -41,6 +46,16 @@ module Duel
41
46
  # has it: passed, failed (or couldn't compare), or neutral.
42
47
  MARKED = {"passed" => "passed", "failed" => "failed", "error" => "failed", "nothing_to_compare" => "neutral"}.freeze
43
48
 
49
+ # The GitHub Actions events of a pull request, as GITHUB_EVENT_NAME
50
+ # has them: a merge queue's too (see Config#fail_on). (A GitLab merge
51
+ # request's is pull_request: see #event.)
52
+ PULL_REQUESTS = %w[pull_request pull_request_target merge_group].freeze
53
+
54
+ # The exit status of a run that was slower outside a merge request on
55
+ # GitLab CI, where only a merge request fails (Config#fail_on): the
56
+ # job that `duel init --gitlab` writes lets it fail, as a warning.
57
+ WARNED = 3
58
+
44
59
  # The variables that set options, to their values, and the one that
45
60
  # sets a flag by being set at all.
46
61
  ENV_OPTIONS = {"DUEL_BASE" => "--base", "DUEL_FILTER" => "--filter"}.freeze
@@ -54,7 +69,8 @@ module Duel
54
69
  end
55
70
 
56
71
  # Runs, and returns the exit status: 0 not slower (or nothing to compare),
57
- # 1 slower (or a benchmark failed), 2 couldn't compare.
72
+ # 1 slower (or a benchmark failed), 2 couldn't compare, 3 slower but
73
+ # warned (see WARNED).
58
74
  def run
59
75
  return Init.new(@argv.drop(1), out: @out, err: @err).run if @argv.first == "init"
60
76
 
@@ -91,9 +107,17 @@ module Duel
91
107
 
92
108
  # The mode --comment names (see Config#comment), or an invalid
93
109
  # argument.
94
- def self.comment_mode(name)
95
- Config::COMMENTS.find { |mode| mode.to_s == name } or
96
- raise OptionParser::InvalidArgument, "#{name} (give #{Config::COMMENTS.join(", ")})"
110
+ def self.comment_mode(name) = named(name, Config::COMMENTS)
111
+
112
+ # The runs --fail-on names (see Config#fail_on), or an invalid
113
+ # argument.
114
+ def self.fail_on(name) = named(name, Config::FAIL_ON)
115
+
116
+ # Which of +values+ (symbols) an option's argument names, or an
117
+ # invalid argument.
118
+ def self.named(name, values)
119
+ values.find { |value| value.to_s == name } or
120
+ raise OptionParser::InvalidArgument, "#{name} (give #{values.join(", ")})"
97
121
  end
98
122
 
99
123
  # The settings, from the environment and then the options.
@@ -101,6 +125,11 @@ module Duel
101
125
  settings = Settings.defaults
102
126
  settings.jobs = [Etc.nprocessors / 2, 1].max
103
127
  parser(settings).parse!(env_options + @argv)
128
+ branch = variable("GITHUB_REF_NAME", "CI_COMMIT_BRANCH")
129
+ base = variable("DUEL_BASE")
130
+ settings.pushed = branch if pushed? && !base.nil? && settings.base == base
131
+ target = variable("GITHUB_BASE_REF", "CI_MERGE_REQUEST_TARGET_BRANCH_NAME")
132
+ settings.target = "origin/#{target}" if target && pull_request?
104
133
  settings.config_path ||= (".duel.rb" if File.exist?(".duel.rb"))
105
134
  settings.config_path &&= File.expand_path(settings.config_path)
106
135
  settings
@@ -117,9 +146,12 @@ module Duel
117
146
  return 0
118
147
  end
119
148
  if settings.base_dir.nil? && baseline.ref.nil?
149
+ return nothing("no commit to compare against", "no commit before this push to compare against") if settings.pushed
150
+
120
151
  return nothing("no release to compare against", "no release tag (#{config.release_tags}) before HEAD to compare against")
121
152
  end
122
153
  if !settings.force && baseline.unchanged?
154
+ @passed = marks_passed?(config, settings)
123
155
  return nothing("nothing to compare",
124
156
  "#{baseline.watched.join(", ")} unchanged since the baseline, so there's nothing to compare (--force to anyway)")
125
157
  end
@@ -130,12 +162,14 @@ module Duel
130
162
  uneven(base_root, candidate)
131
163
  runner = Runner.new(config, settings, base_root, out: @out)
132
164
  results = runner.run
133
- status = runner.passed?(results) ? 0 : 1
165
+ passed = runner.passed?(results)
166
+ status = passed ? 0 : slower(config, settings)
167
+ @passed = (passed || accepted?(config)) && marks_passed?(config, settings)
134
168
  publish(
135
- status.zero? ? "passed" : "failed",
169
+ passed ? "passed" : "failed",
136
170
  -> {
137
171
  Summary.new(runner, results, settings: settings, baseline: described_baseline(config, settings, baseline),
138
- candidate: described_candidate(config, settings)).markdown
172
+ candidate: described_candidate(config, settings), note: @note).markdown
139
173
  },
140
174
  ->(comment) {
141
175
  JSONReport.new(runner, results, settings: settings, exit: status, comment: comment,
@@ -145,6 +179,107 @@ module Duel
145
179
  status
146
180
  end
147
181
 
182
+ # The exit status of a run that was slower (or whose benchmark
183
+ # failed): 1, or 0 where the pull request's label accepts it
184
+ # (Config#accept_label), or on GitHub Actions outside a pull request,
185
+ # where c.fail_on (or --fail-on) says only a pull request fails, with
186
+ # a warning that says so, and a note in the report.
187
+ def slower(config, settings)
188
+ if accepted?(config)
189
+ # (Set: see #accepted?.)
190
+ label = config.accept_label #: String
191
+ return passing("accepted with the #{label} label", "Accepted with the #{Summary.code(label)} label.")
192
+ end
193
+ return 1 unless pull_requests_only?(config, settings)
194
+ return 1 unless outside_pull_request?
195
+
196
+ # (Outside a pull request on CI, so there's one: see #event.)
197
+ run = event #: String
198
+ passing("only a pull request fails (c.fail_on), so this #{run} passes",
199
+ "Only a pull request fails (#{Summary.code("c.fail_on")}), so this #{run} passes.", gitlab? ? WARNED : 0)
200
+ end
201
+
202
+ # Passes a run that was slower (or on GitLab CI, warns: +exit+), as
203
+ # +why+ says, in a warning (on GitHub Actions, an annotation), and as
204
+ # +note+ says in the report.
205
+ def passing(why, note, exit = 0)
206
+ @out.puts "#{gitlab? ? "duel: " : "::warning::Duel: "}slower, but #{why}"
207
+ @note = note
208
+ exit
209
+ end
210
+
211
+ # Whether the pull request's labels have the one that accepts a
212
+ # regression (Config#accept_label).
213
+ def accepted?(config)
214
+ label = config.accept_label
215
+ !label.nil? && labels.include?(label)
216
+ end
217
+
218
+ # The pull request's labels: DUEL_LABELS (a JSON array of their names)
219
+ # on GitHub Actions, or a GitLab merge request's
220
+ # (CI_MERGE_REQUEST_LABELS, by commas), or none, where neither is
221
+ # set, or DUEL_LABELS isn't a JSON array.
222
+ def labels
223
+ set = variables
224
+ return set.fetch("CI_MERGE_REQUEST_LABELS").split(",") if !set.key?("DUEL_LABELS") && set.key?("CI_MERGE_REQUEST_LABELS")
225
+ return [] unless set.key?("DUEL_LABELS")
226
+
227
+ names = JSON.parse(set.fetch("DUEL_LABELS"))
228
+ names.instance_of?(Array) ? names : []
229
+ rescue JSON::ParserError
230
+ []
231
+ end
232
+
233
+ # Whether the workflow marks a push that passed (or had nothing to
234
+ # compare) as the last commit Duel passed: where Config#push_base
235
+ # says to compare a push with that commit.
236
+ def marks_passed?(config, settings) = !settings.pushed.nil? && config.push_base == :last_passed
237
+
238
+ # Whether a regression fails only a pull request (--fail-on, or else
239
+ # Config#fail_on).
240
+ def pull_requests_only?(config, settings) = (settings.fail_on || config.fail_on) == :pull_requests
241
+
242
+ # What a run on CI is for, as GitHub Actions names it
243
+ # (GITHUB_EVENT_NAME): push, pull_request, schedule, say. On GitLab
244
+ # CI, its pipeline's source (CI_PIPELINE_SOURCE), as GitHub would
245
+ # name it: a merge request's is pull_request. Outside CI, nil.
246
+ def event
247
+ github = variable("GITHUB_EVENT_NAME")
248
+ return github if github
249
+ return unless gitlab?
250
+
251
+ source = variable("CI_PIPELINE_SOURCE")
252
+ (source == "merge_request_event") ? "pull_request" : source
253
+ end
254
+
255
+ # Whether this is a run on GitLab CI.
256
+ def gitlab? = variables.key?("GITLAB_CI")
257
+
258
+ # Whether this is a run on CI for something other than a pull request
259
+ # (PULL_REQUESTS): a push, say.
260
+ def outside_pull_request?
261
+ run = event
262
+ !run.nil? && !PULL_REQUESTS.include?(run)
263
+ end
264
+
265
+ # Whether this is a run on CI for a pull request (or a merge queue:
266
+ # see PULL_REQUESTS).
267
+ def pull_request? = PULL_REQUESTS.include?(event)
268
+
269
+ # Whether the baseline is a merge queue's base: on GitHub Actions for
270
+ # a merge queue, as DUEL_BASE gives it.
271
+ def queued?(settings) = event == "merge_group" && settings.base == variable("DUEL_BASE")
272
+
273
+ # Whether this is a run on CI for a push.
274
+ def pushed? = event == "push"
275
+
276
+ # The first of the variables +names+ that's set, or nil.
277
+ def variable(*names)
278
+ set = variables
279
+ name = names.find { |n| set.key?(n) }
280
+ set.fetch(name) if name
281
+ end
282
+
148
283
  # Says there's nothing to compare (+title+, in Markdown), and +why+,
149
284
  # and passes.
150
285
  def nothing(title, why)
@@ -161,14 +296,19 @@ module Duel
161
296
  exit
162
297
  end
163
298
 
164
- # What the workflow does about its comment on a pull request (see
165
- # .comment), for a run whose +status+ is the JSON report's: as
166
- # --comment says, or else the configuration, or the default where the
167
- # configuration didn't load.
299
+ # What the workflow does about its comment on a pull request, or on
300
+ # a pushed commit (see .comment), for a run whose +status+ is the
301
+ # JSON report's: as --comment says, or else the configuration, or the
302
+ # default where the configuration didn't load. It comments on a push
303
+ # only where a regression in one passes (Config#fail_on), so the
304
+ # comment says what the check doesn't.
168
305
  def commenting(status)
169
306
  # (Parsed, so set: see #run.)
170
307
  settings = @settings #: Settings
171
- CLI.comment(settings.comment || (@config || Config.new).comment, status)
308
+ config = @config || Config.new
309
+ return "none" if pushed? && !pull_requests_only?(config, settings)
310
+
311
+ CLI.comment(settings.comment || config.comment, status)
172
312
  end
173
313
 
174
314
  # Writes the reports of a run whose +status+ is the JSON report's
@@ -177,13 +317,17 @@ module Duel
177
317
  # --json names, and the Markdown +markdown+ makes to the file
178
318
  # --markdown names, and added to the job's summary on GitHub Actions
179
319
  # (STEP_SUMMARY). Each is only made if it's written. On GitHub
180
- # Actions, it also sets the step's outputs (STEP_OUTPUT) for the job
181
- # that comments on a pull request: +comment+, and +status+ as the
182
- # comment's marker has it (MARKED).
320
+ # Actions, it also sets the step's outputs (STEP_OUTPUT) for the jobs
321
+ # that comment: +comment+, and +status+ as the comment's marker has
322
+ # it (MARKED), and for the job that marks a push passed, "passed"
323
+ # (see #marks_passed?). Where DUEL_OUTPUT names a file (OUTPUT), it
324
+ # writes them there too, as variables.
183
325
  def publish(status, markdown, json)
184
326
  comment = commenting(status)
185
327
  set = variables
186
- File.write(set.fetch(STEP_OUTPUT), "comment=#{comment}\nstatus=#{MARKED.fetch(status)}\n", mode: "a") if set.key?(STEP_OUTPUT)
328
+ outputs = "comment=#{comment}\nstatus=#{MARKED.fetch(status)}\n#{"passed=true\n" if @passed}"
329
+ File.write(set.fetch(STEP_OUTPUT), outputs, mode: "a") if set.key?(STEP_OUTPUT)
330
+ File.write(set.fetch(OUTPUT), outputs.gsub(/^\w+=/) { |name| "DUEL_#{name.upcase}" }, mode: "a") if set.key?(OUTPUT)
187
331
  path = @json
188
332
  write(path, "#{JSON.pretty_generate(json.call(comment))}\n") if path
189
333
  path = @markdown
@@ -204,7 +348,9 @@ module Duel
204
348
  # The baseline, in Markdown: its directory, or its commit, bare (so
205
349
  # GitHub links it) and followed by a space, never punctuation, and
206
350
  # what chose it: the main branch, the push (on GitHub Actions, the
207
- # commit before it, from DUEL_BASE), the last release, or --base.
351
+ # commit before it, from DUEL_BASE, or the last commit Duel passed),
352
+ # the merge queue (its base, from DUEL_BASE), the last release, or
353
+ # --base.
208
354
  def described_baseline(config, settings, baseline)
209
355
  dir = settings.base_dir
210
356
  return Summary.code(dir) if dir
@@ -212,15 +358,13 @@ module Duel
212
358
  # (There's one: #compare checked.)
213
359
  ref = baseline.ref #: String
214
360
  commit = git(config.root, "rev-parse", "--short", "#{ref}^{commit}").first.chomp
215
- set = variables
216
361
  chosen = case settings.base
217
362
  when nil then "where HEAD left #{Summary.code(baseline.main_branch)}"
218
363
  when Baseline::LAST_RELEASE then "the last release, #{Summary.code(ref)}"
219
364
  else
220
- pushed = set.key?("GITHUB_EVENT_NAME") && set.fetch("GITHUB_EVENT_NAME") == "push" && set.key?("DUEL_BASE") &&
221
- set.fetch("DUEL_BASE") == ref
222
- if !ref.match?(/\A\h+\z/) then Summary.code(ref)
223
- elsif pushed then "the commit before the push"
365
+ if settings.pushed then (ref == baseline.passed) ? "the last commit Duel passed" : "the commit before the push"
366
+ elsif queued?(settings) then "the merge queue's base"
367
+ elsif !ref.match?(/\A\h+\z/) then Summary.code(ref)
224
368
  else "from --base"
225
369
  end
226
370
  end
@@ -248,11 +392,7 @@ module Duel
248
392
 
249
393
  # (There's one: #compare checked.)
250
394
  ref = baseline.ref #: String
251
- named = case settings.base
252
- when nil then baseline.main_branch
253
- when Baseline::LAST_RELEASE then ref
254
- else settings.base
255
- end
395
+ named = settings.base ? ref : baseline.main_branch
256
396
  described.merge("ref" => named, "commit" => git(config.root, "rev-parse", "#{ref}^{commit}").first.chomp)
257
397
  end
258
398
 
@@ -322,6 +462,8 @@ module Duel
322
462
  o.on("--json PATH", "Also write the report to PATH as JSON, for a program to read") { |v| @json = v }
323
463
  o.on("--comment MODE", "When the workflow comments on a pull request: always, results (passed or failed),",
324
464
  "failures, or never (default: c.comment, or failures)") { |v| settings.comment = CLI.comment_mode(v) }
465
+ o.on("--fail-on RUNS", "Which runs a regression fails: every_run, or pull_requests (on GitHub Actions, another",
466
+ "warns and passes) (default: c.fail_on, or every_run)") { |v| settings.fail_on = CLI.fail_on(v) }
325
467
  o.on("--clean", "Remove tmp/duel, and the baseline's git worktree, and stop") { @clean = true }
326
468
  o.on("-v", "--version", "Print the version") do
327
469
  @out.puts "duel #{VERSION}"
data/lib/duel/config.rb CHANGED
@@ -95,6 +95,75 @@ module Duel
95
95
  @comment = mode
96
96
  end
97
97
 
98
+ # Which runs a regression fails (`duel --fail-on` overrides it):
99
+ #
100
+ # - :every_run (the default)
101
+ # - :pull_requests, only a pull request's (or a merge queue's), and a
102
+ # run outside GitHub Actions. On GitHub Actions, a regression in
103
+ # any other run (a push, the schedule, run by hand) passes, with a
104
+ # warning, and the workflow that `duel init` writes comments on a
105
+ # pushed commit with the report, as #comment says, which notifies
106
+ # its author.
107
+ #
108
+ # A run that couldn't compare fails either way.
109
+ #
110
+ # c.fail_on = :pull_requests
111
+ attr_reader :fail_on
112
+
113
+ # The values #fail_on takes.
114
+ FAIL_ON = %i[every_run pull_requests].freeze #: Array[fail_on]
115
+
116
+ def fail_on=(runs)
117
+ raise Error, "c.fail_on = #{runs.inspect}: give #{FAIL_ON.map(&:inspect).join(", ")}" unless FAIL_ON.include?(runs)
118
+
119
+ @fail_on = runs
120
+ end
121
+
122
+ # The name of a label that accepts a regression in a pull request
123
+ # (or nil, the default, for none): on GitHub Actions, the workflow
124
+ # that `duel init` writes passes Duel the pull request's labels
125
+ # (DUEL_LABELS), and with this one, a regression passes, with a
126
+ # warning, and the report says it was accepted. For a push, it's the
127
+ # labels of the pull request that merged it, so where #push_base is
128
+ # :last_passed, the push is marked passed too. The labels are read
129
+ # when the workflow runs, so after adding the label, run it again.
130
+ #
131
+ # c.accept_label = "duel: accept"
132
+ attr_reader :accept_label
133
+
134
+ def accept_label=(label)
135
+ raise Error, "c.accept_label = #{label.inspect}: give a label's name, or nil" unless label.nil? || label.instance_of?(String)
136
+
137
+ @accept_label = label
138
+ end
139
+
140
+ # What the workflow that `duel init` writes compares a push with:
141
+ #
142
+ # - :before (the default), the commit before the push
143
+ # - :last_passed, the last commit Duel passed, which the workflow
144
+ # marks with a branch, duel/passed/main for main (see
145
+ # Baseline#passed_branch),
146
+ # when a push passes, or has nothing to compare. Slowdowns too
147
+ # small for any one push to fail then add up until they fail one,
148
+ # and a push the workflow didn't compare (its run cancelled, say)
149
+ # is compared with the next. After a regression, every push is
150
+ # compared with the commit before it, until one is no slower, or
151
+ # the branch is moved to accept the regression. Without the branch,
152
+ # or where HEAD doesn't have its commit (after a force push, say),
153
+ # a push is compared with the commit before it.
154
+ #
155
+ # c.push_base = :last_passed
156
+ attr_reader :push_base
157
+
158
+ # The values #push_base takes.
159
+ PUSH_BASES = %i[before last_passed].freeze #: Array[push_base]
160
+
161
+ def push_base=(base)
162
+ raise Error, "c.push_base = #{base.inspect}: give #{PUSH_BASES.map(&:inspect).join(", ")}" unless PUSH_BASES.include?(base)
163
+
164
+ @push_base = base
165
+ end
166
+
98
167
  # The metrics #metric added, in order (classes: a Duel::Metric
99
168
  # subclass, or running totals' Snapshot), for a plugin's tests to
100
169
  # look at (their .measures and .settings).
@@ -115,6 +184,8 @@ module Duel
115
184
  @build = :conventional
116
185
  @release_tags = "v*"
117
186
  @comment = :failures
187
+ @fail_on = :every_run
188
+ @push_base = :before
118
189
  @prepares = SCOPES.to_h { |scope| [scope, []] }
119
190
  @metrics = []
120
191
  @in_workers = []
data/lib/duel/init.rb CHANGED
@@ -10,15 +10,17 @@ require_relative "../duel"
10
10
  module Duel
11
11
  # `duel init`: sets Duel up in a gem or an app that isn't a Rails app
12
12
  # (duel-rails's generator sets a Rails app up). It writes .duel.rb, a
13
- # benchmark to start from, and a GitHub Actions workflow, adds a Rake
13
+ # benchmark to start from, and GitHub Actions workflows (Duel's, and
14
+ # one that comments on a pull request with its report), or with
15
+ # --gitlab, GitLab CI jobs to include, adds a Rake
14
16
  # task to the Rakefile, and adds StackProf to the Gemfile (for profiles
15
17
  # of regressions, on CRuby). It overwrites nothing without --force, and
16
18
  # writes nothing with --dry-run.
17
19
  # @api private
18
20
  class Init
19
21
  USAGE = "Usage: duel init [options]\n\n" \
20
- "Sets Duel up in this project: .duel.rb, a benchmark to start from in duel/, a GitHub Actions workflow,\n" \
21
- "a Rake task, and StackProf in the Gemfile.\n\n"
22
+ "Sets Duel up in this project: .duel.rb, a benchmark to start from in duel/, GitHub Actions workflows\n" \
23
+ "(or GitLab CI jobs), a Rake task, and StackProf in the Gemfile.\n\n"
22
24
 
23
25
  TEMPLATES = File.join(__dir__, "templates")
24
26
 
@@ -61,8 +63,13 @@ module Duel
61
63
 
62
64
  # The files it writes, from the project's root, to what's in them.
63
65
  def files
64
- workflow = Pathname(File.realpath(repository)).relative_path_from(File.realpath(@root)).join(".github", "workflows", workflow_name)
65
- {".duel.rb" => render("duel.rb"), benchmark_file => render("benchmark_duel.rb"), workflow.to_s => render("perf.yml")}
66
+ top = Pathname(File.realpath(repository)).relative_path_from(File.realpath(@root))
67
+ written = {".duel.rb" => render("duel.rb"), benchmark_file => render("benchmark_duel.rb")}
68
+ return written.merge(top.join(gitlab_file).to_s => render("duel.gitlab-ci.yml")) if @gitlab
69
+
70
+ workflows = top.join(".github", "workflows")
71
+ written.merge(workflows.join(workflow_name).to_s => render("perf.yml"),
72
+ workflows.join(workflow_name.sub(/\.yml\z/, "-comment.yml")).to_s => render("perf-comment.yml"))
66
73
  end
67
74
 
68
75
  private
@@ -72,6 +79,7 @@ module Duel
72
79
  o.banner = USAGE
73
80
  o.on("--force", "Overwrite files that exist") { @force = true }
74
81
  o.on("--dry-run", "Say what it would do, and write nothing") { @dry_run = true }
82
+ o.on("--gitlab", "Write GitLab CI jobs (to include from .gitlab-ci.yml), not GitHub Actions workflows") { @gitlab = true }
75
83
  o.on("-h", "--help", "Print this help") do
76
84
  @out.puts o
77
85
  throw :help
@@ -156,8 +164,13 @@ module Duel
156
164
  steps << "Set c.library in .duel.rb to what each version's workers require." unless required
157
165
  steps << "Replace the example in #{benchmark_file} with benchmarks of your own code."
158
166
  steps << "Run `bundle exec duel --force`. It compares the code with itself, and passes when the benchmarks are sound."
159
- steps << "Commit and push. On a pull request, the workflow compares the code with the main branch, and comments with the " \
160
- "report when a benchmark fails (c.comment in .duel.rb says when)."
167
+ steps << if @gitlab
168
+ "Include #{gitlab_file} from .gitlab-ci.yml, and add the tokens it says to, for comments. On a merge request, the job " \
169
+ "compares the code with the branch it targets."
170
+ else
171
+ "Commit and push. On a pull request, the workflow compares the code with the branch it targets, and comments with the " \
172
+ "report when a benchmark fails (c.comment in .duel.rb says when)."
173
+ end
161
174
  "\nNext:\n#{steps.each_with_index.map { |step, i| " #{i + 1}. #{step}\n" }.join}"
162
175
  end
163
176
 
@@ -215,5 +228,26 @@ module Duel
215
228
 
216
229
  # What marks the workflow's comment on a pull request.
217
230
  def marker = ["duel", working_directory].compact.join(" ")
231
+
232
+ # Where the GitLab CI jobs are, from the repository's root:
233
+ # .gitlab/duel.gitlab-ci.yml, or for a project in a subdirectory,
234
+ # .gitlab/duel-<its path>.gitlab-ci.yml.
235
+ def gitlab_file = File.join(".gitlab", "#{["duel", working_directory&.tr("/", "-")].compact.join("-")}.gitlab-ci.yml")
236
+
237
+ # The GitLab CI job's name: duel, or for a project in a subdirectory,
238
+ # duel:<its path>, so each of a monorepo's projects has its own.
239
+ def gitlab_job = ["duel", working_directory].compact.join(":")
240
+
241
+ # The Ruby image's tag for GitLab CI: the version .ruby-version names,
242
+ # exactly (so it's the Ruby that c.ruby = :version_file asks for), or
243
+ # this one's minor version.
244
+ def gitlab_ruby
245
+ named = File.read(File.join(@root, ".ruby-version")).split.first if version_file
246
+ named&.delete_prefix("ruby-") || RUBY_VERSION.rpartition(".").first
247
+ end
248
+
249
+ # What names the branches that mark the last commit Duel passed in a
250
+ # push, before the branch pushed's name (see Baseline#passed_branch).
251
+ def passed_branch = ["duel/passed", working_directory&.tr("/", "-")].compact.join("-")
218
252
  end
219
253
  end