duel 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +67 -1
- data/README.md +20 -2
- data/docs/ci.md +495 -61
- data/docs/configuration.md +80 -0
- data/docs/how-it-works.md +4 -1
- data/lib/duel/baseline.rb +30 -4
- data/lib/duel/cli.rb +170 -28
- data/lib/duel/config.rb +71 -0
- data/lib/duel/init.rb +41 -7
- data/lib/duel/runner.rb +1 -1
- data/lib/duel/summary.rb +5 -3
- data/lib/duel/templates/duel.gitlab-ci.yml.erb +148 -0
- data/lib/duel/templates/duel.rb.erb +13 -0
- data/lib/duel/templates/perf-comment.yml.erb +59 -0
- data/lib/duel/templates/perf.yml.erb +116 -46
- data/lib/duel/version.rb +1 -1
- data/sig/duel/baseline.rbs +14 -1
- data/sig/duel/cli.rbs +64 -2
- data/sig/duel/config.rbs +36 -0
- data/sig/duel/init.rbs +13 -0
- data/sig/duel/runner.rbs +15 -3
- data/sig/duel/summary.rbs +3 -1
- metadata +3 -1
data/docs/configuration.md
CHANGED
|
@@ -17,6 +17,9 @@ Duel.configure do |c|
|
|
|
17
17
|
c.release_tags = "release-*" # default: v*
|
|
18
18
|
c.ruby = :version_file # each version's own Ruby (default: the runner's)
|
|
19
19
|
c.comment = :results # when CI comments on a pull request (default: :failures)
|
|
20
|
+
c.fail_on = :pull_requests # which runs a regression fails (default: :every_run)
|
|
21
|
+
c.push_base = :last_passed # what CI compares a push with (default: :before)
|
|
22
|
+
c.accept_label = "duel: accept" # a pull request label that accepts a regression
|
|
20
23
|
c.use :redis # a plugin (see Use plugins)
|
|
21
24
|
|
|
22
25
|
# How a worker loads one version of your code, instead of the library.
|
|
@@ -90,6 +93,83 @@ with a nonzero status, and the job's summary still has the report.
|
|
|
90
93
|
`c.comment` takes only these four values: any other fails the run, and
|
|
91
94
|
says which values it takes.
|
|
92
95
|
|
|
96
|
+
## Fail only a pull request
|
|
97
|
+
|
|
98
|
+
By default, a regression fails every run: a pull request's, a push's,
|
|
99
|
+
and the schedule's. In a project where people push to `main` directly,
|
|
100
|
+
a failed push only gets GitHub's email that the run failed. To report a
|
|
101
|
+
regression in a push without failing it, set `c.fail_on`, and
|
|
102
|
+
`duel --fail-on` overrides it:
|
|
103
|
+
|
|
104
|
+
| Value | Which runs a regression fails |
|
|
105
|
+
|---|---|
|
|
106
|
+
| `:every_run` | Every run. This is the default. |
|
|
107
|
+
| `:pull_requests` | A pull request's (or a merge queue's), and a run outside GitHub Actions. |
|
|
108
|
+
|
|
109
|
+
```ruby
|
|
110
|
+
Duel.configure do |c|
|
|
111
|
+
c.fail_on = :pull_requests
|
|
112
|
+
end
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
With `:pull_requests`, a regression on GitHub Actions outside a pull
|
|
116
|
+
request (a push, the schedule, or a run by hand) passes, with a warning
|
|
117
|
+
in the run's summary, and the job's summary still has the report. On
|
|
118
|
+
GitLab CI, a regression outside a merge request exits with status 3,
|
|
119
|
+
which the [job](ci.md#run-duel-on-gitlab-ci) that `duel init --gitlab`
|
|
120
|
+
writes shows as a warning. The
|
|
121
|
+
[GitHub Actions workflow](ci.md#comment-on-a-pushed-commit) that
|
|
122
|
+
`duel init` writes then comments on a pushed commit with the report,
|
|
123
|
+
as `c.comment` says, which notifies the commit's author. A run that
|
|
124
|
+
couldn't compare fails either way, and on your machine, a regression
|
|
125
|
+
always fails.
|
|
126
|
+
|
|
127
|
+
## Accept a regression with a label
|
|
128
|
+
|
|
129
|
+
Sometimes a change is slower on purpose: it's correct where the old
|
|
130
|
+
code wasn't, say. To let a pull request through anyway, name a label
|
|
131
|
+
that accepts its regression:
|
|
132
|
+
|
|
133
|
+
```ruby
|
|
134
|
+
Duel.configure do |c|
|
|
135
|
+
c.accept_label = "duel: accept"
|
|
136
|
+
end
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
On GitHub Actions, the [workflow](ci.md#accept-a-regression-with-a-label)
|
|
140
|
+
that `duel init` writes passes Duel the pull request's labels. With
|
|
141
|
+
that label, a regression passes, with a warning, and the report says
|
|
142
|
+
that the label accepted it. The labels are read when the workflow runs,
|
|
143
|
+
so after you add the label, run the workflow again (Re-run jobs, on the
|
|
144
|
+
pull request's Checks tab). For a push, Duel reads the labels of the
|
|
145
|
+
pull request that merged it, so with
|
|
146
|
+
[`c.push_base = :last_passed`](#compare-a-push-with-the-last-commit-that-passed),
|
|
147
|
+
the merged regression becomes the commit that later pushes are compared
|
|
148
|
+
with. By default, there's no such label.
|
|
149
|
+
|
|
150
|
+
## Compare a push with the last commit that passed
|
|
151
|
+
|
|
152
|
+
The [GitHub Actions workflow](ci.md#compare-a-push-with-the-last-commit-that-passed)
|
|
153
|
+
that `duel init` writes compares a push to `main` with the commit
|
|
154
|
+
before it. A series of pushes that are each a little slower then each
|
|
155
|
+
pass. `c.push_base` chooses what a push is compared with:
|
|
156
|
+
|
|
157
|
+
| Value | What the workflow compares a push with |
|
|
158
|
+
|---|---|
|
|
159
|
+
| `:before` | The commit before the push. This is the default. |
|
|
160
|
+
| `:last_passed` | The last commit that Duel passed, which a branch marks: `duel/passed/main` for `main`. |
|
|
161
|
+
|
|
162
|
+
```ruby
|
|
163
|
+
Duel.configure do |c|
|
|
164
|
+
c.push_base = :last_passed
|
|
165
|
+
end
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
With `:last_passed`, the slowdowns add up until they fail a push. After
|
|
169
|
+
a regression, every push fails until one is no slower, or until you
|
|
170
|
+
move the branch to accept the regression:
|
|
171
|
+
`git push origin main:duel/passed/main`.
|
|
172
|
+
|
|
93
173
|
## Prepare each worker
|
|
94
174
|
|
|
95
175
|
Each worker can get things of its own, such as a server, so that the
|
data/docs/how-it-works.md
CHANGED
|
@@ -140,7 +140,10 @@ iteration, as they are in the explanation of a regression.
|
|
|
140
140
|
The main branch is the one that `origin/HEAD` names. Without
|
|
141
141
|
`origin/HEAD`, Duel uses `origin/main`, and then `origin/master`. In a
|
|
142
142
|
repository without a remote, it uses `main`, and then `master`. To name
|
|
143
|
-
another branch, set `c.main_branch`.
|
|
143
|
+
another branch, set `c.main_branch`. On GitHub Actions, for a pull
|
|
144
|
+
request, the main branch is the one that the pull request targets
|
|
145
|
+
(`GITHUB_BASE_REF`), such as a stable branch, or another pull request's
|
|
146
|
+
branch in a stack of them.
|
|
144
147
|
|
|
145
148
|
Duel checks out a git baseline as a git worktree next to your project,
|
|
146
149
|
under a hidden name that's as long as your project's directory name, for
|
data/lib/duel/baseline.rb
CHANGED
|
@@ -11,12 +11,15 @@ module Duel
|
|
|
11
11
|
# git commit checked out (and built, see Config#build) where #checkout
|
|
12
12
|
# says. Settings#base names the commit:
|
|
13
13
|
#
|
|
14
|
-
# - nil: where HEAD left the main branch (their merge base):
|
|
14
|
+
# - nil: where HEAD left the main branch (their merge base): for a
|
|
15
|
+
# pull request, the branch it targets (Settings#target), or else
|
|
15
16
|
# Config#main_branch, or the one origin/HEAD names, or origin/main, or
|
|
16
17
|
# origin/master, or (a repository without a remote) main, or master
|
|
17
18
|
# - "last-release": the newest release tag (Config#release_tags) before
|
|
18
19
|
# HEAD, to catch slowdowns too small for any one change to fail but
|
|
19
20
|
# that add up between releases
|
|
21
|
+
# - a push's (Settings#pushed, the branch pushed): the commit before it, or where
|
|
22
|
+
# Config#push_base says, the last commit Duel passed (#passed)
|
|
20
23
|
# - anything else: a git ref
|
|
21
24
|
# @api private
|
|
22
25
|
class Baseline
|
|
@@ -29,17 +32,29 @@ module Duel
|
|
|
29
32
|
@settings = settings
|
|
30
33
|
end
|
|
31
34
|
|
|
32
|
-
# The commit, or nil for "last-release" with no release yet
|
|
35
|
+
# The commit, or nil for "last-release" with no release yet, and for
|
|
36
|
+
# a push with no commit before it here (see #pushed).
|
|
33
37
|
def ref
|
|
34
38
|
return @ref if instance_variable_defined?(:@ref)
|
|
35
39
|
|
|
36
40
|
@ref = case @settings.base
|
|
37
41
|
when nil then merge_base
|
|
38
42
|
when LAST_RELEASE then git("describe", "--tags", "--abbrev=0", "--match", @config.release_tags, "HEAD^", allow_failure: true)
|
|
39
|
-
else @settings.base
|
|
43
|
+
else @settings.pushed ? pushed : @settings.base
|
|
40
44
|
end
|
|
41
45
|
end
|
|
42
46
|
|
|
47
|
+
# The branch that marks the last commit Duel passed in a push to the
|
|
48
|
+
# branch pushed (Settings#pushed), for Config#push_base:
|
|
49
|
+
# duel/passed/main, or for a project in a subdirectory of its
|
|
50
|
+
# repository, with its path (duel/passed-gems-parser/main), so each
|
|
51
|
+
# branch, and each of a monorepo's projects, has its own.
|
|
52
|
+
def passed_branch = "#{prefix.empty? ? "duel/passed" : "duel/passed-#{prefix.tr("/", "-")}"}/#{@settings.pushed}"
|
|
53
|
+
|
|
54
|
+
# The last commit Duel passed in a push, as checked out from GitHub
|
|
55
|
+
# (#passed_branch, on origin).
|
|
56
|
+
def passed = "origin/#{passed_branch}"
|
|
57
|
+
|
|
43
58
|
# Whether nothing #watched changed since the baseline (only for a git
|
|
44
59
|
# baseline compared with the working tree).
|
|
45
60
|
def unchanged?
|
|
@@ -122,7 +137,7 @@ module Duel
|
|
|
122
137
|
|
|
123
138
|
# The main branch (see #ref).
|
|
124
139
|
def main_branch
|
|
125
|
-
@config.main_branch ||
|
|
140
|
+
@settings.target || @config.main_branch ||
|
|
126
141
|
git("symbolic-ref", "--short", "refs/remotes/origin/HEAD", allow_failure: true) ||
|
|
127
142
|
BRANCHES.find { |branch| git("rev-parse", branch, allow_failure: true) } ||
|
|
128
143
|
raise(Error, "no main branch to compare with (set c.main_branch in .duel.rb, or --base)")
|
|
@@ -178,6 +193,17 @@ module Duel
|
|
|
178
193
|
raise Error, "HEAD and #{branch} have no commit in common: #{why}, or compare with --base REF"
|
|
179
194
|
end
|
|
180
195
|
|
|
196
|
+
# A push's baseline: the last commit Duel passed (#passed), where
|
|
197
|
+
# Config#push_base says and HEAD has it, or else the commit before the
|
|
198
|
+
# push, or nil where that isn't here (a branch's first push, or one
|
|
199
|
+
# that rewrote history).
|
|
200
|
+
def pushed
|
|
201
|
+
return passed if @config.push_base == :last_passed && git("merge-base", "--is-ancestor", passed, "HEAD", allow_failure: true)
|
|
202
|
+
|
|
203
|
+
before = @settings.base #: String
|
|
204
|
+
before if git("cat-file", "-e", "#{before}^{commit}", allow_failure: true)
|
|
205
|
+
end
|
|
206
|
+
|
|
181
207
|
# Whether +dir+ is a checkout of this repository (a worktree of it).
|
|
182
208
|
def ours?(dir) = common_dir(dir) == common_dir(@config.root)
|
|
183
209
|
|
data/lib/duel/cli.rb
CHANGED
|
@@ -18,7 +18,8 @@ module Duel
|
|
|
18
18
|
# On GitHub Actions (GITHUB_STEP_SUMMARY), it adds its report to the
|
|
19
19
|
# job's summary, in Markdown (see Summary), and tells the step's
|
|
20
20
|
# outputs (GITHUB_OUTPUT) what to do about the comment on a pull
|
|
21
|
-
# request (see .comment)
|
|
21
|
+
# request (see .comment), and for a push, whether to mark it passed
|
|
22
|
+
# (see Config#push_base). --markdown writes the Markdown to a file, and
|
|
22
23
|
# --json the report as JSON (see JSONReport), whatever the run's
|
|
23
24
|
# outcome, but not for a command line that isn't valid.
|
|
24
25
|
# @api private
|
|
@@ -32,6 +33,10 @@ module Duel
|
|
|
32
33
|
# Where GitHub Actions has a step set its outputs.
|
|
33
34
|
STEP_OUTPUT = "GITHUB_OUTPUT"
|
|
34
35
|
|
|
36
|
+
# Where else to write them, as variables (DUEL_COMMENT, DUEL_STATUS,
|
|
37
|
+
# DUEL_PASSED): on GitLab CI, a dotenv report, for the jobs after.
|
|
38
|
+
OUTPUT = "DUEL_OUTPUT"
|
|
39
|
+
|
|
35
40
|
# The statuses (as the JSON report has them) that the workflow
|
|
36
41
|
# comments on, by when it does (Config#comment, but :never, which
|
|
37
42
|
# comments on none).
|
|
@@ -41,6 +46,16 @@ module Duel
|
|
|
41
46
|
# has it: passed, failed (or couldn't compare), or neutral.
|
|
42
47
|
MARKED = {"passed" => "passed", "failed" => "failed", "error" => "failed", "nothing_to_compare" => "neutral"}.freeze
|
|
43
48
|
|
|
49
|
+
# The GitHub Actions events of a pull request, as GITHUB_EVENT_NAME
|
|
50
|
+
# has them: a merge queue's too (see Config#fail_on). (A GitLab merge
|
|
51
|
+
# request's is pull_request: see #event.)
|
|
52
|
+
PULL_REQUESTS = %w[pull_request pull_request_target merge_group].freeze
|
|
53
|
+
|
|
54
|
+
# The exit status of a run that was slower outside a merge request on
|
|
55
|
+
# GitLab CI, where only a merge request fails (Config#fail_on): the
|
|
56
|
+
# job that `duel init --gitlab` writes lets it fail, as a warning.
|
|
57
|
+
WARNED = 3
|
|
58
|
+
|
|
44
59
|
# The variables that set options, to their values, and the one that
|
|
45
60
|
# sets a flag by being set at all.
|
|
46
61
|
ENV_OPTIONS = {"DUEL_BASE" => "--base", "DUEL_FILTER" => "--filter"}.freeze
|
|
@@ -54,7 +69,8 @@ module Duel
|
|
|
54
69
|
end
|
|
55
70
|
|
|
56
71
|
# Runs, and returns the exit status: 0 not slower (or nothing to compare),
|
|
57
|
-
# 1 slower (or a benchmark failed), 2 couldn't compare
|
|
72
|
+
# 1 slower (or a benchmark failed), 2 couldn't compare, 3 slower but
|
|
73
|
+
# warned (see WARNED).
|
|
58
74
|
def run
|
|
59
75
|
return Init.new(@argv.drop(1), out: @out, err: @err).run if @argv.first == "init"
|
|
60
76
|
|
|
@@ -91,9 +107,17 @@ module Duel
|
|
|
91
107
|
|
|
92
108
|
# The mode --comment names (see Config#comment), or an invalid
|
|
93
109
|
# argument.
|
|
94
|
-
def self.comment_mode(name)
|
|
95
|
-
|
|
96
|
-
|
|
110
|
+
def self.comment_mode(name) = named(name, Config::COMMENTS)
|
|
111
|
+
|
|
112
|
+
# The runs --fail-on names (see Config#fail_on), or an invalid
|
|
113
|
+
# argument.
|
|
114
|
+
def self.fail_on(name) = named(name, Config::FAIL_ON)
|
|
115
|
+
|
|
116
|
+
# Which of +values+ (symbols) an option's argument names, or an
|
|
117
|
+
# invalid argument.
|
|
118
|
+
def self.named(name, values)
|
|
119
|
+
values.find { |value| value.to_s == name } or
|
|
120
|
+
raise OptionParser::InvalidArgument, "#{name} (give #{values.join(", ")})"
|
|
97
121
|
end
|
|
98
122
|
|
|
99
123
|
# The settings, from the environment and then the options.
|
|
@@ -101,6 +125,11 @@ module Duel
|
|
|
101
125
|
settings = Settings.defaults
|
|
102
126
|
settings.jobs = [Etc.nprocessors / 2, 1].max
|
|
103
127
|
parser(settings).parse!(env_options + @argv)
|
|
128
|
+
branch = variable("GITHUB_REF_NAME", "CI_COMMIT_BRANCH")
|
|
129
|
+
base = variable("DUEL_BASE")
|
|
130
|
+
settings.pushed = branch if pushed? && !base.nil? && settings.base == base
|
|
131
|
+
target = variable("GITHUB_BASE_REF", "CI_MERGE_REQUEST_TARGET_BRANCH_NAME")
|
|
132
|
+
settings.target = "origin/#{target}" if target && pull_request?
|
|
104
133
|
settings.config_path ||= (".duel.rb" if File.exist?(".duel.rb"))
|
|
105
134
|
settings.config_path &&= File.expand_path(settings.config_path)
|
|
106
135
|
settings
|
|
@@ -117,9 +146,12 @@ module Duel
|
|
|
117
146
|
return 0
|
|
118
147
|
end
|
|
119
148
|
if settings.base_dir.nil? && baseline.ref.nil?
|
|
149
|
+
return nothing("no commit to compare against", "no commit before this push to compare against") if settings.pushed
|
|
150
|
+
|
|
120
151
|
return nothing("no release to compare against", "no release tag (#{config.release_tags}) before HEAD to compare against")
|
|
121
152
|
end
|
|
122
153
|
if !settings.force && baseline.unchanged?
|
|
154
|
+
@passed = marks_passed?(config, settings)
|
|
123
155
|
return nothing("nothing to compare",
|
|
124
156
|
"#{baseline.watched.join(", ")} unchanged since the baseline, so there's nothing to compare (--force to anyway)")
|
|
125
157
|
end
|
|
@@ -130,12 +162,14 @@ module Duel
|
|
|
130
162
|
uneven(base_root, candidate)
|
|
131
163
|
runner = Runner.new(config, settings, base_root, out: @out)
|
|
132
164
|
results = runner.run
|
|
133
|
-
|
|
165
|
+
passed = runner.passed?(results)
|
|
166
|
+
status = passed ? 0 : slower(config, settings)
|
|
167
|
+
@passed = (passed || accepted?(config)) && marks_passed?(config, settings)
|
|
134
168
|
publish(
|
|
135
|
-
|
|
169
|
+
passed ? "passed" : "failed",
|
|
136
170
|
-> {
|
|
137
171
|
Summary.new(runner, results, settings: settings, baseline: described_baseline(config, settings, baseline),
|
|
138
|
-
candidate: described_candidate(config, settings)).markdown
|
|
172
|
+
candidate: described_candidate(config, settings), note: @note).markdown
|
|
139
173
|
},
|
|
140
174
|
->(comment) {
|
|
141
175
|
JSONReport.new(runner, results, settings: settings, exit: status, comment: comment,
|
|
@@ -145,6 +179,107 @@ module Duel
|
|
|
145
179
|
status
|
|
146
180
|
end
|
|
147
181
|
|
|
182
|
+
# The exit status of a run that was slower (or whose benchmark
|
|
183
|
+
# failed): 1, or 0 where the pull request's label accepts it
|
|
184
|
+
# (Config#accept_label), or on GitHub Actions outside a pull request,
|
|
185
|
+
# where c.fail_on (or --fail-on) says only a pull request fails, with
|
|
186
|
+
# a warning that says so, and a note in the report.
|
|
187
|
+
def slower(config, settings)
|
|
188
|
+
if accepted?(config)
|
|
189
|
+
# (Set: see #accepted?.)
|
|
190
|
+
label = config.accept_label #: String
|
|
191
|
+
return passing("accepted with the #{label} label", "Accepted with the #{Summary.code(label)} label.")
|
|
192
|
+
end
|
|
193
|
+
return 1 unless pull_requests_only?(config, settings)
|
|
194
|
+
return 1 unless outside_pull_request?
|
|
195
|
+
|
|
196
|
+
# (Outside a pull request on CI, so there's one: see #event.)
|
|
197
|
+
run = event #: String
|
|
198
|
+
passing("only a pull request fails (c.fail_on), so this #{run} passes",
|
|
199
|
+
"Only a pull request fails (#{Summary.code("c.fail_on")}), so this #{run} passes.", gitlab? ? WARNED : 0)
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
# Passes a run that was slower (or on GitLab CI, warns: +exit+), as
|
|
203
|
+
# +why+ says, in a warning (on GitHub Actions, an annotation), and as
|
|
204
|
+
# +note+ says in the report.
|
|
205
|
+
def passing(why, note, exit = 0)
|
|
206
|
+
@out.puts "#{gitlab? ? "duel: " : "::warning::Duel: "}slower, but #{why}"
|
|
207
|
+
@note = note
|
|
208
|
+
exit
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
# Whether the pull request's labels have the one that accepts a
|
|
212
|
+
# regression (Config#accept_label).
|
|
213
|
+
def accepted?(config)
|
|
214
|
+
label = config.accept_label
|
|
215
|
+
!label.nil? && labels.include?(label)
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
# The pull request's labels: DUEL_LABELS (a JSON array of their names)
|
|
219
|
+
# on GitHub Actions, or a GitLab merge request's
|
|
220
|
+
# (CI_MERGE_REQUEST_LABELS, by commas), or none, where neither is
|
|
221
|
+
# set, or DUEL_LABELS isn't a JSON array.
|
|
222
|
+
def labels
|
|
223
|
+
set = variables
|
|
224
|
+
return set.fetch("CI_MERGE_REQUEST_LABELS").split(",") if !set.key?("DUEL_LABELS") && set.key?("CI_MERGE_REQUEST_LABELS")
|
|
225
|
+
return [] unless set.key?("DUEL_LABELS")
|
|
226
|
+
|
|
227
|
+
names = JSON.parse(set.fetch("DUEL_LABELS"))
|
|
228
|
+
names.instance_of?(Array) ? names : []
|
|
229
|
+
rescue JSON::ParserError
|
|
230
|
+
[]
|
|
231
|
+
end
|
|
232
|
+
|
|
233
|
+
# Whether the workflow marks a push that passed (or had nothing to
|
|
234
|
+
# compare) as the last commit Duel passed: where Config#push_base
|
|
235
|
+
# says to compare a push with that commit.
|
|
236
|
+
def marks_passed?(config, settings) = !settings.pushed.nil? && config.push_base == :last_passed
|
|
237
|
+
|
|
238
|
+
# Whether a regression fails only a pull request (--fail-on, or else
|
|
239
|
+
# Config#fail_on).
|
|
240
|
+
def pull_requests_only?(config, settings) = (settings.fail_on || config.fail_on) == :pull_requests
|
|
241
|
+
|
|
242
|
+
# What a run on CI is for, as GitHub Actions names it
|
|
243
|
+
# (GITHUB_EVENT_NAME): push, pull_request, schedule, say. On GitLab
|
|
244
|
+
# CI, its pipeline's source (CI_PIPELINE_SOURCE), as GitHub would
|
|
245
|
+
# name it: a merge request's is pull_request. Outside CI, nil.
|
|
246
|
+
def event
|
|
247
|
+
github = variable("GITHUB_EVENT_NAME")
|
|
248
|
+
return github if github
|
|
249
|
+
return unless gitlab?
|
|
250
|
+
|
|
251
|
+
source = variable("CI_PIPELINE_SOURCE")
|
|
252
|
+
(source == "merge_request_event") ? "pull_request" : source
|
|
253
|
+
end
|
|
254
|
+
|
|
255
|
+
# Whether this is a run on GitLab CI.
|
|
256
|
+
def gitlab? = variables.key?("GITLAB_CI")
|
|
257
|
+
|
|
258
|
+
# Whether this is a run on CI for something other than a pull request
|
|
259
|
+
# (PULL_REQUESTS): a push, say.
|
|
260
|
+
def outside_pull_request?
|
|
261
|
+
run = event
|
|
262
|
+
!run.nil? && !PULL_REQUESTS.include?(run)
|
|
263
|
+
end
|
|
264
|
+
|
|
265
|
+
# Whether this is a run on CI for a pull request (or a merge queue:
|
|
266
|
+
# see PULL_REQUESTS).
|
|
267
|
+
def pull_request? = PULL_REQUESTS.include?(event)
|
|
268
|
+
|
|
269
|
+
# Whether the baseline is a merge queue's base: on GitHub Actions for
|
|
270
|
+
# a merge queue, as DUEL_BASE gives it.
|
|
271
|
+
def queued?(settings) = event == "merge_group" && settings.base == variable("DUEL_BASE")
|
|
272
|
+
|
|
273
|
+
# Whether this is a run on CI for a push.
|
|
274
|
+
def pushed? = event == "push"
|
|
275
|
+
|
|
276
|
+
# The first of the variables +names+ that's set, or nil.
|
|
277
|
+
def variable(*names)
|
|
278
|
+
set = variables
|
|
279
|
+
name = names.find { |n| set.key?(n) }
|
|
280
|
+
set.fetch(name) if name
|
|
281
|
+
end
|
|
282
|
+
|
|
148
283
|
# Says there's nothing to compare (+title+, in Markdown), and +why+,
|
|
149
284
|
# and passes.
|
|
150
285
|
def nothing(title, why)
|
|
@@ -161,14 +296,19 @@ module Duel
|
|
|
161
296
|
exit
|
|
162
297
|
end
|
|
163
298
|
|
|
164
|
-
# What the workflow does about its comment on a pull request
|
|
165
|
-
# .comment), for a run whose +status+ is the
|
|
166
|
-
# --comment says, or else the configuration, or the
|
|
167
|
-
# configuration didn't load.
|
|
299
|
+
# What the workflow does about its comment on a pull request, or on
|
|
300
|
+
# a pushed commit (see .comment), for a run whose +status+ is the
|
|
301
|
+
# JSON report's: as --comment says, or else the configuration, or the
|
|
302
|
+
# default where the configuration didn't load. It comments on a push
|
|
303
|
+
# only where a regression in one passes (Config#fail_on), so the
|
|
304
|
+
# comment says what the check doesn't.
|
|
168
305
|
def commenting(status)
|
|
169
306
|
# (Parsed, so set: see #run.)
|
|
170
307
|
settings = @settings #: Settings
|
|
171
|
-
|
|
308
|
+
config = @config || Config.new
|
|
309
|
+
return "none" if pushed? && !pull_requests_only?(config, settings)
|
|
310
|
+
|
|
311
|
+
CLI.comment(settings.comment || config.comment, status)
|
|
172
312
|
end
|
|
173
313
|
|
|
174
314
|
# Writes the reports of a run whose +status+ is the JSON report's
|
|
@@ -177,13 +317,17 @@ module Duel
|
|
|
177
317
|
# --json names, and the Markdown +markdown+ makes to the file
|
|
178
318
|
# --markdown names, and added to the job's summary on GitHub Actions
|
|
179
319
|
# (STEP_SUMMARY). Each is only made if it's written. On GitHub
|
|
180
|
-
# Actions, it also sets the step's outputs (STEP_OUTPUT) for the
|
|
181
|
-
# that
|
|
182
|
-
#
|
|
320
|
+
# Actions, it also sets the step's outputs (STEP_OUTPUT) for the jobs
|
|
321
|
+
# that comment: +comment+, and +status+ as the comment's marker has
|
|
322
|
+
# it (MARKED), and for the job that marks a push passed, "passed"
|
|
323
|
+
# (see #marks_passed?). Where DUEL_OUTPUT names a file (OUTPUT), it
|
|
324
|
+
# writes them there too, as variables.
|
|
183
325
|
def publish(status, markdown, json)
|
|
184
326
|
comment = commenting(status)
|
|
185
327
|
set = variables
|
|
186
|
-
|
|
328
|
+
outputs = "comment=#{comment}\nstatus=#{MARKED.fetch(status)}\n#{"passed=true\n" if @passed}"
|
|
329
|
+
File.write(set.fetch(STEP_OUTPUT), outputs, mode: "a") if set.key?(STEP_OUTPUT)
|
|
330
|
+
File.write(set.fetch(OUTPUT), outputs.gsub(/^\w+=/) { |name| "DUEL_#{name.upcase}" }, mode: "a") if set.key?(OUTPUT)
|
|
187
331
|
path = @json
|
|
188
332
|
write(path, "#{JSON.pretty_generate(json.call(comment))}\n") if path
|
|
189
333
|
path = @markdown
|
|
@@ -204,7 +348,9 @@ module Duel
|
|
|
204
348
|
# The baseline, in Markdown: its directory, or its commit, bare (so
|
|
205
349
|
# GitHub links it) and followed by a space, never punctuation, and
|
|
206
350
|
# what chose it: the main branch, the push (on GitHub Actions, the
|
|
207
|
-
# commit before it, from DUEL_BASE
|
|
351
|
+
# commit before it, from DUEL_BASE, or the last commit Duel passed),
|
|
352
|
+
# the merge queue (its base, from DUEL_BASE), the last release, or
|
|
353
|
+
# --base.
|
|
208
354
|
def described_baseline(config, settings, baseline)
|
|
209
355
|
dir = settings.base_dir
|
|
210
356
|
return Summary.code(dir) if dir
|
|
@@ -212,15 +358,13 @@ module Duel
|
|
|
212
358
|
# (There's one: #compare checked.)
|
|
213
359
|
ref = baseline.ref #: String
|
|
214
360
|
commit = git(config.root, "rev-parse", "--short", "#{ref}^{commit}").first.chomp
|
|
215
|
-
set = variables
|
|
216
361
|
chosen = case settings.base
|
|
217
362
|
when nil then "where HEAD left #{Summary.code(baseline.main_branch)}"
|
|
218
363
|
when Baseline::LAST_RELEASE then "the last release, #{Summary.code(ref)}"
|
|
219
364
|
else
|
|
220
|
-
pushed
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
elsif pushed then "the commit before the push"
|
|
365
|
+
if settings.pushed then (ref == baseline.passed) ? "the last commit Duel passed" : "the commit before the push"
|
|
366
|
+
elsif queued?(settings) then "the merge queue's base"
|
|
367
|
+
elsif !ref.match?(/\A\h+\z/) then Summary.code(ref)
|
|
224
368
|
else "from --base"
|
|
225
369
|
end
|
|
226
370
|
end
|
|
@@ -248,11 +392,7 @@ module Duel
|
|
|
248
392
|
|
|
249
393
|
# (There's one: #compare checked.)
|
|
250
394
|
ref = baseline.ref #: String
|
|
251
|
-
named =
|
|
252
|
-
when nil then baseline.main_branch
|
|
253
|
-
when Baseline::LAST_RELEASE then ref
|
|
254
|
-
else settings.base
|
|
255
|
-
end
|
|
395
|
+
named = settings.base ? ref : baseline.main_branch
|
|
256
396
|
described.merge("ref" => named, "commit" => git(config.root, "rev-parse", "#{ref}^{commit}").first.chomp)
|
|
257
397
|
end
|
|
258
398
|
|
|
@@ -322,6 +462,8 @@ module Duel
|
|
|
322
462
|
o.on("--json PATH", "Also write the report to PATH as JSON, for a program to read") { |v| @json = v }
|
|
323
463
|
o.on("--comment MODE", "When the workflow comments on a pull request: always, results (passed or failed),",
|
|
324
464
|
"failures, or never (default: c.comment, or failures)") { |v| settings.comment = CLI.comment_mode(v) }
|
|
465
|
+
o.on("--fail-on RUNS", "Which runs a regression fails: every_run, or pull_requests (on GitHub Actions, another",
|
|
466
|
+
"warns and passes) (default: c.fail_on, or every_run)") { |v| settings.fail_on = CLI.fail_on(v) }
|
|
325
467
|
o.on("--clean", "Remove tmp/duel, and the baseline's git worktree, and stop") { @clean = true }
|
|
326
468
|
o.on("-v", "--version", "Print the version") do
|
|
327
469
|
@out.puts "duel #{VERSION}"
|
data/lib/duel/config.rb
CHANGED
|
@@ -95,6 +95,75 @@ module Duel
|
|
|
95
95
|
@comment = mode
|
|
96
96
|
end
|
|
97
97
|
|
|
98
|
+
# Which runs a regression fails (`duel --fail-on` overrides it):
|
|
99
|
+
#
|
|
100
|
+
# - :every_run (the default)
|
|
101
|
+
# - :pull_requests, only a pull request's (or a merge queue's), and a
|
|
102
|
+
# run outside GitHub Actions. On GitHub Actions, a regression in
|
|
103
|
+
# any other run (a push, the schedule, run by hand) passes, with a
|
|
104
|
+
# warning, and the workflow that `duel init` writes comments on a
|
|
105
|
+
# pushed commit with the report, as #comment says, which notifies
|
|
106
|
+
# its author.
|
|
107
|
+
#
|
|
108
|
+
# A run that couldn't compare fails either way.
|
|
109
|
+
#
|
|
110
|
+
# c.fail_on = :pull_requests
|
|
111
|
+
attr_reader :fail_on
|
|
112
|
+
|
|
113
|
+
# The values #fail_on takes.
|
|
114
|
+
FAIL_ON = %i[every_run pull_requests].freeze #: Array[fail_on]
|
|
115
|
+
|
|
116
|
+
def fail_on=(runs)
|
|
117
|
+
raise Error, "c.fail_on = #{runs.inspect}: give #{FAIL_ON.map(&:inspect).join(", ")}" unless FAIL_ON.include?(runs)
|
|
118
|
+
|
|
119
|
+
@fail_on = runs
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
# The name of a label that accepts a regression in a pull request
|
|
123
|
+
# (or nil, the default, for none): on GitHub Actions, the workflow
|
|
124
|
+
# that `duel init` writes passes Duel the pull request's labels
|
|
125
|
+
# (DUEL_LABELS), and with this one, a regression passes, with a
|
|
126
|
+
# warning, and the report says it was accepted. For a push, it's the
|
|
127
|
+
# labels of the pull request that merged it, so where #push_base is
|
|
128
|
+
# :last_passed, the push is marked passed too. The labels are read
|
|
129
|
+
# when the workflow runs, so after adding the label, run it again.
|
|
130
|
+
#
|
|
131
|
+
# c.accept_label = "duel: accept"
|
|
132
|
+
attr_reader :accept_label
|
|
133
|
+
|
|
134
|
+
def accept_label=(label)
|
|
135
|
+
raise Error, "c.accept_label = #{label.inspect}: give a label's name, or nil" unless label.nil? || label.instance_of?(String)
|
|
136
|
+
|
|
137
|
+
@accept_label = label
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# What the workflow that `duel init` writes compares a push with:
|
|
141
|
+
#
|
|
142
|
+
# - :before (the default), the commit before the push
|
|
143
|
+
# - :last_passed, the last commit Duel passed, which the workflow
|
|
144
|
+
# marks with a branch, duel/passed/main for main (see
|
|
145
|
+
# Baseline#passed_branch),
|
|
146
|
+
# when a push passes, or has nothing to compare. Slowdowns too
|
|
147
|
+
# small for any one push to fail then add up until they fail one,
|
|
148
|
+
# and a push the workflow didn't compare (its run cancelled, say)
|
|
149
|
+
# is compared with the next. After a regression, every push is
|
|
150
|
+
# compared with the commit before it, until one is no slower, or
|
|
151
|
+
# the branch is moved to accept the regression. Without the branch,
|
|
152
|
+
# or where HEAD doesn't have its commit (after a force push, say),
|
|
153
|
+
# a push is compared with the commit before it.
|
|
154
|
+
#
|
|
155
|
+
# c.push_base = :last_passed
|
|
156
|
+
attr_reader :push_base
|
|
157
|
+
|
|
158
|
+
# The values #push_base takes.
|
|
159
|
+
PUSH_BASES = %i[before last_passed].freeze #: Array[push_base]
|
|
160
|
+
|
|
161
|
+
def push_base=(base)
|
|
162
|
+
raise Error, "c.push_base = #{base.inspect}: give #{PUSH_BASES.map(&:inspect).join(", ")}" unless PUSH_BASES.include?(base)
|
|
163
|
+
|
|
164
|
+
@push_base = base
|
|
165
|
+
end
|
|
166
|
+
|
|
98
167
|
# The metrics #metric added, in order (classes: a Duel::Metric
|
|
99
168
|
# subclass, or running totals' Snapshot), for a plugin's tests to
|
|
100
169
|
# look at (their .measures and .settings).
|
|
@@ -115,6 +184,8 @@ module Duel
|
|
|
115
184
|
@build = :conventional
|
|
116
185
|
@release_tags = "v*"
|
|
117
186
|
@comment = :failures
|
|
187
|
+
@fail_on = :every_run
|
|
188
|
+
@push_base = :before
|
|
118
189
|
@prepares = SCOPES.to_h { |scope| [scope, []] }
|
|
119
190
|
@metrics = []
|
|
120
191
|
@in_workers = []
|
data/lib/duel/init.rb
CHANGED
|
@@ -10,15 +10,17 @@ require_relative "../duel"
|
|
|
10
10
|
module Duel
|
|
11
11
|
# `duel init`: sets Duel up in a gem or an app that isn't a Rails app
|
|
12
12
|
# (duel-rails's generator sets a Rails app up). It writes .duel.rb, a
|
|
13
|
-
# benchmark to start from, and
|
|
13
|
+
# benchmark to start from, and GitHub Actions workflows (Duel's, and
|
|
14
|
+
# one that comments on a pull request with its report), or with
|
|
15
|
+
# --gitlab, GitLab CI jobs to include, adds a Rake
|
|
14
16
|
# task to the Rakefile, and adds StackProf to the Gemfile (for profiles
|
|
15
17
|
# of regressions, on CRuby). It overwrites nothing without --force, and
|
|
16
18
|
# writes nothing with --dry-run.
|
|
17
19
|
# @api private
|
|
18
20
|
class Init
|
|
19
21
|
USAGE = "Usage: duel init [options]\n\n" \
|
|
20
|
-
"Sets Duel up in this project: .duel.rb, a benchmark to start from in duel/,
|
|
21
|
-
"a Rake task, and StackProf in the Gemfile.\n\n"
|
|
22
|
+
"Sets Duel up in this project: .duel.rb, a benchmark to start from in duel/, GitHub Actions workflows\n" \
|
|
23
|
+
"(or GitLab CI jobs), a Rake task, and StackProf in the Gemfile.\n\n"
|
|
22
24
|
|
|
23
25
|
TEMPLATES = File.join(__dir__, "templates")
|
|
24
26
|
|
|
@@ -61,8 +63,13 @@ module Duel
|
|
|
61
63
|
|
|
62
64
|
# The files it writes, from the project's root, to what's in them.
|
|
63
65
|
def files
|
|
64
|
-
|
|
65
|
-
{".duel.rb" => render("duel.rb"), benchmark_file => render("benchmark_duel.rb")
|
|
66
|
+
top = Pathname(File.realpath(repository)).relative_path_from(File.realpath(@root))
|
|
67
|
+
written = {".duel.rb" => render("duel.rb"), benchmark_file => render("benchmark_duel.rb")}
|
|
68
|
+
return written.merge(top.join(gitlab_file).to_s => render("duel.gitlab-ci.yml")) if @gitlab
|
|
69
|
+
|
|
70
|
+
workflows = top.join(".github", "workflows")
|
|
71
|
+
written.merge(workflows.join(workflow_name).to_s => render("perf.yml"),
|
|
72
|
+
workflows.join(workflow_name.sub(/\.yml\z/, "-comment.yml")).to_s => render("perf-comment.yml"))
|
|
66
73
|
end
|
|
67
74
|
|
|
68
75
|
private
|
|
@@ -72,6 +79,7 @@ module Duel
|
|
|
72
79
|
o.banner = USAGE
|
|
73
80
|
o.on("--force", "Overwrite files that exist") { @force = true }
|
|
74
81
|
o.on("--dry-run", "Say what it would do, and write nothing") { @dry_run = true }
|
|
82
|
+
o.on("--gitlab", "Write GitLab CI jobs (to include from .gitlab-ci.yml), not GitHub Actions workflows") { @gitlab = true }
|
|
75
83
|
o.on("-h", "--help", "Print this help") do
|
|
76
84
|
@out.puts o
|
|
77
85
|
throw :help
|
|
@@ -156,8 +164,13 @@ module Duel
|
|
|
156
164
|
steps << "Set c.library in .duel.rb to what each version's workers require." unless required
|
|
157
165
|
steps << "Replace the example in #{benchmark_file} with benchmarks of your own code."
|
|
158
166
|
steps << "Run `bundle exec duel --force`. It compares the code with itself, and passes when the benchmarks are sound."
|
|
159
|
-
steps <<
|
|
160
|
-
"
|
|
167
|
+
steps << if @gitlab
|
|
168
|
+
"Include #{gitlab_file} from .gitlab-ci.yml, and add the tokens it says to, for comments. On a merge request, the job " \
|
|
169
|
+
"compares the code with the branch it targets."
|
|
170
|
+
else
|
|
171
|
+
"Commit and push. On a pull request, the workflow compares the code with the branch it targets, and comments with the " \
|
|
172
|
+
"report when a benchmark fails (c.comment in .duel.rb says when)."
|
|
173
|
+
end
|
|
161
174
|
"\nNext:\n#{steps.each_with_index.map { |step, i| " #{i + 1}. #{step}\n" }.join}"
|
|
162
175
|
end
|
|
163
176
|
|
|
@@ -215,5 +228,26 @@ module Duel
|
|
|
215
228
|
|
|
216
229
|
# What marks the workflow's comment on a pull request.
|
|
217
230
|
def marker = ["duel", working_directory].compact.join(" ")
|
|
231
|
+
|
|
232
|
+
# Where the GitLab CI jobs are, from the repository's root:
|
|
233
|
+
# .gitlab/duel.gitlab-ci.yml, or for a project in a subdirectory,
|
|
234
|
+
# .gitlab/duel-<its path>.gitlab-ci.yml.
|
|
235
|
+
def gitlab_file = File.join(".gitlab", "#{["duel", working_directory&.tr("/", "-")].compact.join("-")}.gitlab-ci.yml")
|
|
236
|
+
|
|
237
|
+
# The GitLab CI job's name: duel, or for a project in a subdirectory,
|
|
238
|
+
# duel:<its path>, so each of a monorepo's projects has its own.
|
|
239
|
+
def gitlab_job = ["duel", working_directory].compact.join(":")
|
|
240
|
+
|
|
241
|
+
# The Ruby image's tag for GitLab CI: the version .ruby-version names,
|
|
242
|
+
# exactly (so it's the Ruby that c.ruby = :version_file asks for), or
|
|
243
|
+
# this one's minor version.
|
|
244
|
+
def gitlab_ruby
|
|
245
|
+
named = File.read(File.join(@root, ".ruby-version")).split.first if version_file
|
|
246
|
+
named&.delete_prefix("ruby-") || RUBY_VERSION.rpartition(".").first
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
# What names the branches that mark the last commit Duel passed in a
|
|
250
|
+
# push, before the branch pushed's name (see Baseline#passed_branch).
|
|
251
|
+
def passed_branch = ["duel/passed", working_directory&.tr("/", "-")].compact.join("-")
|
|
218
252
|
end
|
|
219
253
|
end
|