raider_reporter 0.3.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +21 -0
- data/README.md +120 -3
- data/lib/raider_reporter/azure_env.rb +36 -0
- data/lib/raider_reporter/ci_detector.rb +55 -1
- data/lib/raider_reporter/cli.rb +80 -5
- data/lib/raider_reporter/collector.rb +10 -0
- data/lib/raider_reporter/comment/azure_client.rb +110 -0
- data/lib/raider_reporter/comment/github_client.rb +1 -1
- data/lib/raider_reporter/configuration.rb +4 -2
- data/lib/raider_reporter/cucumber/formatter.rb +258 -0
- data/lib/raider_reporter/cucumber.rb +40 -0
- data/lib/raider_reporter/rspec/formatter.rb +4 -9
- data/lib/raider_reporter/shard_detector.rb +16 -2
- data/lib/raider_reporter/text.rb +19 -0
- data/lib/raider_reporter/version.rb +1 -1
- data/lib/raider_reporter.rb +10 -1
- metadata +13 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bcf65fcd172398a1040a5b15c3bff03150bde91ea1c80ddb9cb1a2ca731f1496
|
|
4
|
+
data.tar.gz: bfdaf38608619f2831d108b0a170a0da7aa8aa7eb65a1cf8d0b42703e44c58ad
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c635eb065857b147b25758e304e9a2efdf9e5885bfc1767beac062b3b9ea1dd6e344ca3463265f70f78ebb2ae3aca3f1388f8ed6f27bf425b8692e9eb846ed34
|
|
7
|
+
data.tar.gz: 8851d8bf5aa69eda7d96f42f6b8a72c893ba306fd82906b66dda19ab21dba0d8c122de1be2df94106454db4e27b1fbef06e6d8a8d4a659509763c2b4b53bee72
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,26 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.5.0 — 2026-10-12
|
|
4
|
+
|
|
5
|
+
- Add Azure Pipelines support: detect the build (`provider: azure_pipelines`, build URL, attempt, job, branch, commit), report Azure Repos pull requests in `run.git.pr`, and derive a stable id-based build key. `strategy: parallel` jobs are detected as shards of one build
|
|
6
|
+
- `raider-reporter comment` now works on Azure Pipelines: it creates or updates one tagged comment thread on the pull request (closed when the build passed) and sets a `raider/tests` PR status. Uses `System.AccessToken` with the PR statuses API version 7.1; the build service needs "Contribute to pull requests" on the repository. GitHub-hosted repositories built by Azure Pipelines are skipped with a notice
|
|
7
|
+
- Schema v1, additive: optional `run.git.pr.url` (set by GitHub Actions and Azure Pipelines)
|
|
8
|
+
- Unexpanded Azure macros such as `$(RAIDER_TOKEN)` or `$(System.AccessToken)` are treated as missing instead of being sent as a token
|
|
9
|
+
- Fix: redact the token before truncating error bodies in the GitHub and Azure clients, so a token straddling the 200-character cut can no longer leak partially
|
|
10
|
+
- Notices: the missing-permission notice uses `<project>` when the project name is unknown, and the success notice prints `PR #<n>` when the PR URL is unknown
|
|
11
|
+
- `raider-reporter comment` now runs only on GitHub Actions (`GITHUB_ACTIONS=true`) or Azure Pipelines; elsewhere it prints `skipped: unsupported CI` and exits 0
|
|
12
|
+
- No breaking changes
|
|
13
|
+
|
|
14
|
+
## 0.4.0 — 2026-10-11
|
|
15
|
+
|
|
16
|
+
- Add a Cucumber reporter (Cucumber >= 9, < 12; not a gem dependency). `require "raider_reporter/cucumber"` from `features/support/` registers it in plugin mode (recommended, works with any `--format` and no `--out`); `--format RaiderReporter::Cucumber::Formatter --out <file>` also works. Using both registers one listener
|
|
17
|
+
- Identity: `file_path` is project-relative (`features/login.feature`), `full_name` is `"<Feature>: <Scenario>"` (outline rows add `(<Examples name>, row <n>)`; when several Examples tables share a label, each is numbered: `(Examples 1, row 1)`, `(Examples 2, row 1)`), `identifier` is `<file>:<line>`
|
|
18
|
+
- Report the failing step (or Before/After hook) with its location in the failure message; ambiguous maps to failed, undefined to pending
|
|
19
|
+
- Retries: only the final attempt is kept, with `attempts` set and earlier attachments dropped
|
|
20
|
+
- Attachments: `RaiderReporter.attach` and Cucumber's `attach(path, media_type)` with an existing file path are captured; inline data and `log` messages are ignored
|
|
21
|
+
- Dry runs (`cucumber --dry-run`) write and upload nothing; `parallel_tests` shards write `run-<n>.json`
|
|
22
|
+
- No breaking changes: RSpec behaviour is unchanged
|
|
23
|
+
|
|
3
24
|
## 0.3.0 — 2026-10-11
|
|
4
25
|
|
|
5
26
|
- Report pull request identity: `run.ci.repository`, `run.git.default_branch` and `run.git.pr` (number, base/head branch, head SHA) on `pull_request` and `pull_request_target` events. Requires a Raider server that accepts `git.pr`
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# raider_reporter
|
|
2
2
|
|
|
3
|
-
Raider-schema test reporting for RSpec. `raider_reporter` is an RSpec formatter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
|
|
3
|
+
Raider-schema test reporting for RSpec and Cucumber. `raider_reporter` is an RSpec formatter and a Cucumber reporter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
|
|
4
4
|
|
|
5
5
|
It has no runtime dependencies and requires Ruby 3.2 or newer.
|
|
6
6
|
|
|
@@ -26,7 +26,80 @@ Enable the formatter in `.rspec`:
|
|
|
26
26
|
--format RaiderReporter::RSpec::Formatter
|
|
27
27
|
```
|
|
28
28
|
|
|
29
|
-
Run `bundle exec rspec` as usual. Results land in `raider-results/run.json`. A dry run (`rspec --dry-run`) writes and uploads nothing.
|
|
29
|
+
Run `bundle exec rspec` as usual. (For Cucumber, see [Cucumber](#cucumber).) Results land in `raider-results/run.json`. A dry run (`rspec --dry-run`) writes and uploads nothing.
|
|
30
|
+
|
|
31
|
+
## Cucumber
|
|
32
|
+
|
|
33
|
+
Requirements: Cucumber 9, 10 or 11 in your own Gemfile (`gem "cucumber"`). It is not a dependency of this gem; requiring `raider_reporter/cucumber` without Cucumber raises a `LoadError` that says so. Cucumber 9.2.1 on Ruby 4 also needs `gem "logger"` in your Gemfile, because Cucumber requires it without declaring it.
|
|
34
|
+
|
|
35
|
+
### Plugin mode (recommended)
|
|
36
|
+
|
|
37
|
+
Require the reporter from a file under `features/support/`, for example `features/support/raider.rb`:
|
|
38
|
+
|
|
39
|
+
```ruby
|
|
40
|
+
require "raider_reporter/cucumber"
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Then run Cucumber with any normal formatter and no `--out`:
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
# cucumber.yml
|
|
47
|
+
default: --format pretty
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The require registers one listener on Cucumber's event bus, so it sits next to `pretty`, `progress` or any other formatter. Results land in `raider-results/run.json`, and a dry run (`cucumber --dry-run`) writes and uploads nothing. Plugin mode registers once per Ruby process, because `require` is cached: a second in-process `Cucumber::Cli::Main` run in the same process does not register again.
|
|
51
|
+
|
|
52
|
+
### Formatter mode
|
|
53
|
+
|
|
54
|
+
You can also name the formatter explicitly:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
cucumber --format pretty --format RaiderReporter::Cucumber::Formatter --out raider-results/cucumber.log
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
`--out` is required here. Cucumber rejects a second formatter on the same output stream ("All but one formatter must use --out", also in 9.2.1), and this formatter writes nothing to its stream, so give it any file path (it is never created). If both modes are active, only one listener is registered.
|
|
61
|
+
|
|
62
|
+
### How tests are identified
|
|
63
|
+
|
|
64
|
+
- `file_path` is relative to the project root (`features/login.feature`), even when Cucumber is given `./features/login.feature` or an absolute path.
|
|
65
|
+
- `full_name` is `"<Feature name>: <Scenario name>"`. Scenario Outline rows are `"<Feature name>: <Scenario name> (<Examples name or 'Examples'>, row <n>)"`, where `n` is the 1-based data row of that Examples table. When two or more Examples tables of one outline share a label (for example two unnamed ones), every table with that label is numbered by its position among them, `(Examples 1, row 1)`, `(Examples 2, row 1)`, so each row keeps a distinct name; tables with a unique label are unchanged.
|
|
66
|
+
- `identifier` is `"<file_path>:<line>"`: the scenario line, or the Examples row line for outline rows.
|
|
67
|
+
- `tags` are the feature, scenario and Examples tags, merged, sorted, unique, without `@`.
|
|
68
|
+
|
|
69
|
+
Statuses: passed, failed and ambiguous (reported as failed), pending and undefined (reported as pending), skipped.
|
|
70
|
+
|
|
71
|
+
### Failures
|
|
72
|
+
|
|
73
|
+
A failed test reports the step that failed, not the whole scenario:
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
When I fail on purpose (features/login.feature:9)
|
|
77
|
+
|
|
78
|
+
expected 5, got 4
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
A failing `Before` or `After` hook is named `Before hook` or `After hook` in place of the step, so an `After` hook that raises after every step passed is still a failed test. Messages are capped at 10,000 characters and backtraces at 50 lines.
|
|
82
|
+
|
|
83
|
+
### Retries
|
|
84
|
+
|
|
85
|
+
With `--retry n`, only the final attempt of each scenario is reported, and `attempts` is the number of attempts. Each attempt starts with fresh attachments; attachments from earlier attempts are dropped.
|
|
86
|
+
|
|
87
|
+
### Attachments
|
|
88
|
+
|
|
89
|
+
Both of these are captured while a scenario runs:
|
|
90
|
+
|
|
91
|
+
```ruby
|
|
92
|
+
After do |scenario|
|
|
93
|
+
attach(path, "image/png") if scenario.failed? # Cucumber's own attach, with an existing file path (a Pathname works too)
|
|
94
|
+
RaiderReporter.attach("screenshot", path, type: :png) # or the Raider helper, as in RSpec
|
|
95
|
+
end
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
Cucumber's `attach` is captured only when given the path of an existing file. Inline data (`attach("some text", "text/plain")`) and `log` messages are ignored.
|
|
99
|
+
|
|
100
|
+
### Parallel runs with Cucumber
|
|
101
|
+
|
|
102
|
+
`parallel_tests` works with no configuration for Cucumber too: each process writes `run-<n>.json`. See [Parallel runs](#parallel-runs).
|
|
30
103
|
|
|
31
104
|
## Attaching screenshots and videos
|
|
32
105
|
|
|
@@ -122,6 +195,49 @@ On pull requests, `raider-reporter comment` posts one comment with the build's r
|
|
|
122
195
|
- The command never fails your CI: it always exits 0 and prints notices to stderr, prefixed `[raider_reporter]`. It needs `GITHUB_TOKEN`, `GITHUB_REPOSITORY`, `GITHUB_EVENT_PATH`, `RAIDER_TOKEN` and `RAIDER_ENDPOINT`; if one is missing, or the event is not a pull request, it does nothing.
|
|
123
196
|
- It requires a Raider server that supports the summary endpoint and the `git.pr` field. Upgrade the server before using it.
|
|
124
197
|
|
|
198
|
+
## Azure Pipelines
|
|
199
|
+
|
|
200
|
+
The gem detects Azure Pipelines when `TF_BUILD` is `true`. It reports the build (`provider: azure_pipelines`, build URL, attempt, job), the branch and commit, the pull request and a build key, so shards of one build group together. On a pull request from Azure Repos, `raider-reporter comment` posts one comment thread with the results and sets a `raider/tests` status on the PR.
|
|
201
|
+
|
|
202
|
+
```yaml
|
|
203
|
+
jobs:
|
|
204
|
+
- job: tests
|
|
205
|
+
steps:
|
|
206
|
+
- task: UseRubyVersion@0
|
|
207
|
+
inputs: { versionSpec: '3.4' }
|
|
208
|
+
- script: bundle install
|
|
209
|
+
displayName: Install gems
|
|
210
|
+
- script: bundle exec rspec spec/ --format documentation
|
|
211
|
+
displayName: Run tests
|
|
212
|
+
env:
|
|
213
|
+
RAIDER_TOKEN: $(RAIDER_TOKEN)
|
|
214
|
+
RAIDER_ENDPOINT: $(RAIDER_ENDPOINT)
|
|
215
|
+
- task: PublishPipelineArtifact@1
|
|
216
|
+
condition: always()
|
|
217
|
+
inputs: { targetPath: raider-results, artifact: raider-results }
|
|
218
|
+
- job: raider_comment
|
|
219
|
+
dependsOn: tests
|
|
220
|
+
condition: and(always(), eq(variables['Build.Reason'], 'PullRequest'))
|
|
221
|
+
steps:
|
|
222
|
+
- task: UseRubyVersion@0
|
|
223
|
+
inputs: { versionSpec: '3.4' }
|
|
224
|
+
- script: bundle install
|
|
225
|
+
- script: bundle exec raider-reporter comment
|
|
226
|
+
displayName: Comment Raider results on the pull request
|
|
227
|
+
env:
|
|
228
|
+
RAIDER_TOKEN: $(RAIDER_TOKEN)
|
|
229
|
+
RAIDER_ENDPOINT: $(RAIDER_ENDPOINT)
|
|
230
|
+
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
- **One-time grant:** give "Contribute to pull requests" to `<Project> Build Service (<org>)` on the repository (Project settings > Repositories > Security). Without it the command prints a notice naming the exact identity and exits 0.
|
|
234
|
+
- **Map secrets explicitly:** Azure does not expose secret variables to scripts automatically. Map `RAIDER_TOKEN` and `System.AccessToken` with `env:` as shown. If a mapped variable is undefined, Azure passes the literal `$(...)`; the gem treats it as missing: results are still written locally but not uploaded, and `raider-reporter comment` prints a skip notice.
|
|
235
|
+
- **Pull request builds:** for Azure Repos, YAML `pr:` triggers are ignored. PR builds (and therefore the comment job) run only when the pipeline is added as a **Build validation** branch policy on the target branch.
|
|
236
|
+
- **Same stage:** keep the comment job in the same stage as the test jobs; the build key includes `System.StageAttempt`.
|
|
237
|
+
- **Azure Repos only:** the PR comment works for repositories hosted in Azure Repos. A GitHub-hosted repository built by Azure Pipelines still uploads results, but `comment` prints `skipped: not an Azure Repos pull request`.
|
|
238
|
+
- **Parallel jobs:** `strategy: parallel: N` shards are detected automatically and grouped into one build, like GitHub matrix jobs.
|
|
239
|
+
- The command never fails your pipeline: it always exits 0 and prints notices to stderr, prefixed `[raider_reporter]`. It needs `SYSTEM_ACCESSTOKEN`, `RAIDER_TOKEN` and `RAIDER_ENDPOINT`, and requires a Raider server that supports the summary endpoint.
|
|
240
|
+
|
|
125
241
|
## Parallel runs
|
|
126
242
|
|
|
127
243
|
When a suite is split across several processes or CI jobs, each one uploads its own run. Raider groups runs of the same CI build, so the shards show up together as one build. Each run records its shard (`index`, `total` and a `label`) and, on GitHub Actions, a build key made from the repository, run id and run attempt. A re-run attempt of a workflow is a separate build.
|
|
@@ -183,6 +299,7 @@ Tip: run `rm -rf raider-results` before a parallel run. Each process only cleans
|
|
|
183
299
|
## Examples
|
|
184
300
|
|
|
185
301
|
- `examples/rspec_basic` is a runnable RSpec project with a passing, a failing (with an attachment) and a skipped example. Run it with `cd examples/rspec_basic && bundle install && bundle exec rspec`; the failure is deliberate.
|
|
302
|
+
- `examples/cucumber_basic` is the same for Cucumber (plugin mode): a passing scenario, a failing one with an attachment, and an outline. Run it with `cd examples/cucumber_basic && bundle install && bundle exec cucumber`; the failure is deliberate.
|
|
186
303
|
- `examples/github_actions.yml` shows the workflow steps for CI.
|
|
187
304
|
|
|
188
305
|
## Schema
|
|
@@ -191,7 +308,7 @@ Tip: run `rm -rf raider-results` before a parallel run. Each process only cleans
|
|
|
191
308
|
|
|
192
309
|
## Contributing
|
|
193
310
|
|
|
194
|
-
Bug reports and pull requests are welcome at https://github.com/RaiderHQ/raider_reporter. Run `bundle install && bundle exec rake` to run the specs.
|
|
311
|
+
Bug reports and pull requests are welcome at https://github.com/RaiderHQ/raider_reporter. Run `bundle install && bundle exec rake` to run the specs. To test against another Cucumber, set `CUCUMBER_VERSION` (for example `CUCUMBER_VERSION="~> 9.0" bundle update cucumber && bundle exec rspec`).
|
|
195
312
|
|
|
196
313
|
## License
|
|
197
314
|
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RaiderReporter
|
|
4
|
+
# Reads Azure Pipelines variables. An undefined macro is left literal by Azure
|
|
5
|
+
# ("$(System.AccessToken)"), so such values count as missing.
|
|
6
|
+
module AzureEnv
|
|
7
|
+
MACRO = /\A\$\(.+\)\z/
|
|
8
|
+
|
|
9
|
+
module_function
|
|
10
|
+
|
|
11
|
+
def value(env, name)
|
|
12
|
+
raw = env[name].to_s.strip
|
|
13
|
+
raw.empty? || raw.match?(MACRO) ? nil : raw
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def azure?(env)
|
|
17
|
+
env["GITHUB_ACTIONS"] != "true" && value(env, "TF_BUILD")&.casecmp?("true") == true
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def collection_base(env)
|
|
21
|
+
uri = value(env, "SYSTEM_COLLECTIONURI")
|
|
22
|
+
uri && "#{uri.sub(%r{/+\z}, '')}/"
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def azure_repos_pr?(env)
|
|
26
|
+
value(env, "BUILD_REASON") == "PullRequest" && value(env, "BUILD_REPOSITORY_PROVIDER") == "TfsGit"
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def strip_heads(ref)
|
|
30
|
+
return nil if ref.nil?
|
|
31
|
+
return ref.delete_prefix("refs/heads/") if ref.start_with?("refs/heads/")
|
|
32
|
+
|
|
33
|
+
ref.start_with?("refs/") ? nil : ref
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "erb"
|
|
3
4
|
require "open3"
|
|
5
|
+
require "raider_reporter/azure_env"
|
|
4
6
|
require "raider_reporter/github_event"
|
|
5
7
|
|
|
6
8
|
module RaiderReporter
|
|
@@ -14,6 +16,7 @@ module RaiderReporter
|
|
|
14
16
|
end
|
|
15
17
|
|
|
16
18
|
def ci
|
|
19
|
+
return azure_ci if AzureEnv.azure?(@env)
|
|
17
20
|
return nil unless @env["GITHUB_ACTIONS"] == "true"
|
|
18
21
|
|
|
19
22
|
run_id = @env["GITHUB_RUN_ID"]
|
|
@@ -38,6 +41,40 @@ module RaiderReporter
|
|
|
38
41
|
|
|
39
42
|
private
|
|
40
43
|
|
|
44
|
+
def azure_ci
|
|
45
|
+
v = ->(name) { AzureEnv.value(@env, name) }
|
|
46
|
+
base = AzureEnv.collection_base(@env)
|
|
47
|
+
project = v.("SYSTEM_TEAMPROJECT")
|
|
48
|
+
build_id = v.("BUILD_BUILDID")
|
|
49
|
+
attempt = v.("SYSTEM_STAGEATTEMPT") || "1"
|
|
50
|
+
build_url = ("#{base}#{ERB::Util.url_encode(project)}/_build/results?buildId=#{build_id}" if base && project && build_id)
|
|
51
|
+
ids = [base&.sub(%r{\Ahttps?://}i, "")&.chomp("/"), v.("SYSTEM_TEAMPROJECTID"), v.("BUILD_REPOSITORY_ID")]
|
|
52
|
+
build_key = ("azure_pipelines:#{ids.join('/')}:#{build_id}:#{attempt}" if ids.all? && build_id)
|
|
53
|
+
repo = v.("BUILD_REPOSITORY_NAME")
|
|
54
|
+
{ "provider" => "azure_pipelines", "build_url" => build_url, "build_id" => build_id, "attempt" => attempt,
|
|
55
|
+
"job" => v.("SYSTEM_JOBDISPLAYNAME"), "build_key" => build_key,
|
|
56
|
+
"repository" => (project && repo ? "#{project}/#{repo}" : nil) }.compact
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def azure_pull_request
|
|
60
|
+
return nil unless AzureEnv.azure_repos_pr?(@env)
|
|
61
|
+
|
|
62
|
+
v = ->(name) { AzureEnv.value(@env, name) }
|
|
63
|
+
number = Integer(v.("SYSTEM_PULLREQUEST_PULLREQUESTID").to_s, exception: false)
|
|
64
|
+
base = AzureEnv.strip_heads(v.("SYSTEM_PULLREQUEST_TARGETBRANCH"))
|
|
65
|
+
head = AzureEnv.strip_heads(v.("SYSTEM_PULLREQUEST_SOURCEBRANCH"))
|
|
66
|
+
sha = v.("SYSTEM_PULLREQUEST_SOURCECOMMITID")
|
|
67
|
+
return nil unless number&.positive? && [base, head, sha].all? { |s| s.is_a?(String) && !s.empty? }
|
|
68
|
+
|
|
69
|
+
collection = AzureEnv.collection_base(@env)
|
|
70
|
+
project = v.("SYSTEM_TEAMPROJECT")
|
|
71
|
+
repo = v.("BUILD_REPOSITORY_NAME")
|
|
72
|
+
url = if collection && project && repo
|
|
73
|
+
"#{collection}#{ERB::Util.url_encode(project)}/_git/#{ERB::Util.url_encode(repo)}/pullrequest/#{number}"
|
|
74
|
+
end
|
|
75
|
+
{ "number" => number, "base_branch" => base, "head_branch" => head, "head_sha" => sha, "url" => url }.compact
|
|
76
|
+
end
|
|
77
|
+
|
|
41
78
|
def event
|
|
42
79
|
return @event if defined?(@event)
|
|
43
80
|
|
|
@@ -57,6 +94,7 @@ module RaiderReporter
|
|
|
57
94
|
end
|
|
58
95
|
|
|
59
96
|
def pull_request
|
|
97
|
+
return azure_pull_request if AzureEnv.azure?(@env)
|
|
60
98
|
return nil unless event && PR_EVENTS.include?(@env["GITHUB_EVENT_NAME"])
|
|
61
99
|
|
|
62
100
|
pr = event["pull_request"]
|
|
@@ -69,7 +107,10 @@ module RaiderReporter
|
|
|
69
107
|
return nil unless number.is_a?(Integer) && number >= 1
|
|
70
108
|
return nil unless [base, head, sha].all? { |v| v.is_a?(String) && !v.empty? }
|
|
71
109
|
|
|
72
|
-
|
|
110
|
+
server = @env["GITHUB_SERVER_URL"].to_s
|
|
111
|
+
repo = @env["GITHUB_REPOSITORY"].to_s
|
|
112
|
+
url = ("#{server}/#{repo}/pull/#{number}" unless server.empty? || repo.empty?)
|
|
113
|
+
{ "number" => number, "base_branch" => base, "head_branch" => head, "head_sha" => sha, "url" => url }.compact
|
|
73
114
|
rescue StandardError
|
|
74
115
|
nil
|
|
75
116
|
end
|
|
@@ -88,6 +129,8 @@ module RaiderReporter
|
|
|
88
129
|
end
|
|
89
130
|
|
|
90
131
|
def branch
|
|
132
|
+
return azure_branch if AzureEnv.azure?(@env)
|
|
133
|
+
|
|
91
134
|
head_ref = @env["GITHUB_HEAD_REF"].to_s
|
|
92
135
|
return head_ref unless head_ref.empty?
|
|
93
136
|
|
|
@@ -97,7 +140,18 @@ module RaiderReporter
|
|
|
97
140
|
local == "HEAD" ? nil : local # "HEAD" means detached, not a branch name
|
|
98
141
|
end
|
|
99
142
|
|
|
143
|
+
def azure_branch
|
|
144
|
+
pr = pull_request
|
|
145
|
+
name = pr ? pr["head_branch"] : AzureEnv.strip_heads(AzureEnv.value(@env, "BUILD_SOURCEBRANCH"))
|
|
146
|
+
return name if name
|
|
147
|
+
|
|
148
|
+
local = @git.call("rev-parse", "--abbrev-ref", "HEAD")
|
|
149
|
+
local == "HEAD" ? nil : local
|
|
150
|
+
end
|
|
151
|
+
|
|
100
152
|
def sha
|
|
153
|
+
return AzureEnv.value(@env, "BUILD_SOURCEVERSION") || @git.call("rev-parse", "HEAD") if AzureEnv.azure?(@env)
|
|
154
|
+
|
|
101
155
|
@env["GITHUB_SHA"] || @git.call("rev-parse", "HEAD")
|
|
102
156
|
end
|
|
103
157
|
|
data/lib/raider_reporter/cli.rb
CHANGED
|
@@ -8,7 +8,13 @@ module RaiderReporter
|
|
|
8
8
|
class CLI
|
|
9
9
|
MARKER = "<!-- raider-reporter:build-comment -->"
|
|
10
10
|
CONTEXT = "raider/tests"
|
|
11
|
-
|
|
11
|
+
GITHUB_REQUIRED_ENV = %w[GITHUB_TOKEN GITHUB_REPOSITORY GITHUB_EVENT_PATH RAIDER_TOKEN RAIDER_ENDPOINT].freeze
|
|
12
|
+
REQUIRED_ENV = GITHUB_REQUIRED_ENV
|
|
13
|
+
AZURE_REQUIRED_ENV = %w[SYSTEM_ACCESSTOKEN SYSTEM_COLLECTIONURI SYSTEM_TEAMPROJECTID BUILD_REPOSITORY_ID
|
|
14
|
+
RAIDER_TOKEN RAIDER_ENDPOINT].freeze
|
|
15
|
+
THREAD_STATUS = { "passed" => "closed" }.freeze
|
|
16
|
+
AZURE_STATES = { "passed" => "succeeded", "failed" => "failed", "incomplete" => "pending" }.freeze
|
|
17
|
+
TOKEN_HINT = "map it with env: SYSTEM_ACCESSTOKEN: $(System.AccessToken)"
|
|
12
18
|
STATUS_STATES = { "passed" => "success", "failed" => "failure", "incomplete" => "pending" }.freeze
|
|
13
19
|
FALLBACK_DESCRIPTIONS = {
|
|
14
20
|
"passed" => "Raider: tests passed",
|
|
@@ -19,10 +25,14 @@ module RaiderReporter
|
|
|
19
25
|
USAGE = <<~TEXT
|
|
20
26
|
Usage: raider-reporter comment
|
|
21
27
|
|
|
22
|
-
Posts (or updates) the Raider build summary
|
|
23
|
-
|
|
28
|
+
Posts (or updates) the Raider build summary on the current pull request and sets a status.
|
|
29
|
+
Intended for GitHub Actions and Azure Pipelines (Azure Repos); always exits 0.
|
|
24
30
|
|
|
25
|
-
|
|
31
|
+
GitHub Actions: comment + "#{CONTEXT}" commit status.
|
|
32
|
+
Environment: #{GITHUB_REQUIRED_ENV.join(', ')} (GITHUB_API_URL optional).
|
|
33
|
+
Azure Pipelines: one PR thread updated in place + a "raider/tests" PR status.
|
|
34
|
+
Environment: #{AZURE_REQUIRED_ENV.join(', ')}.
|
|
35
|
+
Map the token in the pipeline: env: SYSTEM_ACCESSTOKEN: $(System.AccessToken)
|
|
26
36
|
TEXT
|
|
27
37
|
|
|
28
38
|
def initialize(argv, env: ENV, out: $stdout, err: $stderr)
|
|
@@ -48,7 +58,17 @@ module RaiderReporter
|
|
|
48
58
|
private
|
|
49
59
|
|
|
50
60
|
def comment
|
|
51
|
-
|
|
61
|
+
if @env["GITHUB_ACTIONS"] == "true"
|
|
62
|
+
github_comment
|
|
63
|
+
elsif AzureEnv.azure?(@env)
|
|
64
|
+
azure_comment
|
|
65
|
+
else
|
|
66
|
+
notice("skipped: unsupported CI")
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def github_comment
|
|
71
|
+
missing = GITHUB_REQUIRED_ENV.select { |name| @env[name].to_s.strip.empty? }
|
|
52
72
|
return notice("skipped: missing #{missing.join(', ')}") unless missing.empty?
|
|
53
73
|
|
|
54
74
|
event = GithubEvent.load(@env)
|
|
@@ -66,6 +86,61 @@ module RaiderReporter
|
|
|
66
86
|
notice("comment failed: #{e.class}: #{e.message}")
|
|
67
87
|
end
|
|
68
88
|
|
|
89
|
+
def azure_comment
|
|
90
|
+
missing = AZURE_REQUIRED_ENV.select { |name| AzureEnv.value(@env, name).nil? }
|
|
91
|
+
return notice(azure_missing_message(missing)) unless missing.empty?
|
|
92
|
+
|
|
93
|
+
detector = CiDetector.new(env: @env, git: ->(*) {})
|
|
94
|
+
pr = detector.git&.dig("pr")
|
|
95
|
+
return notice("skipped: not an Azure Repos pull request") unless pr
|
|
96
|
+
|
|
97
|
+
build_key = detector.ci&.dig("build_key")
|
|
98
|
+
return notice("skipped: could not determine the build key") unless build_key
|
|
99
|
+
|
|
100
|
+
publish_azure(pr, build_key)
|
|
101
|
+
rescue Comment::AzureClient::Forbidden
|
|
102
|
+
notice(azure_forbidden_message)
|
|
103
|
+
rescue StandardError => e
|
|
104
|
+
notice("comment failed: #{e.class}: #{e.message}")
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def azure_missing_message(missing)
|
|
108
|
+
message = "skipped: missing #{missing.join(', ')}"
|
|
109
|
+
message += "; #{TOKEN_HINT}" if missing.include?("SYSTEM_ACCESSTOKEN")
|
|
110
|
+
message
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
def publish_azure(pr, build_key)
|
|
114
|
+
summaries = Comment::SummaryClient.new(endpoint: AzureEnv.value(@env, "RAIDER_ENDPOINT"),
|
|
115
|
+
token: AzureEnv.value(@env, "RAIDER_TOKEN"))
|
|
116
|
+
summary = summaries.fetch(build_key)
|
|
117
|
+
return notice("summary unavailable: #{summaries.last_error}") unless summary
|
|
118
|
+
|
|
119
|
+
azure = Comment::AzureClient.new(token: AzureEnv.value(@env, "SYSTEM_ACCESSTOKEN"), base_url: azure_pr_url(pr))
|
|
120
|
+
result = azure.upsert_thread(summary.markdown, MARKER, status: THREAD_STATUS.fetch(summary.state, "active"))
|
|
121
|
+
azure.set_status(state: AZURE_STATES.fetch(summary.state, "pending"), description: description(summary),
|
|
122
|
+
target_url: summary.url)
|
|
123
|
+
notice("PR thread #{result}: #{pr['url'] || "PR ##{pr['number']}"}")
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
def azure_pr_url(pr)
|
|
127
|
+
"#{AzureEnv.collection_base(@env)}#{AzureEnv.value(@env, 'SYSTEM_TEAMPROJECTID')}/_apis/git/repositories/" \
|
|
128
|
+
"#{AzureEnv.value(@env, 'BUILD_REPOSITORY_ID')}/pullRequests/#{pr['number']}"
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
def azure_forbidden_message
|
|
132
|
+
project = AzureEnv.value(@env, "SYSTEM_TEAMPROJECT") || "<project>"
|
|
133
|
+
"skipped: the build service cannot write to this PR — grant \"Contribute to pull requests\" to " \
|
|
134
|
+
"\"#{project} Build Service (#{azure_org})\""
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
def azure_org
|
|
138
|
+
uri = URI.parse(AzureEnv.collection_base(@env).to_s)
|
|
139
|
+
uri.host.to_s.end_with?("visualstudio.com") ? uri.host.split(".").first : uri.path.split("/").reject(&:empty?).first
|
|
140
|
+
rescue URI::InvalidURIError
|
|
141
|
+
nil
|
|
142
|
+
end
|
|
143
|
+
|
|
69
144
|
def publish(pr, build_key)
|
|
70
145
|
summaries = Comment::SummaryClient.new(endpoint: @env["RAIDER_ENDPOINT"], token: @env["RAIDER_TOKEN"])
|
|
71
146
|
summary = summaries.fetch(build_key)
|
|
@@ -55,8 +55,18 @@ module RaiderReporter
|
|
|
55
55
|
end
|
|
56
56
|
|
|
57
57
|
def finish_test(**fields)
|
|
58
|
+
add_result(build_test(**fields))
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# Closes the current test and returns its result without recording it, so a
|
|
62
|
+
# framework that retries can hold one result per test and replace it per attempt.
|
|
63
|
+
def build_test(**fields)
|
|
58
64
|
result = TestResult.new(**fields, attachments: @pending_attachments || [])
|
|
59
65
|
@pending_attachments = nil
|
|
66
|
+
result
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def add_result(result)
|
|
60
70
|
@results << result
|
|
61
71
|
result
|
|
62
72
|
end
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "net/http"
|
|
4
|
+
require "uri"
|
|
5
|
+
require "json"
|
|
6
|
+
|
|
7
|
+
module RaiderReporter
|
|
8
|
+
module Comment
|
|
9
|
+
# Minimal Azure DevOps REST client: one Raider thread per pull request, plus a PR status.
|
|
10
|
+
class AzureClient
|
|
11
|
+
class Error < StandardError; end
|
|
12
|
+
# 401/403/404: the build service identity cannot write to this PR.
|
|
13
|
+
class Forbidden < Error; end
|
|
14
|
+
|
|
15
|
+
API_VERSION = "7.1"
|
|
16
|
+
STATUS_API_VERSION = "7.1" # confirmed in the 7.1 Pull Request Statuses doc; see spec header
|
|
17
|
+
PROPERTY = "Raider.BuildComment"
|
|
18
|
+
DESCRIPTION_LIMIT = 140
|
|
19
|
+
|
|
20
|
+
def initialize(token:, base_url:, timeout: 30)
|
|
21
|
+
@token = token
|
|
22
|
+
@base_url = base_url.to_s.sub(%r{/+\z}, "")
|
|
23
|
+
@timeout = timeout
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def upsert_thread(markdown, marker, status:)
|
|
27
|
+
thread = find_thread(marker)
|
|
28
|
+
if thread
|
|
29
|
+
comment_id = thread["comments"].first["id"]
|
|
30
|
+
request(:patch, "/threads/#{thread['id']}/comments/#{comment_id}", { content: markdown })
|
|
31
|
+
request(:patch, "/threads/#{thread['id']}", { status: status })
|
|
32
|
+
:updated
|
|
33
|
+
else
|
|
34
|
+
request(:post, "/threads", { comments: [{ parentCommentId: 0, content: markdown, commentType: 1 }],
|
|
35
|
+
status: status,
|
|
36
|
+
properties: { PROPERTY => { "$type" => "System.Int32", "$value" => 1 } } })
|
|
37
|
+
:created
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def set_status(state:, description:, target_url:)
|
|
42
|
+
request(:post, "/statuses", { state: state, description: description.to_s[0, DESCRIPTION_LIMIT],
|
|
43
|
+
targetUrl: target_url, context: { genre: "raider", name: "tests" } },
|
|
44
|
+
version: STATUS_API_VERSION)
|
|
45
|
+
nil
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
def find_thread(marker)
|
|
51
|
+
list = request(:get, "/threads")
|
|
52
|
+
threads = list.is_a?(Hash) ? list["value"] : nil
|
|
53
|
+
raise Error, "unexpected threads response" unless threads.is_a?(Array)
|
|
54
|
+
|
|
55
|
+
threads.find { |t| ours?(t, marker) }
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Only a property-tagged thread whose live first comment starts with the marker is ours.
|
|
59
|
+
def ours?(thread, marker)
|
|
60
|
+
return false unless thread.is_a?(Hash) && thread["properties"].is_a?(Hash) && thread["properties"].key?(PROPERTY)
|
|
61
|
+
return false unless thread["comments"].is_a?(Array)
|
|
62
|
+
|
|
63
|
+
first = thread["comments"].first
|
|
64
|
+
first.is_a?(Hash) && !first["isDeleted"] && first["content"].to_s.start_with?(marker)
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def redact(text)
|
|
68
|
+
@token.to_s.empty? ? text.to_s : text.to_s.gsub(@token.to_s, "[REDACTED]")
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def request(verb, path, payload = nil, version: nil)
|
|
72
|
+
uri = URI.parse("#{@base_url}#{path}?api-version=#{version || API_VERSION}")
|
|
73
|
+
raise Error, "Azure DevOps URL must start with http:// or https://" unless uri.is_a?(URI::HTTP)
|
|
74
|
+
|
|
75
|
+
req = build_request(verb, uri, payload)
|
|
76
|
+
response = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
|
|
77
|
+
open_timeout: @timeout, read_timeout: @timeout,
|
|
78
|
+
write_timeout: @timeout) { |http| http.request(req) }
|
|
79
|
+
handle(response)
|
|
80
|
+
rescue Error
|
|
81
|
+
raise
|
|
82
|
+
rescue StandardError => e
|
|
83
|
+
raise Error, redact("#{e.class}: #{e.message}")
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def build_request(verb, uri, json)
|
|
87
|
+
klass = { get: Net::HTTP::Get, post: Net::HTTP::Post, patch: Net::HTTP::Patch }.fetch(verb)
|
|
88
|
+
req = klass.new(uri)
|
|
89
|
+
req["Authorization"] = "Bearer #{@token}"
|
|
90
|
+
req["Accept"] = "application/json"
|
|
91
|
+
req["User-Agent"] = "raider_reporter/#{VERSION}"
|
|
92
|
+
if json
|
|
93
|
+
req["Content-Type"] = "application/json"
|
|
94
|
+
req.body = JSON.generate(json)
|
|
95
|
+
end
|
|
96
|
+
req
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def handle(response)
|
|
100
|
+
code = response.code.to_i
|
|
101
|
+
raise Forbidden, "HTTP #{code}" if [401, 403, 404].include?(code)
|
|
102
|
+
raise Error, "HTTP #{code}: #{redact(response.body.to_s)[0, 200]}" unless (200..299).cover?(code)
|
|
103
|
+
|
|
104
|
+
JSON.parse(response.body.to_s)
|
|
105
|
+
rescue JSON::ParserError
|
|
106
|
+
raise Error, "Azure DevOps returned invalid JSON"
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
end
|
|
110
|
+
end
|
|
@@ -101,7 +101,7 @@ module RaiderReporter
|
|
|
101
101
|
def handle(response)
|
|
102
102
|
code = response.code.to_i
|
|
103
103
|
raise Forbidden, "HTTP #{code}" if [403, 404].include?(code)
|
|
104
|
-
raise Error,
|
|
104
|
+
raise Error, "HTTP #{code}: #{redact(response.body.to_s)[0, 200]}" unless (200..299).cover?(code)
|
|
105
105
|
|
|
106
106
|
JSON.parse(response.body.to_s)
|
|
107
107
|
rescue JSON::ParserError
|
|
@@ -1,12 +1,14 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "raider_reporter/azure_env"
|
|
4
|
+
|
|
3
5
|
module RaiderReporter
|
|
4
6
|
class Configuration
|
|
5
7
|
attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :environment, :max_attachment_bytes
|
|
6
8
|
|
|
7
9
|
def initialize(env = ENV)
|
|
8
|
-
@token = env
|
|
9
|
-
@endpoint = env
|
|
10
|
+
@token = AzureEnv.value(env, "RAIDER_TOKEN")
|
|
11
|
+
@endpoint = AzureEnv.value(env, "RAIDER_ENDPOINT")
|
|
10
12
|
@output_dir = "raider-results"
|
|
11
13
|
@clean_output_dir = true
|
|
12
14
|
@environment = {}
|
|
@@ -0,0 +1,258 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "pathname"
|
|
4
|
+
require "cucumber"
|
|
5
|
+
require "cucumber/formatter/ast_lookup"
|
|
6
|
+
require "raider_reporter"
|
|
7
|
+
require "raider_reporter/text"
|
|
8
|
+
|
|
9
|
+
module RaiderReporter
|
|
10
|
+
module Cucumber
|
|
11
|
+
# Translates Cucumber events into Collector calls. Every handler is guarded:
|
|
12
|
+
# a reporting bug must never fail or crash the user's run. The formatter
|
|
13
|
+
# writes nothing to its IO; the report goes to RaiderReporter's own writers.
|
|
14
|
+
#
|
|
15
|
+
# Inside this namespace `Cucumber` means RaiderReporter::Cucumber, so the
|
|
16
|
+
# gem's constants are always written ::Cucumber.
|
|
17
|
+
class Formatter
|
|
18
|
+
STATUSES = {
|
|
19
|
+
passed: "passed", failed: "failed", ambiguous: "failed",
|
|
20
|
+
pending: "pending", undefined: "pending", skipped: "skipped"
|
|
21
|
+
}.freeze
|
|
22
|
+
|
|
23
|
+
# Passed to the World's `attach`; reports an existing file as an attachment.
|
|
24
|
+
module AttachCapture
|
|
25
|
+
# Takes whatever Cucumber passes (file, media_type, filename, and any
|
|
26
|
+
# argument a later release adds) and forwards it untouched.
|
|
27
|
+
ruby2_keywords def attach(*args, &block)
|
|
28
|
+
result = super
|
|
29
|
+
Formatter.record_attachment(args[0], args[1])
|
|
30
|
+
result
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
@claimed = []
|
|
35
|
+
|
|
36
|
+
class << self
|
|
37
|
+
# One listener per run, whether it arrives as a --format or as a plugin.
|
|
38
|
+
# Cucumber copies the configuration for --out, but the event bus is shared.
|
|
39
|
+
def claim(config)
|
|
40
|
+
bus = config.event_bus
|
|
41
|
+
return false if @claimed.any? { |claimed| claimed.equal?(bus) }
|
|
42
|
+
|
|
43
|
+
@claimed << bus
|
|
44
|
+
true
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# Inline text, IO objects and missing paths are ignored: only a file that
|
|
48
|
+
# exists can be copied. Never raises into the user's step.
|
|
49
|
+
def record_attachment(file, media_type)
|
|
50
|
+
# `log` is attach with this media type; a logged message is not a file.
|
|
51
|
+
return if media_type.to_s.start_with?("text/x.cucumber.log")
|
|
52
|
+
|
|
53
|
+
file = file.to_path if file.respond_to?(:to_path)
|
|
54
|
+
return unless file.is_a?(String) && RaiderReporter.collector&.in_test? && File.file?(file)
|
|
55
|
+
|
|
56
|
+
RaiderReporter.attach(File.basename(file), file, type: media_type)
|
|
57
|
+
rescue StandardError => e
|
|
58
|
+
RaiderReporter.warn("#{e.class}: #{e.message}")
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# Called when the run ends, so the list does not outlive the run.
|
|
62
|
+
def release(config)
|
|
63
|
+
bus = config.event_bus
|
|
64
|
+
@claimed.reject! { |claimed| claimed.equal?(bus) }
|
|
65
|
+
RaiderReporter.dry_run = false
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Cucumber hands steps a World, not the formatter, so `attach` is observed
|
|
69
|
+
# by prepending to ProtoWorld. Installed once, at the first formatter.
|
|
70
|
+
# Event-bus alternatives were checked in cucumber 9.2.1, 10.2.0 and 11.1.1:
|
|
71
|
+
# there is no :attach_called event before 11, and even there (and in the
|
|
72
|
+
# :envelope attachment message) a file path has already been read into
|
|
73
|
+
# bytes with no name, so the basename the report needs is only available
|
|
74
|
+
# in ProtoWorld#attach, which exists unchanged in all three versions.
|
|
75
|
+
def capture_attach
|
|
76
|
+
return if @capturing
|
|
77
|
+
|
|
78
|
+
::Cucumber::Glue::ProtoWorld.prepend(AttachCapture)
|
|
79
|
+
@capturing = true
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def initialize(config)
|
|
84
|
+
@buffer = {}
|
|
85
|
+
@attempts = Hash.new(0)
|
|
86
|
+
@attempt = nil
|
|
87
|
+
guard { setup(config) if self.class.claim(config) }
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
private
|
|
91
|
+
|
|
92
|
+
def setup(config)
|
|
93
|
+
@config = config
|
|
94
|
+
@ast = ::Cucumber::Formatter::AstLookup.new(config)
|
|
95
|
+
@collector = RaiderReporter.start_collector!
|
|
96
|
+
RaiderReporter.dry_run = config.dry_run?
|
|
97
|
+
config.on_event(:test_run_started) { |_event| guard { @collector.start_run } }
|
|
98
|
+
config.on_event(:test_case_started) { |event| guard { case_started(event.test_case) } }
|
|
99
|
+
config.on_event(:test_step_finished) { |event| guard { step_finished(event.test_step, event.result) } }
|
|
100
|
+
config.on_event(:test_case_finished) { |event| guard { case_finished(event.test_case, event.result) } }
|
|
101
|
+
config.on_event(:test_run_finished) { |_event| guard { run_finished } }
|
|
102
|
+
# Last, and on its own: a broken attach hook may only lose attachments.
|
|
103
|
+
safely(nil) { self.class.capture_attach }
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
# Every attempt starts with an empty attachment list.
|
|
107
|
+
def case_started(test_case)
|
|
108
|
+
@collector.start_test
|
|
109
|
+
@attempt = { test_case: test_case }
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
# The first non-passed step explains why the scenario did not pass.
|
|
113
|
+
def step_finished(test_step, result)
|
|
114
|
+
return unless @attempt
|
|
115
|
+
return if @attempt[:problem] || result.passed?
|
|
116
|
+
|
|
117
|
+
@attempt[:problem] = { step: test_step, result: result }
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
def case_finished(test_case, result)
|
|
121
|
+
problem = @attempt && @attempt[:problem]
|
|
122
|
+
@attempt = nil
|
|
123
|
+
status = STATUSES.fetch(result.to_sym, "pending")
|
|
124
|
+
# Always buffer: a lookup problem may lose detail, never the scenario.
|
|
125
|
+
failure = status == "failed" ? safely(nil) { failure_for(problem, result, test_case) } : nil
|
|
126
|
+
fields = identity_for(test_case)
|
|
127
|
+
key = "#{test_case.location}|#{test_case.name}"
|
|
128
|
+
@attempts[key] += 1
|
|
129
|
+
# The newest attempt replaces the older one, taking its attachments with it.
|
|
130
|
+
@buffer[key] = @collector.build_test(
|
|
131
|
+
**fields, status: status, duration_ms: duration_ms(result), attempts: @attempts[key], failure: failure
|
|
132
|
+
)
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
def run_finished
|
|
136
|
+
@buffer.each_value { |result| @collector.add_result(result) }
|
|
137
|
+
@buffer.clear
|
|
138
|
+
RaiderReporter.finish!($stderr)
|
|
139
|
+
ensure
|
|
140
|
+
self.class.release(@config)
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
def identity_for(test_case)
|
|
144
|
+
file_path = relative_path(test_case.location.file)
|
|
145
|
+
line_number = test_case.location.lines.max
|
|
146
|
+
{
|
|
147
|
+
file_path: file_path,
|
|
148
|
+
line_number: line_number,
|
|
149
|
+
full_name: Text.utf8(full_name(test_case)),
|
|
150
|
+
identifier: "#{file_path}:#{line_number}",
|
|
151
|
+
tags: test_case.tags.map { |tag| tag.name.to_s.delete_prefix("@") }.uniq.sort
|
|
152
|
+
}
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
def full_name(test_case)
|
|
156
|
+
safely(test_case.name) do
|
|
157
|
+
feature = @ast.gherkin_document(test_case.location.file)&.feature
|
|
158
|
+
name = scenario_name(@ast.scenario_source(test_case))
|
|
159
|
+
next test_case.name unless name
|
|
160
|
+
|
|
161
|
+
feature ? "#{feature.name}: #{name}" : name
|
|
162
|
+
end
|
|
163
|
+
end
|
|
164
|
+
|
|
165
|
+
# nil when the scenario cannot be located in the AST.
|
|
166
|
+
def scenario_name(source)
|
|
167
|
+
case source&.type
|
|
168
|
+
when :ScenarioOutline
|
|
169
|
+
examples = source.examples
|
|
170
|
+
row = examples.table_body.index { |candidate| candidate.location.line == source.row.location.line }
|
|
171
|
+
row && "#{source.scenario_outline.name} (#{examples_label(source)}, row #{row + 1})"
|
|
172
|
+
when :Scenario
|
|
173
|
+
source.scenario.name
|
|
174
|
+
end
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
# "Examples" or the table's name. When several tables of one outline share a
|
|
178
|
+
# label (two unnamed ones, say), each gets its 1-based position among them,
|
|
179
|
+
# so rows of different tables never collide on full_name.
|
|
180
|
+
def examples_label(source)
|
|
181
|
+
tables = source.scenario_outline.examples
|
|
182
|
+
label = examples_name(source.examples)
|
|
183
|
+
same = tables.select { |table| examples_name(table) == label }
|
|
184
|
+
return label if same.size < 2
|
|
185
|
+
|
|
186
|
+
"#{label} #{same.index { |table| table.equal?(source.examples) || table.location.line == source.examples.location.line } + 1}"
|
|
187
|
+
end
|
|
188
|
+
|
|
189
|
+
def examples_name(table)
|
|
190
|
+
table.name.to_s.strip.empty? ? "Examples" : table.name.strip
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
# step_info is { step:, result: } for the first non-passed step, if any.
|
|
194
|
+
def failure_for(step_info, result, test_case)
|
|
195
|
+
exception = result.respond_to?(:exception) ? result.exception : nil
|
|
196
|
+
exception ||= step_info && step_info[:result].respond_to?(:exception) ? step_info[:result].exception : nil
|
|
197
|
+
return nil unless exception
|
|
198
|
+
|
|
199
|
+
Failure.new(
|
|
200
|
+
message: Text.utf8("#{step_label(step_info, test_case)}\n\n#{exception.message}")[0, Text::MAX_MESSAGE],
|
|
201
|
+
exception_class: Text.utf8(exception.class.name),
|
|
202
|
+
backtrace: (exception.backtrace || []).first(Text::MAX_BACKTRACE).map { |line| Text.utf8(line) }
|
|
203
|
+
)
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
def step_label(step_info, test_case)
|
|
207
|
+
return "Step" unless step_info
|
|
208
|
+
|
|
209
|
+
test_step = step_info[:step]
|
|
210
|
+
location = "(#{relative_path(test_step.location.file)}:#{test_step.location.lines.min})"
|
|
211
|
+
if test_step.hook?
|
|
212
|
+
"#{hook_kind(test_step, test_case)} hook #{location}"
|
|
213
|
+
else
|
|
214
|
+
"#{step_keyword(test_step)}#{test_step.text} #{location}"
|
|
215
|
+
end
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
def step_keyword(test_step)
|
|
219
|
+
safely("") { @ast.step_source(test_step).step.keyword.to_s }
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
# A hook with a real step before it runs after the scenario started.
|
|
223
|
+
def hook_kind(test_step, test_case)
|
|
224
|
+
steps = test_case.test_steps
|
|
225
|
+
index = steps.index { |candidate| candidate.equal?(test_step) }
|
|
226
|
+
index && steps.first(index).any? { |candidate| !candidate.hook? } ? "After" : "Before"
|
|
227
|
+
end
|
|
228
|
+
|
|
229
|
+
def duration_ms(result)
|
|
230
|
+
(result.duration.nanoseconds / 1_000_000.0).round
|
|
231
|
+
rescue StandardError
|
|
232
|
+
0
|
|
233
|
+
end
|
|
234
|
+
|
|
235
|
+
def relative_path(path)
|
|
236
|
+
absolute = Pathname.new(File.expand_path(path.to_s, Dir.pwd))
|
|
237
|
+
relative = absolute.relative_path_from(Pathname.new(Dir.pwd)).to_s
|
|
238
|
+
relative.start_with?("..") ? path.to_s.delete_prefix("./") : relative
|
|
239
|
+
rescue ArgumentError
|
|
240
|
+
# Windows: a path on another drive has no relative form.
|
|
241
|
+
path.to_s.delete_prefix("./")
|
|
242
|
+
end
|
|
243
|
+
|
|
244
|
+
def safely(fallback)
|
|
245
|
+
yield
|
|
246
|
+
rescue StandardError => e
|
|
247
|
+
RaiderReporter.warn("#{e.class}: #{e.message}")
|
|
248
|
+
fallback
|
|
249
|
+
end
|
|
250
|
+
|
|
251
|
+
def guard
|
|
252
|
+
yield
|
|
253
|
+
rescue StandardError => e
|
|
254
|
+
RaiderReporter.warn("#{e.class}: #{e.message}")
|
|
255
|
+
end
|
|
256
|
+
end
|
|
257
|
+
end
|
|
258
|
+
end
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
begin
|
|
4
|
+
require "cucumber"
|
|
5
|
+
rescue LoadError => e
|
|
6
|
+
raise unless e.path == "cucumber"
|
|
7
|
+
|
|
8
|
+
raise LoadError, "raider_reporter/cucumber needs the cucumber gem (>= 9, < 12) in your Gemfile"
|
|
9
|
+
end
|
|
10
|
+
require "raider_reporter"
|
|
11
|
+
require_relative "cucumber/formatter"
|
|
12
|
+
|
|
13
|
+
module RaiderReporter
|
|
14
|
+
module Cucumber
|
|
15
|
+
# Plugin mode: requiring this file from features/support registers the
|
|
16
|
+
# listener through Cucumber's InstallPlugin hook, so no --format is needed
|
|
17
|
+
# (and none of the "one formatter per stream" juggling). Outside a Cucumber
|
|
18
|
+
# run there is no step-definition registry and this is a no-op.
|
|
19
|
+
def self.install_plugin
|
|
20
|
+
# Cucumber's public InstallPlugin DSL method, called on a throwaway object
|
|
21
|
+
# (it is only an instance method of Dsl; main is extended with it).
|
|
22
|
+
Object.new.extend(::Cucumber::Glue::Dsl).InstallPlugin(name: "raider_reporter") do |config, *|
|
|
23
|
+
RaiderReporter::Cucumber::Formatter.new(config)
|
|
24
|
+
end
|
|
25
|
+
true
|
|
26
|
+
rescue StandardError => e
|
|
27
|
+
# Outside a run there is no registry (Dsl has no rb_language) and nothing to
|
|
28
|
+
# report. Inside one, losing the listener must not be silent.
|
|
29
|
+
RaiderReporter.warn("could not register the Cucumber plugin: #{e.class}: #{e.message}") if run_active?
|
|
30
|
+
false
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def self.run_active?
|
|
34
|
+
!::Cucumber::Glue::Dsl.instance_variable_get(:@rb_language).nil?
|
|
35
|
+
end
|
|
36
|
+
private_class_method :run_active?
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
RaiderReporter::Cucumber.install_plugin
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "rspec/core"
|
|
4
4
|
require "raider_reporter"
|
|
5
|
+
require "raider_reporter/text"
|
|
5
6
|
|
|
6
7
|
module RaiderReporter
|
|
7
8
|
module RSpec
|
|
@@ -13,8 +14,8 @@ module RaiderReporter
|
|
|
13
14
|
:example_failed, :example_pending, :close
|
|
14
15
|
|
|
15
16
|
NON_TAG_KEYS = %i[pending skip].freeze
|
|
16
|
-
MAX_MESSAGE =
|
|
17
|
-
MAX_BACKTRACE =
|
|
17
|
+
MAX_MESSAGE = Text::MAX_MESSAGE
|
|
18
|
+
MAX_BACKTRACE = Text::MAX_BACKTRACE
|
|
18
19
|
|
|
19
20
|
def initialize(output)
|
|
20
21
|
@output = output
|
|
@@ -77,14 +78,8 @@ module RaiderReporter
|
|
|
77
78
|
)
|
|
78
79
|
end
|
|
79
80
|
|
|
80
|
-
# JSON generation raises on invalid UTF-8, which would lose the whole run.
|
|
81
81
|
def utf8(value)
|
|
82
|
-
|
|
83
|
-
if [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].include?(str.encoding)
|
|
84
|
-
str.dup.force_encoding(Encoding::UTF_8).scrub("?")
|
|
85
|
-
else
|
|
86
|
-
str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "?")
|
|
87
|
-
end
|
|
82
|
+
Text.utf8(value)
|
|
88
83
|
end
|
|
89
84
|
|
|
90
85
|
def guard
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "raider_reporter/azure_env"
|
|
4
|
+
|
|
3
5
|
module RaiderReporter
|
|
4
6
|
# Identifies which slice of a parallel suite this process is.
|
|
5
7
|
class ShardDetector
|
|
@@ -9,7 +11,7 @@ module RaiderReporter
|
|
|
9
11
|
end
|
|
10
12
|
|
|
11
13
|
def shard
|
|
12
|
-
index, total, suffix = composed ||
|
|
14
|
+
index, total, suffix = composed || outer || parallel_tests
|
|
13
15
|
return nil unless index
|
|
14
16
|
|
|
15
17
|
{ "index" => index, "total" => total, "label" => "#{@job || "local"}##{suffix || index}" }
|
|
@@ -19,7 +21,7 @@ module RaiderReporter
|
|
|
19
21
|
|
|
20
22
|
# A matrix leg that also runs parallel_tests: every process gets a unique index.
|
|
21
23
|
def composed
|
|
22
|
-
explicit_index, explicit_total =
|
|
24
|
+
explicit_index, explicit_total = outer
|
|
23
25
|
parallel_index, = parallel_tests
|
|
24
26
|
groups = positive(@env["PARALLEL_TEST_GROUPS"])
|
|
25
27
|
return nil unless explicit_index && parallel_index && groups
|
|
@@ -29,6 +31,18 @@ module RaiderReporter
|
|
|
29
31
|
"#{explicit_index}.#{parallel_index}"]
|
|
30
32
|
end
|
|
31
33
|
|
|
34
|
+
def outer
|
|
35
|
+
explicit || azure_parallel
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def azure_parallel
|
|
39
|
+
return nil unless AzureEnv.azure?(@env)
|
|
40
|
+
|
|
41
|
+
total = positive(AzureEnv.value(@env, "SYSTEM_TOTALJOBSINPHASE"))
|
|
42
|
+
index = positive(AzureEnv.value(@env, "SYSTEM_JOBPOSITIONINPHASE"))
|
|
43
|
+
[index, total] if index && total && total > 1
|
|
44
|
+
end
|
|
45
|
+
|
|
32
46
|
def explicit
|
|
33
47
|
index = positive(@env["RAIDER_SHARD_INDEX"])
|
|
34
48
|
[index, positive(@env["RAIDER_SHARD_TOTAL"])] if index
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RaiderReporter
|
|
4
|
+
# Text limits and cleaning shared by the framework formatters.
|
|
5
|
+
module Text
|
|
6
|
+
MAX_MESSAGE = 10_000
|
|
7
|
+
MAX_BACKTRACE = 50
|
|
8
|
+
|
|
9
|
+
# JSON generation raises on invalid UTF-8, which would lose the whole run.
|
|
10
|
+
def self.utf8(value)
|
|
11
|
+
str = value.to_s
|
|
12
|
+
if [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].include?(str.encoding)
|
|
13
|
+
str.dup.force_encoding(Encoding::UTF_8).scrub("?")
|
|
14
|
+
else
|
|
15
|
+
str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "?")
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
data/lib/raider_reporter.rb
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
require "time"
|
|
4
4
|
require "rbconfig"
|
|
5
5
|
require_relative "raider_reporter/version"
|
|
6
|
+
require_relative "raider_reporter/text"
|
|
6
7
|
require_relative "raider_reporter/models"
|
|
7
8
|
require_relative "raider_reporter/payload"
|
|
8
9
|
require_relative "raider_reporter/configuration"
|
|
@@ -14,6 +15,7 @@ require_relative "raider_reporter/shard_detector"
|
|
|
14
15
|
require_relative "raider_reporter/writers/uploader"
|
|
15
16
|
require_relative "raider_reporter/comment/summary_client"
|
|
16
17
|
require_relative "raider_reporter/comment/github_client"
|
|
18
|
+
require_relative "raider_reporter/comment/azure_client"
|
|
17
19
|
|
|
18
20
|
module RaiderReporter
|
|
19
21
|
class << self
|
|
@@ -27,13 +29,18 @@ module RaiderReporter
|
|
|
27
29
|
yield configuration
|
|
28
30
|
end
|
|
29
31
|
|
|
32
|
+
# Set by the Cucumber formatter when the run is a --dry-run.
|
|
33
|
+
attr_writer :dry_run
|
|
34
|
+
|
|
30
35
|
def reset!
|
|
36
|
+
@dry_run = false
|
|
31
37
|
@collector&.cleanup
|
|
32
38
|
@configuration = nil
|
|
33
39
|
@collector = nil
|
|
34
40
|
end
|
|
35
41
|
|
|
36
42
|
def start_collector!
|
|
43
|
+
@dry_run = false
|
|
37
44
|
@collector = Collector.new(
|
|
38
45
|
max_attachment_bytes: configuration.max_attachment_bytes,
|
|
39
46
|
warn: ->(message) { warn(message) }
|
|
@@ -104,8 +111,10 @@ module RaiderReporter
|
|
|
104
111
|
"Raider report uploaded"
|
|
105
112
|
end
|
|
106
113
|
|
|
107
|
-
# `rspec --dry-run`
|
|
114
|
+
# `rspec --dry-run` and `cucumber --dry-run` report every example as passed without running it.
|
|
108
115
|
def dry_run?
|
|
116
|
+
return true if @dry_run
|
|
117
|
+
|
|
109
118
|
defined?(::RSpec.configuration) && ::RSpec.configuration.respond_to?(:dry_run?) &&
|
|
110
119
|
::RSpec.configuration.dry_run?
|
|
111
120
|
end
|
metadata
CHANGED
|
@@ -1,16 +1,17 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: raider_reporter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Augustin Gottlieb
|
|
8
8
|
bindir: exe
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
11
|
dependencies: []
|
|
12
|
-
description: RSpec
|
|
13
|
-
schema,
|
|
12
|
+
description: RSpec and Cucumber formatters that record results, screenshots and videos
|
|
13
|
+
in the Raider schema, write them locally, and optionally upload them to a Raider
|
|
14
|
+
server.
|
|
14
15
|
email:
|
|
15
16
|
- 33221555+aguspe@users.noreply.github.com
|
|
16
17
|
executables:
|
|
@@ -23,19 +24,24 @@ files:
|
|
|
23
24
|
- README.md
|
|
24
25
|
- exe/raider-reporter
|
|
25
26
|
- lib/raider_reporter.rb
|
|
27
|
+
- lib/raider_reporter/azure_env.rb
|
|
26
28
|
- lib/raider_reporter/ci_detector.rb
|
|
27
29
|
- lib/raider_reporter/cli.rb
|
|
28
30
|
- lib/raider_reporter/collector.rb
|
|
31
|
+
- lib/raider_reporter/comment/azure_client.rb
|
|
29
32
|
- lib/raider_reporter/comment/github_client.rb
|
|
30
33
|
- lib/raider_reporter/comment/summary_client.rb
|
|
31
34
|
- lib/raider_reporter/configuration.rb
|
|
32
35
|
- lib/raider_reporter/content_types.rb
|
|
36
|
+
- lib/raider_reporter/cucumber.rb
|
|
37
|
+
- lib/raider_reporter/cucumber/formatter.rb
|
|
33
38
|
- lib/raider_reporter/github_event.rb
|
|
34
39
|
- lib/raider_reporter/models.rb
|
|
35
40
|
- lib/raider_reporter/payload.rb
|
|
36
41
|
- lib/raider_reporter/rspec.rb
|
|
37
42
|
- lib/raider_reporter/rspec/formatter.rb
|
|
38
43
|
- lib/raider_reporter/shard_detector.rb
|
|
44
|
+
- lib/raider_reporter/text.rb
|
|
39
45
|
- lib/raider_reporter/version.rb
|
|
40
46
|
- lib/raider_reporter/writers/local.rb
|
|
41
47
|
- lib/raider_reporter/writers/uploader.rb
|
|
@@ -60,7 +66,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
60
66
|
- !ruby/object:Gem::Version
|
|
61
67
|
version: '0'
|
|
62
68
|
requirements: []
|
|
63
|
-
rubygems_version:
|
|
69
|
+
rubygems_version: 4.0.6
|
|
64
70
|
specification_version: 4
|
|
65
|
-
summary:
|
|
71
|
+
summary: RSpec and Cucumber formatters that write Raider reports locally and upload
|
|
72
|
+
them to Raider
|
|
66
73
|
test_files: []
|