raider_reporter 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +8 -0
- data/LICENSE +21 -0
- data/README.md +110 -0
- data/lib/raider_reporter/ci_detector.rb +51 -0
- data/lib/raider_reporter/collector.rb +87 -0
- data/lib/raider_reporter/configuration.rb +26 -0
- data/lib/raider_reporter/content_types.rb +33 -0
- data/lib/raider_reporter/models.rb +14 -0
- data/lib/raider_reporter/payload.rb +81 -0
- data/lib/raider_reporter/rspec/formatter.rb +97 -0
- data/lib/raider_reporter/rspec.rb +4 -0
- data/lib/raider_reporter/version.rb +5 -0
- data/lib/raider_reporter/writers/local.rb +34 -0
- data/lib/raider_reporter/writers/uploader.rb +114 -0
- data/lib/raider_reporter.rb +112 -0
- metadata +59 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: b1e069eeb6569206c085d042b6b3e26f0e35bbb2459b60dccfecf6d744aaef86
|
|
4
|
+
data.tar.gz: f117594302d75a2d97edfba870beb04e69d477acbb6f80ab276689e55d3c531a
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: ad5162cad62d6efed699f663f977540ef84f1dc640845d112c8f594bac34767aec0224a341e55e4ebb354b5a59b57bc13c5b4deacd071b8ade6bbade90b85366
|
|
7
|
+
data.tar.gz: 6b68c2e3ab889e30a2ff1e9c6cb075f7e3e8556d92557e60f36360daf232e9f9ed72d8041592f8766c8496bc5d303700222e2b96f1df685b8813ece4627c1375
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 — 2026-10-11
|
|
4
|
+
|
|
5
|
+
- RSpec formatter writing Raider schema v1 to `raider-results/run.json`
|
|
6
|
+
- Screenshot and video attachments via `RaiderReporter.attach`, copied when attached (5 MB limit per file)
|
|
7
|
+
- Optional fail-open upload to a Raider server (`RAIDER_TOKEN`, `RAIDER_ENDPOINT`), capped at 45 MB / 1000 attachment files per upload
|
|
8
|
+
- GitHub Actions CI and git metadata detection
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Augustin Gottlieb
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# raider_reporter
|
|
2
|
+
|
|
3
|
+
Raider-schema test reporting for RSpec. `raider_reporter` is an RSpec formatter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
|
|
4
|
+
|
|
5
|
+
It has no runtime dependencies and requires Ruby 3.2 or newer.
|
|
6
|
+
|
|
7
|
+
## Install
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
# Gemfile
|
|
11
|
+
gem "raider_reporter"
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## Setup
|
|
15
|
+
|
|
16
|
+
Require the RSpec integration in `spec/spec_helper.rb`:
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
require "raider_reporter/rspec"
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Enable the formatter in `.rspec`:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
--format progress
|
|
26
|
+
--format RaiderReporter::RSpec::Formatter
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Run `bundle exec rspec` as usual. Results land in `raider-results/run.json`. A dry run (`rspec --dry-run`) writes and uploads nothing.
|
|
30
|
+
|
|
31
|
+
## Attaching screenshots and videos
|
|
32
|
+
|
|
33
|
+
Call `RaiderReporter.attach` while an example is running, for example from an `after` hook:
|
|
34
|
+
|
|
35
|
+
```ruby
|
|
36
|
+
RSpec.configure do |config|
|
|
37
|
+
config.after do |example|
|
|
38
|
+
next unless example.exception
|
|
39
|
+
|
|
40
|
+
FileUtils.mkdir_p("tmp/screenshots")
|
|
41
|
+
path = "tmp/screenshots/#{example.id.gsub(/\W/, '_')}.png"
|
|
42
|
+
driver.save_screenshot(path)
|
|
43
|
+
RaiderReporter.attach("screenshot", path, type: :png)
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
`attach` copies the file right away, so the attachment keeps the bytes the file had at that moment even if the same path is overwritten by a later example. Using a path per example (as above) still makes the files on disk easier to find.
|
|
49
|
+
|
|
50
|
+
`attach` returns `true` on success. It returns `false` and prints a `[raider_reporter]` warning, never raising, if it is called outside a running example (for instance when the formatter is not loaded), if the file does not exist, or if the file is larger than 5 MB.
|
|
51
|
+
|
|
52
|
+
## Configuration
|
|
53
|
+
|
|
54
|
+
```ruby
|
|
55
|
+
RaiderReporter.configure do |c|
|
|
56
|
+
c.output_dir = "raider-results"
|
|
57
|
+
c.environment = { "browser" => "chrome", "suite" => "smoke", "headless" => true }
|
|
58
|
+
end
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
| Option | Default | Description |
|
|
62
|
+
| --- | --- | --- |
|
|
63
|
+
| `token` | `ENV["RAIDER_TOKEN"]` | API token used for uploads. |
|
|
64
|
+
| `endpoint` | `ENV["RAIDER_ENDPOINT"]` | Base URL of the Raider server to upload to. |
|
|
65
|
+
| `output_dir` | `"raider-results"` | Directory for local output. |
|
|
66
|
+
| `clean_output_dir` | `true` | Remove the previous `run.json` and `attachments/` directory inside `output_dir` before writing. Nothing else in `output_dir` is touched. |
|
|
67
|
+
| `environment` | `{}` | Free-form key/values stored with the run (browser, suite name, and so on). Values are JSON values: strings, booleans or numbers. |
|
|
68
|
+
|
|
69
|
+
## Local output
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
raider-results/
|
|
73
|
+
run.json # the run in schema v1
|
|
74
|
+
attachments/ # copies of attached screenshots and videos
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Add `raider-results/` to your `.gitignore`.
|
|
78
|
+
|
|
79
|
+
## Uploading to Raider
|
|
80
|
+
|
|
81
|
+
If both a token and an endpoint are set, the run (plus attachments) is uploaded when the suite finishes:
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
RAIDER_TOKEN=... RAIDER_ENDPOINT=https://your-raider-server bundle exec rspec
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
On success the formatter prints `Raider report: <url>` (or `Raider report uploaded (run <id>)` if the server's response has no URL). If either value is missing, nothing is uploaded and the local files are still written.
|
|
88
|
+
|
|
89
|
+
An upload carries at most 45 MB of attachments and at most 1000 attachment files. Attachments past that budget are left out of the upload (in order, with one warning) but stay in the local `raider-results/`.
|
|
90
|
+
|
|
91
|
+
Reporting is fail-open: no error inside the gem, network failure, timeout or non-2xx response will raise or change RSpec's exit status. Each connect/write/read step of the upload times out after 30 seconds. Problems are printed as warnings prefixed with `[raider_reporter]`. An endpoint without `http://` or `https://` (for example `localhost:3000`) produces a warning and nothing is uploaded; a trailing slash is fine.
|
|
92
|
+
|
|
93
|
+
The hosted Raider service is not public yet; uploads go to a Raider server whose address you provide in `RAIDER_ENDPOINT`.
|
|
94
|
+
|
|
95
|
+
## Examples
|
|
96
|
+
|
|
97
|
+
- `examples/rspec_basic` is a runnable RSpec project with a passing, a failing (with an attachment) and a skipped example. Run it with `cd examples/rspec_basic && bundle install && bundle exec rspec`; the failure is deliberate.
|
|
98
|
+
- `examples/github_actions.yml` shows the workflow steps for CI.
|
|
99
|
+
|
|
100
|
+
## Schema
|
|
101
|
+
|
|
102
|
+
`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are exactly `passed`, `failed`, `pending` and `skipped`.
|
|
103
|
+
|
|
104
|
+
## Contributing
|
|
105
|
+
|
|
106
|
+
Bug reports and pull requests are welcome at https://github.com/RaiderHQ/raider_reporter. Run `bundle install && bundle exec rake` to run the specs.
|
|
107
|
+
|
|
108
|
+
## License
|
|
109
|
+
|
|
110
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "open3"
|
|
4
|
+
|
|
5
|
+
module RaiderReporter
|
|
6
|
+
class CiDetector
|
|
7
|
+
def initialize(env: ENV, git: method(:git_command))
|
|
8
|
+
@env = env
|
|
9
|
+
@git = git
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
def ci
|
|
13
|
+
return nil unless @env["GITHUB_ACTIONS"] == "true"
|
|
14
|
+
|
|
15
|
+
run_id = @env["GITHUB_RUN_ID"]
|
|
16
|
+
build_url = if @env["GITHUB_SERVER_URL"] && @env["GITHUB_REPOSITORY"] && run_id
|
|
17
|
+
"#{@env['GITHUB_SERVER_URL']}/#{@env['GITHUB_REPOSITORY']}/actions/runs/#{run_id}"
|
|
18
|
+
end
|
|
19
|
+
{ "provider" => "github_actions", "build_url" => build_url, "build_id" => run_id }.compact
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def git
|
|
23
|
+
info = { "branch" => branch, "commit_sha" => sha, "commit_message" => @git.call("log", "-1", "--pretty=%s") }
|
|
24
|
+
info = info.compact
|
|
25
|
+
info.empty? ? nil : info
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
def branch
|
|
31
|
+
head_ref = @env["GITHUB_HEAD_REF"].to_s
|
|
32
|
+
return head_ref unless head_ref.empty?
|
|
33
|
+
|
|
34
|
+
return @env["GITHUB_REF_NAME"] if @env["GITHUB_REF_NAME"]
|
|
35
|
+
|
|
36
|
+
local = @git.call("rev-parse", "--abbrev-ref", "HEAD")
|
|
37
|
+
local == "HEAD" ? nil : local # "HEAD" means detached, not a branch name
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def sha
|
|
41
|
+
@env["GITHUB_SHA"] || @git.call("rev-parse", "HEAD")
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def git_command(*args)
|
|
45
|
+
out, _err, status = Open3.capture3("git", *args)
|
|
46
|
+
status.success? ? out.strip : nil
|
|
47
|
+
rescue StandardError
|
|
48
|
+
nil
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "securerandom"
|
|
4
|
+
require "tmpdir"
|
|
5
|
+
require "fileutils"
|
|
6
|
+
|
|
7
|
+
module RaiderReporter
|
|
8
|
+
# Accumulates one suite run. Framework formatters call it; writers read the Run it returns.
|
|
9
|
+
class Collector
|
|
10
|
+
def initialize(max_attachment_bytes:, warn:, clock: -> { Time.now.utc }, uuid: -> { SecureRandom.uuid })
|
|
11
|
+
@max_attachment_bytes = max_attachment_bytes
|
|
12
|
+
@warn = warn
|
|
13
|
+
@clock = clock
|
|
14
|
+
@uuid = uuid
|
|
15
|
+
@results = []
|
|
16
|
+
@pending_attachments = nil
|
|
17
|
+
@staged = 0
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def start_run
|
|
21
|
+
@started_at = @clock.call
|
|
22
|
+
@results = []
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def start_test
|
|
26
|
+
@pending_attachments = []
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def in_test?
|
|
30
|
+
!@pending_attachments.nil?
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def attach(attachment)
|
|
34
|
+
return false unless in_test?
|
|
35
|
+
|
|
36
|
+
@pending_attachments << attachment
|
|
37
|
+
true
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Copies a file into a collector-owned staging dir so later writes to the
|
|
41
|
+
# same path (a reused screenshot name) cannot change what this example sent.
|
|
42
|
+
def stage(path)
|
|
43
|
+
@staging_dir ||= Dir.mktmpdir("raider_reporter")
|
|
44
|
+
dir = File.join(@staging_dir, @staged.to_s)
|
|
45
|
+
@staged += 1
|
|
46
|
+
FileUtils.mkdir_p(dir)
|
|
47
|
+
copy = File.join(dir, File.basename(path))
|
|
48
|
+
FileUtils.cp(path, copy)
|
|
49
|
+
copy
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def cleanup
|
|
53
|
+
FileUtils.rm_rf(@staging_dir) if @staging_dir
|
|
54
|
+
@staging_dir = nil
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def finish_test(**fields)
|
|
58
|
+
result = TestResult.new(**fields, attachments: @pending_attachments || [])
|
|
59
|
+
@pending_attachments = nil
|
|
60
|
+
@results << result
|
|
61
|
+
result
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def finish_run(ci:, git:, environment:)
|
|
65
|
+
Run.new(
|
|
66
|
+
client_run_id: @uuid.call, started_at: @started_at, finished_at: @clock.call,
|
|
67
|
+
ci: ci, git: git, environment: environment,
|
|
68
|
+
results: @results.map { |r| r.with(attachments: r.attachments.select { |a| keep?(a) }) }
|
|
69
|
+
)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
private
|
|
73
|
+
|
|
74
|
+
def keep?(attachment)
|
|
75
|
+
path = attachment.source_path
|
|
76
|
+
unless File.file?(path)
|
|
77
|
+
@warn.call("attachment #{path} no longer exists; skipped")
|
|
78
|
+
return false
|
|
79
|
+
end
|
|
80
|
+
if File.size(path) > @max_attachment_bytes
|
|
81
|
+
@warn.call("attachment #{path} is larger than #{@max_attachment_bytes} bytes; skipped")
|
|
82
|
+
return false
|
|
83
|
+
end
|
|
84
|
+
true
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
end
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RaiderReporter
|
|
4
|
+
class Configuration
|
|
5
|
+
attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :environment, :max_attachment_bytes
|
|
6
|
+
|
|
7
|
+
def initialize(env = ENV)
|
|
8
|
+
@token = env["RAIDER_TOKEN"]
|
|
9
|
+
@endpoint = env["RAIDER_ENDPOINT"]
|
|
10
|
+
@output_dir = "raider-results"
|
|
11
|
+
@clean_output_dir = true
|
|
12
|
+
@environment = {}
|
|
13
|
+
@max_attachment_bytes = 5 * 1024 * 1024
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def upload?
|
|
17
|
+
present?(token) && present?(endpoint)
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
private
|
|
21
|
+
|
|
22
|
+
def present?(value)
|
|
23
|
+
!value.nil? && !value.to_s.strip.empty?
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RaiderReporter
|
|
4
|
+
module ContentTypes
|
|
5
|
+
BY_EXTENSION = {
|
|
6
|
+
".png" => "image/png", ".jpg" => "image/jpeg", ".jpeg" => "image/jpeg", ".gif" => "image/gif",
|
|
7
|
+
".webp" => "image/webp", ".mp4" => "video/mp4", ".webm" => "video/webm", ".mov" => "video/quicktime",
|
|
8
|
+
".txt" => "text/plain", ".log" => "text/plain", ".html" => "text/html", ".json" => "application/json"
|
|
9
|
+
}.freeze
|
|
10
|
+
BY_SYMBOL = { png: "image/png", jpg: "image/jpeg", jpeg: "image/jpeg", mp4: "video/mp4",
|
|
11
|
+
webm: "video/webm", text: "text/plain", html: "text/html", json: "application/json" }.freeze
|
|
12
|
+
|
|
13
|
+
MIME = %r{\A[\w.+-]+/[\w.+-]+\z}
|
|
14
|
+
|
|
15
|
+
module_function
|
|
16
|
+
|
|
17
|
+
def resolve(type, path)
|
|
18
|
+
if type.is_a?(String)
|
|
19
|
+
return type if type.match?(MIME)
|
|
20
|
+
|
|
21
|
+
type = nil
|
|
22
|
+
end
|
|
23
|
+
return BY_SYMBOL.fetch(type) if BY_SYMBOL.key?(type)
|
|
24
|
+
|
|
25
|
+
from_ext = BY_EXTENSION[File.extname(path.to_s).downcase]
|
|
26
|
+
return from_ext if from_ext && (type.nil? || from_ext.start_with?("#{type}/"))
|
|
27
|
+
return "video/mp4" if type == :video
|
|
28
|
+
return "image/png" if type == :image
|
|
29
|
+
|
|
30
|
+
"application/octet-stream"
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RaiderReporter
|
|
4
|
+
SCHEMA_VERSION = 1
|
|
5
|
+
STATUSES = %w[passed failed pending skipped].freeze
|
|
6
|
+
|
|
7
|
+
Failure = Data.define(:message, :exception_class, :backtrace)
|
|
8
|
+
Attachment = Data.define(:name, :source_path, :content_type)
|
|
9
|
+
TestResult = Data.define(
|
|
10
|
+
:identifier, :full_name, :file_path, :line_number, :status,
|
|
11
|
+
:duration_ms, :attempts, :tags, :failure, :attachments
|
|
12
|
+
)
|
|
13
|
+
Run = Data.define(:client_run_id, :started_at, :finished_at, :ci, :git, :environment, :results)
|
|
14
|
+
end
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RaiderReporter
|
|
4
|
+
# Serializes a Run into schema v1. Local mode points attachments at copied
|
|
5
|
+
# files; upload mode points them at multipart part names.
|
|
6
|
+
class Payload
|
|
7
|
+
MODES = %i[local upload].freeze
|
|
8
|
+
|
|
9
|
+
attr_reader :run
|
|
10
|
+
|
|
11
|
+
def initialize(run)
|
|
12
|
+
@run = run
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def to_h(mode:)
|
|
16
|
+
raise ArgumentError, "unknown mode #{mode.inspect}" unless MODES.include?(mode)
|
|
17
|
+
|
|
18
|
+
entries = attachment_entries.each
|
|
19
|
+
{
|
|
20
|
+
"schema_version" => SCHEMA_VERSION,
|
|
21
|
+
"client_run_id" => @run.client_run_id,
|
|
22
|
+
"run" => run_hash,
|
|
23
|
+
"results" => @run.results.map do |result|
|
|
24
|
+
result_hash(result).merge(
|
|
25
|
+
"attachments" => result.attachments.map { attachment_hash(entries.next, mode) }
|
|
26
|
+
)
|
|
27
|
+
end
|
|
28
|
+
}
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def attachment_entries
|
|
32
|
+
@attachment_entries ||= @run.results.flat_map(&:attachments).each_with_index.map do |attachment, index|
|
|
33
|
+
{
|
|
34
|
+
attachment: attachment,
|
|
35
|
+
part: "att_#{index}",
|
|
36
|
+
path: "attachments/#{index}-#{File.basename(attachment.source_path)}"
|
|
37
|
+
}
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
private
|
|
42
|
+
|
|
43
|
+
def run_hash
|
|
44
|
+
{
|
|
45
|
+
"started_at" => @run.started_at&.utc&.iso8601,
|
|
46
|
+
"finished_at" => @run.finished_at&.utc&.iso8601,
|
|
47
|
+
"ci" => @run.ci,
|
|
48
|
+
"git" => @run.git,
|
|
49
|
+
"environment" => @run.environment
|
|
50
|
+
}.compact
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def result_hash(result)
|
|
54
|
+
hash = {
|
|
55
|
+
"identifier" => result.identifier,
|
|
56
|
+
"full_name" => result.full_name,
|
|
57
|
+
"file_path" => result.file_path,
|
|
58
|
+
"line_number" => result.line_number,
|
|
59
|
+
"status" => result.status,
|
|
60
|
+
"duration_ms" => result.duration_ms,
|
|
61
|
+
"attempts" => result.attempts,
|
|
62
|
+
"tags" => result.tags
|
|
63
|
+
}
|
|
64
|
+
hash["failure"] = failure_hash(result.failure) if result.failure
|
|
65
|
+
hash
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def failure_hash(failure)
|
|
69
|
+
{
|
|
70
|
+
"message" => failure.message,
|
|
71
|
+
"exception_class" => failure.exception_class,
|
|
72
|
+
"backtrace" => failure.backtrace
|
|
73
|
+
}
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def attachment_hash(entry, mode)
|
|
77
|
+
ref = mode == :local ? { "path" => entry[:path] } : { "part" => entry[:part] }
|
|
78
|
+
{ "name" => entry[:attachment].name, "content_type" => entry[:attachment].content_type }.merge(ref)
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
end
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rspec/core"
|
|
4
|
+
require "raider_reporter"
|
|
5
|
+
|
|
6
|
+
module RaiderReporter
|
|
7
|
+
module RSpec
|
|
8
|
+
# Translates RSpec notifications into Collector calls. Every hook is guarded:
|
|
9
|
+
# a reporting bug must never fail or crash the user's suite.
|
|
10
|
+
class Formatter
|
|
11
|
+
::RSpec::Core::Formatters.register self,
|
|
12
|
+
:start, :example_started, :example_passed,
|
|
13
|
+
:example_failed, :example_pending, :close
|
|
14
|
+
|
|
15
|
+
NON_TAG_KEYS = %i[pending skip].freeze
|
|
16
|
+
MAX_MESSAGE = 10_000
|
|
17
|
+
MAX_BACKTRACE = 50
|
|
18
|
+
|
|
19
|
+
def initialize(output)
|
|
20
|
+
@output = output
|
|
21
|
+
@collector = RaiderReporter.start_collector!
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def start(_notification)
|
|
25
|
+
guard { @collector.start_run }
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def example_started(_notification)
|
|
29
|
+
guard { @collector.start_test }
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def example_passed(notification)
|
|
33
|
+
guard { record(notification.example, "passed") }
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def example_failed(notification)
|
|
37
|
+
guard { record(notification.example, "failed") }
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def example_pending(notification)
|
|
41
|
+
guard do
|
|
42
|
+
example = notification.example
|
|
43
|
+
record(example, example.metadata[:skip] ? "skipped" : "pending")
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def close(_notification)
|
|
48
|
+
guard { RaiderReporter.finish!(@output) }
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
private
|
|
52
|
+
|
|
53
|
+
def record(example, status)
|
|
54
|
+
metadata = example.metadata
|
|
55
|
+
execution = example.execution_result
|
|
56
|
+
@collector.finish_test(
|
|
57
|
+
identifier: example.id,
|
|
58
|
+
full_name: utf8(example.full_description),
|
|
59
|
+
file_path: metadata[:file_path].to_s.delete_prefix("./"),
|
|
60
|
+
line_number: metadata[:line_number],
|
|
61
|
+
status: status,
|
|
62
|
+
duration_ms: ((execution.run_time || 0) * 1000).round,
|
|
63
|
+
attempts: metadata.fetch(:retry_attempts, 0).to_i + 1,
|
|
64
|
+
tags: metadata.select { |key, value| value == true && !NON_TAG_KEYS.include?(key) }.keys.map(&:to_s).sort,
|
|
65
|
+
failure: status == "failed" ? failure_for(execution.exception) : nil
|
|
66
|
+
)
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def failure_for(exception)
|
|
70
|
+
return nil unless exception
|
|
71
|
+
|
|
72
|
+
backtrace = ::RSpec.configuration.backtrace_formatter.format_backtrace(exception.backtrace || [])
|
|
73
|
+
Failure.new(
|
|
74
|
+
message: utf8(exception.message)[0, MAX_MESSAGE],
|
|
75
|
+
exception_class: utf8(exception.class.name),
|
|
76
|
+
backtrace: backtrace.first(MAX_BACKTRACE).map { |line| utf8(line) }
|
|
77
|
+
)
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
# JSON generation raises on invalid UTF-8, which would lose the whole run.
|
|
81
|
+
def utf8(value)
|
|
82
|
+
str = value.to_s
|
|
83
|
+
if [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].include?(str.encoding)
|
|
84
|
+
str.dup.force_encoding(Encoding::UTF_8).scrub("?")
|
|
85
|
+
else
|
|
86
|
+
str.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "?")
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def guard
|
|
91
|
+
yield
|
|
92
|
+
rescue StandardError => e
|
|
93
|
+
RaiderReporter.warn("#{e.class}: #{e.message}")
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "fileutils"
|
|
4
|
+
require "json"
|
|
5
|
+
|
|
6
|
+
module RaiderReporter
|
|
7
|
+
module Writers
|
|
8
|
+
class Local
|
|
9
|
+
def initialize(config)
|
|
10
|
+
@config = config
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def write(payload)
|
|
14
|
+
dir = File.expand_path(@config.output_dir)
|
|
15
|
+
clean(dir) if @config.clean_output_dir
|
|
16
|
+
FileUtils.mkdir_p(File.join(dir, "attachments"))
|
|
17
|
+
payload.attachment_entries.each do |entry|
|
|
18
|
+
FileUtils.cp(entry[:attachment].source_path, File.join(dir, entry[:path]))
|
|
19
|
+
end
|
|
20
|
+
run_json = File.join(dir, "run.json")
|
|
21
|
+
File.write(run_json, JSON.pretty_generate(payload.to_h(mode: :local)))
|
|
22
|
+
run_json
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
private
|
|
26
|
+
|
|
27
|
+
# Remove only what this writer creates, so output_dir "." or "spec" is safe.
|
|
28
|
+
def clean(dir)
|
|
29
|
+
FileUtils.rm_f(File.join(dir, "run.json"))
|
|
30
|
+
FileUtils.rm_rf(File.join(dir, "attachments"))
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "net/http"
|
|
4
|
+
require "uri"
|
|
5
|
+
require "json"
|
|
6
|
+
require "securerandom"
|
|
7
|
+
|
|
8
|
+
module RaiderReporter
|
|
9
|
+
module Writers
|
|
10
|
+
class Uploader
|
|
11
|
+
Result = Data.define(:ok, :url, :run_id, :error)
|
|
12
|
+
# Keeps uploads under the server's 50 MB request limit and multipart part limit.
|
|
13
|
+
MAX_UPLOAD_ATTACHMENT_BYTES = 45 * 1024 * 1024
|
|
14
|
+
MAX_UPLOAD_ATTACHMENTS = 1000
|
|
15
|
+
|
|
16
|
+
def initialize(config, timeout: 30, warn: RaiderReporter.method(:warn))
|
|
17
|
+
@config = config
|
|
18
|
+
@timeout = timeout
|
|
19
|
+
@warn = warn
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def upload(payload)
|
|
23
|
+
uri = runs_uri
|
|
24
|
+
payload = within_budget(payload)
|
|
25
|
+
request = Net::HTTP::Post.new(uri)
|
|
26
|
+
request["Authorization"] = "Bearer #{@config.token}"
|
|
27
|
+
request["User-Agent"] = "raider_reporter/#{VERSION}"
|
|
28
|
+
boundary = "----raider#{SecureRandom.hex(12)}"
|
|
29
|
+
body = "".b
|
|
30
|
+
add_field(body, boundary, "payload", JSON.generate(payload.to_h(mode: :upload)))
|
|
31
|
+
payload.attachment_entries.each do |entry|
|
|
32
|
+
add_file(body, boundary, entry[:part], entry[:attachment])
|
|
33
|
+
end
|
|
34
|
+
body << "--#{boundary}--\r\n".b
|
|
35
|
+
request.body = body
|
|
36
|
+
request["Content-Type"] = "multipart/form-data; boundary=#{boundary}"
|
|
37
|
+
|
|
38
|
+
response = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
|
|
39
|
+
open_timeout: @timeout, read_timeout: @timeout,
|
|
40
|
+
write_timeout: @timeout) do |http|
|
|
41
|
+
http.request(request)
|
|
42
|
+
end
|
|
43
|
+
to_result(response)
|
|
44
|
+
rescue StandardError => e
|
|
45
|
+
Result.new(ok: false, url: nil, run_id: nil, error: "#{e.class}: #{e.message}")
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
# Keeps the leading attachments that fit the byte and count budget and
|
|
51
|
+
# rebuilds the payload without the rest, so part names stay contiguous.
|
|
52
|
+
# The local run.json is written from the original payload and keeps all.
|
|
53
|
+
def within_budget(payload)
|
|
54
|
+
entries = payload.attachment_entries
|
|
55
|
+
bytes = 0
|
|
56
|
+
keep = entries.index do |entry|
|
|
57
|
+
bytes += File.size?(entry[:attachment].source_path).to_i
|
|
58
|
+
bytes > MAX_UPLOAD_ATTACHMENT_BYTES
|
|
59
|
+
end || entries.size
|
|
60
|
+
keep = [keep, MAX_UPLOAD_ATTACHMENTS].min
|
|
61
|
+
return payload if keep == entries.size
|
|
62
|
+
|
|
63
|
+
dropped = entries.size - keep
|
|
64
|
+
@warn.call("not uploading #{dropped} #{dropped == 1 ? 'attachment' : 'attachments'}: over the upload " \
|
|
65
|
+
"budget of #{MAX_UPLOAD_ATTACHMENT_BYTES / (1024 * 1024)} MB / " \
|
|
66
|
+
"#{MAX_UPLOAD_ATTACHMENTS} files; they remain in #{@config.output_dir}")
|
|
67
|
+
seen = 0
|
|
68
|
+
run = payload.run
|
|
69
|
+
Payload.new(run.with(results: run.results.map do |result|
|
|
70
|
+
result.with(attachments: result.attachments.select { (seen += 1) <= keep })
|
|
71
|
+
end))
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def add_field(body, boundary, name, value)
|
|
75
|
+
body << "--#{boundary}\r\nContent-Disposition: form-data; name=\"#{name}\"\r\n\r\n".b
|
|
76
|
+
body << value.to_s.b << "\r\n".b
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def add_file(body, boundary, name, attachment)
|
|
80
|
+
filename = File.basename(attachment.source_path).gsub(/[^\w.\-]/, "_")
|
|
81
|
+
body << ("--#{boundary}\r\nContent-Disposition: form-data; name=\"#{name}\"; filename=\"#{filename}\"\r\n" \
|
|
82
|
+
"Content-Type: #{attachment.content_type}\r\n\r\n").b
|
|
83
|
+
body << File.binread(attachment.source_path) << "\r\n".b
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def runs_uri
|
|
87
|
+
uri = URI.parse(@config.endpoint.to_s.strip.chomp("/") + "/api/v1/runs")
|
|
88
|
+
raise ArgumentError, "RAIDER_ENDPOINT must start with http:// or https://" unless uri.is_a?(URI::HTTP)
|
|
89
|
+
|
|
90
|
+
uri
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def to_result(response)
|
|
94
|
+
if response.is_a?(Net::HTTPSuccess)
|
|
95
|
+
body = parse_body(response.body)
|
|
96
|
+
Result.new(ok: true, url: present(body["url"]), run_id: present(body["run_id"]), error: nil)
|
|
97
|
+
else
|
|
98
|
+
Result.new(ok: false, url: nil, run_id: nil, error: "HTTP #{response.code}: #{response.body.to_s[0, 200]}")
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def parse_body(text)
|
|
103
|
+
parsed = JSON.parse(text.to_s)
|
|
104
|
+
parsed.is_a?(Hash) ? parsed : {}
|
|
105
|
+
rescue JSON::ParserError
|
|
106
|
+
{}
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def present(value)
|
|
110
|
+
value.nil? || value.to_s.strip.empty? ? nil : value
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
end
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "time"
|
|
4
|
+
require "rbconfig"
|
|
5
|
+
require_relative "raider_reporter/version"
|
|
6
|
+
require_relative "raider_reporter/models"
|
|
7
|
+
require_relative "raider_reporter/payload"
|
|
8
|
+
require_relative "raider_reporter/configuration"
|
|
9
|
+
require_relative "raider_reporter/content_types"
|
|
10
|
+
require_relative "raider_reporter/collector"
|
|
11
|
+
require_relative "raider_reporter/writers/local"
|
|
12
|
+
require_relative "raider_reporter/ci_detector"
|
|
13
|
+
require_relative "raider_reporter/writers/uploader"
|
|
14
|
+
|
|
15
|
+
module RaiderReporter
|
|
16
|
+
class << self
|
|
17
|
+
attr_reader :collector
|
|
18
|
+
|
|
19
|
+
def configuration
|
|
20
|
+
@configuration ||= Configuration.new
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def configure
|
|
24
|
+
yield configuration
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def reset!
|
|
28
|
+
@collector&.cleanup
|
|
29
|
+
@configuration = nil
|
|
30
|
+
@collector = nil
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def start_collector!
|
|
34
|
+
@collector = Collector.new(
|
|
35
|
+
max_attachment_bytes: configuration.max_attachment_bytes,
|
|
36
|
+
warn: ->(message) { warn(message) }
|
|
37
|
+
)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def attach(name, path, type: nil)
|
|
41
|
+
unless collector&.in_test?
|
|
42
|
+
warn("attach(#{name.inspect}) called outside a running example; ignored")
|
|
43
|
+
return false
|
|
44
|
+
end
|
|
45
|
+
unless File.file?(path.to_s)
|
|
46
|
+
warn("attach(#{name.inspect}): file #{path} not found; ignored")
|
|
47
|
+
return false
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
limit = configuration.max_attachment_bytes
|
|
51
|
+
if File.size(path.to_s) > limit
|
|
52
|
+
warn("attach(#{name.inspect}): file #{path} is larger than #{limit} bytes; ignored")
|
|
53
|
+
return false
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
collector.attach(Attachment.new(name: name.to_s, source_path: collector.stage(path.to_s),
|
|
57
|
+
content_type: ContentTypes.resolve(type, path)))
|
|
58
|
+
rescue StandardError => e
|
|
59
|
+
warn("attach(#{name.inspect}) failed: #{e.class}: #{e.message}")
|
|
60
|
+
false
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def warn(message)
|
|
64
|
+
# $stderr.puts, not Kernel.warn: -W0 must not silence reporting problems.
|
|
65
|
+
$stderr.puts("[raider_reporter] #{message}")
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def finish!(output = $stdout)
|
|
69
|
+
return nil if dry_run?
|
|
70
|
+
|
|
71
|
+
detector = CiDetector.new
|
|
72
|
+
run = collector.finish_run(ci: detector.ci, git: detector.git, environment: default_environment)
|
|
73
|
+
payload = Payload.new(run)
|
|
74
|
+
begin
|
|
75
|
+
Writers::Local.new(configuration).write(payload)
|
|
76
|
+
rescue StandardError => e
|
|
77
|
+
warn("could not write #{configuration.output_dir}: #{e.class}: #{e.message}")
|
|
78
|
+
end
|
|
79
|
+
return nil unless configuration.upload?
|
|
80
|
+
|
|
81
|
+
result = Writers::Uploader.new(configuration).upload(payload)
|
|
82
|
+
if result.ok
|
|
83
|
+
output.puts(success_line(result))
|
|
84
|
+
else
|
|
85
|
+
warn("upload failed: #{result.error}")
|
|
86
|
+
end
|
|
87
|
+
nil
|
|
88
|
+
ensure
|
|
89
|
+
collector&.cleanup
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
private
|
|
93
|
+
|
|
94
|
+
def success_line(result)
|
|
95
|
+
return "Raider report: #{result.url}" if result.url
|
|
96
|
+
return "Raider report uploaded (run #{result.run_id})" if result.run_id
|
|
97
|
+
|
|
98
|
+
"Raider report uploaded"
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
# `rspec --dry-run` reports every example as passed without running it.
|
|
102
|
+
def dry_run?
|
|
103
|
+
defined?(::RSpec.configuration) && ::RSpec.configuration.respond_to?(:dry_run?) &&
|
|
104
|
+
::RSpec.configuration.dry_run?
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def default_environment
|
|
108
|
+
{ "ruby" => RUBY_VERSION, "os" => RbConfig::CONFIG["host_os"] }
|
|
109
|
+
.merge((configuration.environment || {}).transform_keys(&:to_s))
|
|
110
|
+
end
|
|
111
|
+
end
|
|
112
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: raider_reporter
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Augustin Gottlieb
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies: []
|
|
12
|
+
description: RSpec formatter that records results, screenshots and videos in the Raider
|
|
13
|
+
schema, writes them locally, and optionally uploads them to a Raider server.
|
|
14
|
+
email:
|
|
15
|
+
- 33221555+aguspe@users.noreply.github.com
|
|
16
|
+
executables: []
|
|
17
|
+
extensions: []
|
|
18
|
+
extra_rdoc_files: []
|
|
19
|
+
files:
|
|
20
|
+
- CHANGELOG.md
|
|
21
|
+
- LICENSE
|
|
22
|
+
- README.md
|
|
23
|
+
- lib/raider_reporter.rb
|
|
24
|
+
- lib/raider_reporter/ci_detector.rb
|
|
25
|
+
- lib/raider_reporter/collector.rb
|
|
26
|
+
- lib/raider_reporter/configuration.rb
|
|
27
|
+
- lib/raider_reporter/content_types.rb
|
|
28
|
+
- lib/raider_reporter/models.rb
|
|
29
|
+
- lib/raider_reporter/payload.rb
|
|
30
|
+
- lib/raider_reporter/rspec.rb
|
|
31
|
+
- lib/raider_reporter/rspec/formatter.rb
|
|
32
|
+
- lib/raider_reporter/version.rb
|
|
33
|
+
- lib/raider_reporter/writers/local.rb
|
|
34
|
+
- lib/raider_reporter/writers/uploader.rb
|
|
35
|
+
homepage: https://github.com/RaiderHQ/raider_reporter
|
|
36
|
+
licenses:
|
|
37
|
+
- MIT
|
|
38
|
+
metadata:
|
|
39
|
+
source_code_uri: https://github.com/RaiderHQ/raider_reporter
|
|
40
|
+
changelog_uri: https://github.com/RaiderHQ/raider_reporter/blob/main/CHANGELOG.md
|
|
41
|
+
rubygems_mfa_required: 'true'
|
|
42
|
+
rdoc_options: []
|
|
43
|
+
require_paths:
|
|
44
|
+
- lib
|
|
45
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
46
|
+
requirements:
|
|
47
|
+
- - ">="
|
|
48
|
+
- !ruby/object:Gem::Version
|
|
49
|
+
version: '3.2'
|
|
50
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
51
|
+
requirements:
|
|
52
|
+
- - ">="
|
|
53
|
+
- !ruby/object:Gem::Version
|
|
54
|
+
version: '0'
|
|
55
|
+
requirements: []
|
|
56
|
+
rubygems_version: 4.0.6
|
|
57
|
+
specification_version: 4
|
|
58
|
+
summary: 'Test reporting for Ruby Raider: local run.json and Raider uploads'
|
|
59
|
+
test_files: []
|