raider_reporter 0.5.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +14 -0
- data/README.md +16 -2
- data/lib/raider_reporter/app_frame.rb +57 -0
- data/lib/raider_reporter/cli.rb +23 -2
- data/lib/raider_reporter/configuration.rb +2 -1
- data/lib/raider_reporter/cucumber/formatter.rb +8 -1
- data/lib/raider_reporter/models.rb +5 -1
- data/lib/raider_reporter/payload.rb +3 -1
- data/lib/raider_reporter/rspec/formatter.rb +17 -1
- data/lib/raider_reporter/version.rb +1 -1
- data/lib/raider_reporter/writers/html.rb +291 -0
- data/lib/raider_reporter/writers/local.rb +3 -0
- data/lib/raider_reporter.rb +14 -1
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6eb732095ea2284078a27ae18b268a3fe33a128e2ec131f95999bd278cd15cd2
|
|
4
|
+
data.tar.gz: 556f21195f523faabf7785eada01089a5537a7a4f052617152de87a0673c362b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ecf148c6450bdd03be6ed3634a89149ecd117d4b7e02cc65196d91c66cab050bedab950d3db7a2ebf9b66186cab381ff32d91e6783796f72bcb8a7d6a54877d9
|
|
7
|
+
data.tar.gz: 6dfe2cd5cb2ed735e124f580501c90ed9ef6ace13a21d7c37c7e8f8ed0bb25a6044f0ba248891b4c943898757f58bd077d7b57f9b05c0230cfce697d393f736b
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.7.0 — 2026-10-18
|
|
4
|
+
|
|
5
|
+
- Add a local HTML report: every run now also writes `raider-results/index.html`, one self-contained page (inline CSS and JS, no network requests) built from every `run*.json` in the folder, so shards from one machine share a report. Failures come first with their message, exception class, backtrace and screenshots; a filter box narrows by test name or file. Open it in a browser after a run, or publish it as a CI artifact
|
|
6
|
+
- `raider-reporter report [dir]` rebuilds the report from the run files on disk (default `raider-results`); exits 1 when there are no run files or the report cannot be written
|
|
7
|
+
- `config.html_report = false` turns the report off. An existing `index.html` that raider_reporter did not write is never overwritten or removed; a warning is printed instead
|
|
8
|
+
- Fail-open: a report error prints `[raider_reporter] HTML report failed: …` and leaves `run.json`, the upload and the exit status unchanged
|
|
9
|
+
- No breaking changes
|
|
10
|
+
|
|
11
|
+
## 0.6.0 — 2026-10-14
|
|
12
|
+
|
|
13
|
+
- Schema v1, additive: optional `failure.app_frame`, the first line of the raw exception backtrace that lies inside the project root (`Dir.pwd` when the reporter starts), outside `vendor/`, `.bundle/`, `Gem.path` and Ruby's library directory, and not `<internal:...>`. The path is made project-relative and the rest of the line is unchanged, e.g. `spec/pages/login_page.rb:42:in 'LoginPage#submit'`; at most 500 characters. The key is omitted when no line qualifies or computing it fails, so payloads without it are unchanged. Raider uses it to group failures by error
|
|
14
|
+
- Reported by both the RSpec and Cucumber reporters
|
|
15
|
+
- No breaking changes
|
|
16
|
+
|
|
3
17
|
## 0.5.1 — 2026-10-13
|
|
4
18
|
|
|
5
19
|
- Fix: a failed upload now reports one short line. An HTML error page (for example a server 500) is summarised as `the server returned an HTML error page` instead of being printed, other bodies are collapsed to one line of at most 200 characters, and the token is redacted from upload errors, including exception messages
|
data/README.md
CHANGED
|
@@ -136,7 +136,8 @@ end
|
|
|
136
136
|
| `token` | `ENV["RAIDER_TOKEN"]` | API token used for uploads. |
|
|
137
137
|
| `endpoint` | `ENV["RAIDER_ENDPOINT"]` | Base URL of the Raider server to upload to. |
|
|
138
138
|
| `output_dir` | `"raider-results"` | Directory for local output. |
|
|
139
|
-
| `clean_output_dir` | `true` | Remove the previous `run.json` and `attachments/` directory (`run-<n>.json` and `attachments-<n>/` when sharded) inside `output_dir` before writing. Nothing else in `output_dir` is touched. |
|
|
139
|
+
| `clean_output_dir` | `true` | Remove the previous `run.json` and `attachments/` directory (`run-<n>.json` and `attachments-<n>/` when sharded) inside `output_dir` before writing. Nothing else in `output_dir` is touched, except a raider_reporter-written `index.html`. |
|
|
140
|
+
| `html_report` | `true` | Write `index.html`, the local HTML report, into `output_dir` after each run. See [Local report](#local-report). |
|
|
140
141
|
| `environment` | `{}` | Free-form key/values stored with the run (browser, suite name, and so on). Values are JSON values: strings, booleans or numbers. |
|
|
141
142
|
|
|
142
143
|
## Local output
|
|
@@ -145,10 +146,23 @@ end
|
|
|
145
146
|
raider-results/
|
|
146
147
|
run.json # the run in schema v1
|
|
147
148
|
attachments/ # copies of attached screenshots and videos
|
|
149
|
+
index.html # the local report, see below
|
|
148
150
|
```
|
|
149
151
|
|
|
150
152
|
Add `raider-results/` to your `.gitignore`. When a run is sharded, the files are named per shard; see [Parallel runs](#parallel-runs).
|
|
151
153
|
|
|
154
|
+
## Local report
|
|
155
|
+
|
|
156
|
+
Every run also writes `raider-results/index.html`: one self-contained page (inline CSS and JS, no network requests, so it works offline) built from every `run*.json` in the folder. It shows totals and pass rate, then failures first with message, exception class, backtrace and screenshots, followed by the other statuses; a filter box narrows by test name or file. Open it in a browser or publish it as a CI artifact.
|
|
157
|
+
|
|
158
|
+
Rebuild it from the run files on disk, for example after `parallel_rspec` has written one file per shard:
|
|
159
|
+
|
|
160
|
+
```
|
|
161
|
+
raider-reporter report [dir] # default: raider-results; prints the path, exits 1 when there are no run files or the report cannot be written
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Set `config.html_report = false` to turn it off. An `index.html` that raider_reporter did not write is never overwritten or removed; a warning is printed instead.
|
|
165
|
+
|
|
152
166
|
## Uploading to Raider
|
|
153
167
|
|
|
154
168
|
If both a token and an endpoint are set, the run (plus attachments) is uploaded when the suite finishes:
|
|
@@ -305,7 +319,7 @@ Tip: run `rm -rf raider-results` before a parallel run. Each process only cleans
|
|
|
305
319
|
|
|
306
320
|
## Schema
|
|
307
321
|
|
|
308
|
-
`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are exactly `passed`, `failed`, `pending` and `skipped`.
|
|
322
|
+
`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are exactly `passed`, `failed`, `pending` and `skipped`. In both RSpec and Cucumber, a failure may carry `app_frame` (when a project frame is found): the first backtrace line inside your project, for example `spec/pages/login_page.rb:42:in 'LoginPage#submit'`, which Raider uses to group failures with the same error.
|
|
309
323
|
|
|
310
324
|
## Contributing
|
|
311
325
|
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rbconfig"
|
|
4
|
+
|
|
5
|
+
module RaiderReporter
|
|
6
|
+
# Finds the first project frame of a raw backtrace, so the server can group
|
|
7
|
+
# failures by where the project's own code raised them.
|
|
8
|
+
module AppFrame
|
|
9
|
+
MAX_LENGTH = 500
|
|
10
|
+
|
|
11
|
+
# Returns "relative/path.rb:42:in 'Class#method'" or nil when no line
|
|
12
|
+
# of the backtrace lies in project code.
|
|
13
|
+
def self.from(backtrace, root:)
|
|
14
|
+
return nil if backtrace.nil? || backtrace.empty? || root.nil?
|
|
15
|
+
|
|
16
|
+
base = "#{File.expand_path(root.to_s).chomp("/")}/"
|
|
17
|
+
excluded = excluded_prefixes
|
|
18
|
+
Array(backtrace).each do |line|
|
|
19
|
+
line = line.to_s
|
|
20
|
+
path = line[/\A(.+?):\d+/, 1]
|
|
21
|
+
next if path.nil? || path.start_with?("<internal:", "(", "-")
|
|
22
|
+
|
|
23
|
+
absolute = File.expand_path(path, base)
|
|
24
|
+
next unless File.file?(absolute)
|
|
25
|
+
|
|
26
|
+
absolute = inside_root(absolute, base)
|
|
27
|
+
next unless absolute
|
|
28
|
+
next if excluded.any? { |prefix| absolute.start_with?(prefix) }
|
|
29
|
+
|
|
30
|
+
relative = absolute.delete_prefix(base)
|
|
31
|
+
next if relative.start_with?("vendor/", ".bundle/") || relative.include?("/.bundle/")
|
|
32
|
+
|
|
33
|
+
return Text.utf8("#{relative}#{line[path.length..]}")[0, MAX_LENGTH]
|
|
34
|
+
end
|
|
35
|
+
nil
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Returns the path (rewritten under base when only the real paths match,
|
|
39
|
+
# e.g. a symlinked tmpdir) or nil when it lies outside the root.
|
|
40
|
+
def self.inside_root(absolute, base)
|
|
41
|
+
return absolute if absolute.start_with?(base)
|
|
42
|
+
|
|
43
|
+
real_base = "#{File.realpath(base)}/"
|
|
44
|
+
real = File.realpath(absolute)
|
|
45
|
+
real.start_with?(real_base) ? "#{base}#{real.delete_prefix(real_base)}" : nil
|
|
46
|
+
rescue SystemCallError
|
|
47
|
+
nil
|
|
48
|
+
end
|
|
49
|
+
private_class_method :inside_root
|
|
50
|
+
|
|
51
|
+
def self.excluded_prefixes
|
|
52
|
+
dirs = Array(Gem.path) + [RbConfig::CONFIG["rubylibdir"]]
|
|
53
|
+
dirs.compact.map(&:to_s).reject(&:empty?).map { |dir| "#{File.expand_path(dir).chomp("/")}/" }
|
|
54
|
+
end
|
|
55
|
+
private_class_method :excluded_prefixes
|
|
56
|
+
end
|
|
57
|
+
end
|
data/lib/raider_reporter/cli.rb
CHANGED
|
@@ -3,8 +3,9 @@
|
|
|
3
3
|
require "raider_reporter"
|
|
4
4
|
|
|
5
5
|
module RaiderReporter
|
|
6
|
-
# `raider-reporter comment
|
|
7
|
-
#
|
|
6
|
+
# `raider-reporter comment` posts the build summary as a PR comment and sets a commit status
|
|
7
|
+
# (fail-open: always exits 0, notices go to stderr). `raider-reporter report [dir]` rebuilds the
|
|
8
|
+
# local HTML report (exits 1 when there are no run files or the report cannot be written).
|
|
8
9
|
class CLI
|
|
9
10
|
MARKER = "<!-- raider-reporter:build-comment -->"
|
|
10
11
|
CONTEXT = "raider/tests"
|
|
@@ -23,6 +24,10 @@ module RaiderReporter
|
|
|
23
24
|
}.freeze
|
|
24
25
|
DEFAULT_API_URL = "https://api.github.com"
|
|
25
26
|
USAGE = <<~TEXT
|
|
27
|
+
Usage: raider-reporter report [dir]
|
|
28
|
+
|
|
29
|
+
Rebuilds <dir>/index.html (default raider-results) from every run*.json in it. Prints its path.
|
|
30
|
+
|
|
26
31
|
Usage: raider-reporter comment
|
|
27
32
|
|
|
28
33
|
Posts (or updates) the Raider build summary on the current pull request and sets a status.
|
|
@@ -43,6 +48,8 @@ module RaiderReporter
|
|
|
43
48
|
end
|
|
44
49
|
|
|
45
50
|
def run
|
|
51
|
+
return report if @argv.first == "report" && !@argv.include?("--help") && !@argv.include?("-h")
|
|
52
|
+
|
|
46
53
|
command = @argv.first
|
|
47
54
|
if command == "comment" && !@argv.include?("--help") && !@argv.include?("-h")
|
|
48
55
|
comment
|
|
@@ -57,6 +64,20 @@ module RaiderReporter
|
|
|
57
64
|
|
|
58
65
|
private
|
|
59
66
|
|
|
67
|
+
def report
|
|
68
|
+
dir = File.expand_path(@argv[1] || "raider-results")
|
|
69
|
+
path = Writers::Html.new.write(dir)
|
|
70
|
+
unless path
|
|
71
|
+
@err.puts("[raider_reporter] no run*.json in #{dir}")
|
|
72
|
+
return 1
|
|
73
|
+
end
|
|
74
|
+
@out.puts(path)
|
|
75
|
+
0
|
|
76
|
+
rescue StandardError => e
|
|
77
|
+
@err.puts("[raider_reporter] report failed: #{e.class}: #{e.message}")
|
|
78
|
+
1
|
|
79
|
+
end
|
|
80
|
+
|
|
60
81
|
def comment
|
|
61
82
|
if @env["GITHUB_ACTIONS"] == "true"
|
|
62
83
|
github_comment
|
|
@@ -4,13 +4,14 @@ require "raider_reporter/azure_env"
|
|
|
4
4
|
|
|
5
5
|
module RaiderReporter
|
|
6
6
|
class Configuration
|
|
7
|
-
attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :environment, :max_attachment_bytes
|
|
7
|
+
attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :html_report, :environment, :max_attachment_bytes
|
|
8
8
|
|
|
9
9
|
def initialize(env = ENV)
|
|
10
10
|
@token = AzureEnv.value(env, "RAIDER_TOKEN")
|
|
11
11
|
@endpoint = AzureEnv.value(env, "RAIDER_ENDPOINT")
|
|
12
12
|
@output_dir = "raider-results"
|
|
13
13
|
@clean_output_dir = true
|
|
14
|
+
@html_report = true
|
|
14
15
|
@environment = {}
|
|
15
16
|
@max_attachment_bytes = 5 * 1024 * 1024
|
|
16
17
|
end
|
|
@@ -199,10 +199,17 @@ module RaiderReporter
|
|
|
199
199
|
Failure.new(
|
|
200
200
|
message: Text.utf8("#{step_label(step_info, test_case)}\n\n#{exception.message}")[0, Text::MAX_MESSAGE],
|
|
201
201
|
exception_class: Text.utf8(exception.class.name),
|
|
202
|
-
backtrace: (exception.backtrace || []).first(Text::MAX_BACKTRACE).map { |line| Text.utf8(line) }
|
|
202
|
+
backtrace: (exception.backtrace || []).first(Text::MAX_BACKTRACE).map { |line| Text.utf8(line) },
|
|
203
|
+
app_frame: app_frame_for(exception)
|
|
203
204
|
)
|
|
204
205
|
end
|
|
205
206
|
|
|
207
|
+
def app_frame_for(exception)
|
|
208
|
+
AppFrame.from(exception.backtrace, root: RaiderReporter.project_root || Dir.pwd)
|
|
209
|
+
rescue StandardError
|
|
210
|
+
nil
|
|
211
|
+
end
|
|
212
|
+
|
|
206
213
|
def step_label(step_info, test_case)
|
|
207
214
|
return "Step" unless step_info
|
|
208
215
|
|
|
@@ -4,7 +4,11 @@ module RaiderReporter
|
|
|
4
4
|
SCHEMA_VERSION = 1
|
|
5
5
|
STATUSES = %w[passed failed pending skipped].freeze
|
|
6
6
|
|
|
7
|
-
Failure = Data.define(:message, :exception_class, :backtrace)
|
|
7
|
+
Failure = Data.define(:message, :exception_class, :backtrace, :app_frame) do
|
|
8
|
+
def initialize(message:, exception_class:, backtrace:, app_frame: nil)
|
|
9
|
+
super
|
|
10
|
+
end
|
|
11
|
+
end
|
|
8
12
|
Attachment = Data.define(:name, :source_path, :content_type)
|
|
9
13
|
TestResult = Data.define(
|
|
10
14
|
:identifier, :full_name, :file_path, :line_number, :status,
|
|
@@ -73,11 +73,13 @@ module RaiderReporter
|
|
|
73
73
|
end
|
|
74
74
|
|
|
75
75
|
def failure_hash(failure)
|
|
76
|
-
{
|
|
76
|
+
hash = {
|
|
77
77
|
"message" => failure.message,
|
|
78
78
|
"exception_class" => failure.exception_class,
|
|
79
79
|
"backtrace" => failure.backtrace
|
|
80
80
|
}
|
|
81
|
+
hash["app_frame"] = failure.app_frame if failure.app_frame
|
|
82
|
+
hash
|
|
81
83
|
end
|
|
82
84
|
|
|
83
85
|
def attachment_hash(entry, mode)
|
|
@@ -74,10 +74,26 @@ module RaiderReporter
|
|
|
74
74
|
Failure.new(
|
|
75
75
|
message: utf8(exception.message)[0, MAX_MESSAGE],
|
|
76
76
|
exception_class: utf8(exception.class.name),
|
|
77
|
-
backtrace: backtrace.first(MAX_BACKTRACE).map { |line| utf8(line) }
|
|
77
|
+
backtrace: backtrace.first(MAX_BACKTRACE).map { |line| utf8(line) },
|
|
78
|
+
app_frame: app_frame_for(exception)
|
|
78
79
|
)
|
|
79
80
|
end
|
|
80
81
|
|
|
82
|
+
# From the raw backtrace: RSpec's formatter strips gem and project-support frames.
|
|
83
|
+
# MultipleExceptionError (aggregate_failures) has no backtrace of its own.
|
|
84
|
+
def raw_backtrace(exception)
|
|
85
|
+
return exception.backtrace if exception.backtrace
|
|
86
|
+
return nil unless exception.respond_to?(:all_exceptions)
|
|
87
|
+
|
|
88
|
+
exception.all_exceptions.first&.backtrace
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def app_frame_for(exception)
|
|
92
|
+
AppFrame.from(raw_backtrace(exception), root: RaiderReporter.project_root || Dir.pwd)
|
|
93
|
+
rescue StandardError
|
|
94
|
+
nil
|
|
95
|
+
end
|
|
96
|
+
|
|
81
97
|
def utf8(value)
|
|
82
98
|
Text.utf8(value)
|
|
83
99
|
end
|
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "erb"
|
|
4
|
+
require "fileutils"
|
|
5
|
+
require "json"
|
|
6
|
+
require "time"
|
|
7
|
+
|
|
8
|
+
module RaiderReporter
|
|
9
|
+
module Writers
|
|
10
|
+
# Builds index.html from every run*.json in a results folder: one page with inline
|
|
11
|
+
# CSS and JS and no network requests, so a run can be read without Raider or Java.
|
|
12
|
+
class Html
|
|
13
|
+
class ForeignFile < StandardError; end
|
|
14
|
+
|
|
15
|
+
FILE = "index.html"
|
|
16
|
+
MARKER = '<meta name="generator" content="raider_reporter">'
|
|
17
|
+
ROW_LIMIT = 2_000
|
|
18
|
+
RUN_FILE = /\Arun(-[^\/]+)?\.json\z/
|
|
19
|
+
SECTIONS = [%w[failed Failed], %w[passed Passed], %w[pending Pending], %w[skipped Skipped]].freeze
|
|
20
|
+
PROVIDERS = { "github_actions" => "GitHub Actions", "azure_pipelines" => "Azure Pipelines" }.freeze
|
|
21
|
+
|
|
22
|
+
def self.ours?(path)
|
|
23
|
+
File.file?(path) && File.read(path, 4096).to_s.include?(MARKER)
|
|
24
|
+
rescue SystemCallError
|
|
25
|
+
false
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def write(dir)
|
|
29
|
+
dir = File.expand_path(dir)
|
|
30
|
+
files = run_files(dir)
|
|
31
|
+
return nil if files.empty?
|
|
32
|
+
|
|
33
|
+
target = File.join(dir, FILE)
|
|
34
|
+
if File.exist?(target) && !self.class.ours?(target)
|
|
35
|
+
raise ForeignFile, "#{target} exists and was not written by raider_reporter; not overwritten"
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
runs, skipped = load(files)
|
|
39
|
+
runs, ignored = newest_run(runs)
|
|
40
|
+
tmp = "#{target}.tmp-#{Process.pid}"
|
|
41
|
+
File.write(tmp, render(runs.map { |r| r[:data] }, skipped, ignored, project_name))
|
|
42
|
+
File.rename(tmp, target)
|
|
43
|
+
target
|
|
44
|
+
ensure
|
|
45
|
+
FileUtils.rm_f(tmp) if tmp && File.exist?(tmp)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
private
|
|
49
|
+
|
|
50
|
+
def run_files(dir)
|
|
51
|
+
return [] unless File.directory?(dir)
|
|
52
|
+
|
|
53
|
+
Dir.children(dir).grep(RUN_FILE).sort.map { |name| File.join(dir, name) }
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def load(files)
|
|
57
|
+
runs = []
|
|
58
|
+
skipped = []
|
|
59
|
+
files.each do |file|
|
|
60
|
+
data = JSON.parse(File.read(file))
|
|
61
|
+
raise TypeError unless data.is_a?(Hash) && data["results"].is_a?(Array)
|
|
62
|
+
|
|
63
|
+
runs << { name: File.basename(file), data: data, mtime: File.mtime(file) }
|
|
64
|
+
rescue StandardError
|
|
65
|
+
skipped << File.basename(file)
|
|
66
|
+
end
|
|
67
|
+
[runs, skipped]
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
SHARD_WINDOW = 300
|
|
71
|
+
|
|
72
|
+
# Old files from earlier runs stay in the folder, so report only the newest run:
|
|
73
|
+
# its own file, or all the shard files that belong to it. Everything else is named in the footer.
|
|
74
|
+
def newest_run(runs)
|
|
75
|
+
return [runs, []] if runs.empty?
|
|
76
|
+
|
|
77
|
+
newest = runs.max_by { |r| time(meta_of(r)["finished_at"]) || r[:mtime] }
|
|
78
|
+
keep = shard_of(newest) ? runs.select { |r| same_run?(r, newest) } : [newest]
|
|
79
|
+
[keep, (runs - keep).map { |r| r[:name] }.sort]
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def meta_of(run) = run[:data]["run"].is_a?(Hash) ? run[:data]["run"] : {}
|
|
83
|
+
|
|
84
|
+
def shard_of(run) = meta_of(run)["shard"].is_a?(Hash) ? meta_of(run)["shard"] : nil
|
|
85
|
+
|
|
86
|
+
def same_run?(run, newest)
|
|
87
|
+
return true if run.equal?(newest)
|
|
88
|
+
|
|
89
|
+
shard = shard_of(run)
|
|
90
|
+
return false unless shard
|
|
91
|
+
|
|
92
|
+
total = shard_of(newest)["total"]
|
|
93
|
+
return false if total && shard["total"] && shard["total"] != total
|
|
94
|
+
|
|
95
|
+
(started_at(run) - started_at(newest)).abs <= SHARD_WINDOW
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def started_at(run) = time(meta_of(run)["started_at"]) || run[:mtime]
|
|
99
|
+
|
|
100
|
+
def project_name
|
|
101
|
+
name = File.basename(Dir.pwd)
|
|
102
|
+
name.empty? || name == "/" ? "Raider run" : name
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def render(runs, skipped, ignored, project)
|
|
106
|
+
results = runs.flat_map { |run| run["results"] }.select { |r| r.is_a?(Hash) }
|
|
107
|
+
meta = runs.map { |run| run["run"].is_a?(Hash) ? run["run"] : {} }
|
|
108
|
+
counts = SECTIONS.to_h { |status, _| [status, results.count { |r| r["status"] == status }] }
|
|
109
|
+
counts["flaky"] = results.count { |r| r["status"] == "passed" && r["attempts"].to_i > 1 }
|
|
110
|
+
|
|
111
|
+
<<~HTML
|
|
112
|
+
<!doctype html>
|
|
113
|
+
<html lang="en">
|
|
114
|
+
<head>
|
|
115
|
+
<meta charset="utf-8">
|
|
116
|
+
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
117
|
+
#{MARKER}
|
|
118
|
+
<title>#{h(project)} · Raider report</title>
|
|
119
|
+
<style>#{CSS}</style>
|
|
120
|
+
</head>
|
|
121
|
+
<body>
|
|
122
|
+
<header>
|
|
123
|
+
<h1>#{h(project)}</h1>
|
|
124
|
+
<p class="totals">#{totals(counts)}</p>
|
|
125
|
+
<p class="meta">#{meta_line(meta)}</p>
|
|
126
|
+
</header>
|
|
127
|
+
<main>
|
|
128
|
+
#{results.empty? ? '<p class="empty">No results</p>' : filter_box}
|
|
129
|
+
#{SECTIONS.map { |status, label| section(status, label, results) }.join}
|
|
130
|
+
</main>
|
|
131
|
+
<footer>#{footer(skipped, ignored)}</footer>
|
|
132
|
+
<script>#{JS}</script>
|
|
133
|
+
</body>
|
|
134
|
+
</html>
|
|
135
|
+
HTML
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
def totals(counts)
|
|
139
|
+
passed = counts["passed"]
|
|
140
|
+
flaky = counts["flaky"]
|
|
141
|
+
decided = passed + counts["failed"]
|
|
142
|
+
rate = decided.zero? ? "—" : "#{(passed * 100.0 / decided).round}%"
|
|
143
|
+
parts = SECTIONS.map { |status, _| %(<span class="b #{status}">#{counts[status]} #{status}</span>) }
|
|
144
|
+
parts << %(<span class="b flaky">#{flaky} flaky</span>) if flaky.positive?
|
|
145
|
+
"#{parts.join(' ')} <span class=\"rate\">Pass rate #{rate}</span>"
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
def meta_line(meta)
|
|
149
|
+
parts = []
|
|
150
|
+
duration = run_duration(meta)
|
|
151
|
+
parts << "Duration #{h(duration)}" if duration
|
|
152
|
+
git = meta.map { |m| m["git"] }.find { |g| g.is_a?(Hash) } || {}
|
|
153
|
+
parts << "Branch #{h(git['branch'])}" if git["branch"]
|
|
154
|
+
parts << "Commit #{h(git['commit_sha'].to_s[0, 7])}" if git["commit_sha"]
|
|
155
|
+
ci = meta.map { |m| m["ci"] }.find { |c| c.is_a?(Hash) } || {}
|
|
156
|
+
parts << ci_link(ci) if ci["provider"]
|
|
157
|
+
parts << "Generated #{h(Time.now.utc.strftime('%Y-%m-%d %H:%M UTC'))}"
|
|
158
|
+
parts.join(" · ")
|
|
159
|
+
end
|
|
160
|
+
|
|
161
|
+
def ci_link(ci)
|
|
162
|
+
label = h(PROVIDERS.fetch(ci["provider"], ci["provider"]))
|
|
163
|
+
url = ci["build_url"].to_s
|
|
164
|
+
url.match?(%r{\Ahttps?://}) ? %(<a href="#{h(url)}">#{label}</a>) : label
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
def run_duration(meta)
|
|
168
|
+
starts = meta.filter_map { |m| time(m["started_at"]) }
|
|
169
|
+
ends = meta.filter_map { |m| time(m["finished_at"]) }
|
|
170
|
+
return nil if starts.empty? || ends.empty? || ends.max < starts.min
|
|
171
|
+
|
|
172
|
+
format_ms(((ends.max - starts.min) * 1000).round)
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
def time(value)
|
|
176
|
+
Time.iso8601(value.to_s)
|
|
177
|
+
rescue ArgumentError
|
|
178
|
+
nil
|
|
179
|
+
end
|
|
180
|
+
|
|
181
|
+
def filter_box
|
|
182
|
+
'<input id="q" type="search" placeholder="Filter by test name or file" aria-label="Filter tests">'
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
def section(status, label, results)
|
|
186
|
+
rows = results.select { |r| r["status"] == status }
|
|
187
|
+
return "" if rows.empty?
|
|
188
|
+
|
|
189
|
+
shown = rows.first(ROW_LIMIT).map { |r| row(r) }.join
|
|
190
|
+
more = rows.size > ROW_LIMIT ? %(<p class="more">#{rows.size - ROW_LIMIT} more not shown</p>) : ""
|
|
191
|
+
open = status == "failed" ? " open" : ""
|
|
192
|
+
%(<details class="s"#{open}><summary>#{label} (#{rows.size})</summary>#{shown}#{more}</details>\n)
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
def row(result)
|
|
196
|
+
key = "#{result['full_name']} #{result['file_path']}".downcase
|
|
197
|
+
location = [result["file_path"], result["line_number"]].compact.join(":")
|
|
198
|
+
attempts = result["attempts"].to_i > 1 ? %( <span class="att">#{result['attempts'].to_i} attempts</span>) : ""
|
|
199
|
+
<<~ROW
|
|
200
|
+
<div class="r #{h(result['status'])}" data-k="#{h(key)}">
|
|
201
|
+
<p class="n"><span class="b #{h(result['status'])}">#{h(result['status'])}</span> #{h(result['full_name'])}</p>
|
|
202
|
+
<p class="l">#{h(location)} · #{h(format_ms(result['duration_ms'].to_i))}#{attempts}</p>
|
|
203
|
+
#{failure(result['failure'])}#{attachments(result['attachments'])}
|
|
204
|
+
</div>
|
|
205
|
+
ROW
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
def failure(failure)
|
|
209
|
+
return "" unless failure.is_a?(Hash)
|
|
210
|
+
|
|
211
|
+
trace = Array(failure["backtrace"]).join("\n")
|
|
212
|
+
backtrace = trace.empty? ? "" : "<details><summary>Backtrace</summary><pre>#{h(trace)}</pre></details>"
|
|
213
|
+
%(<p class="x">#{h(failure['exception_class'])}</p><pre>#{h(failure['message'])}</pre>#{backtrace})
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
def attachments(list)
|
|
217
|
+
items = Array(list).filter_map do |attachment|
|
|
218
|
+
next unless attachment.is_a?(Hash)
|
|
219
|
+
|
|
220
|
+
href = safe_href(attachment["path"])
|
|
221
|
+
next unless href
|
|
222
|
+
|
|
223
|
+
if attachment["content_type"].to_s.start_with?("image/")
|
|
224
|
+
%(<a href="#{href}"><img loading="lazy" src="#{href}" alt="#{h(attachment['name'])}"></a>)
|
|
225
|
+
else
|
|
226
|
+
%(<a href="#{href}">#{h(attachment['name'])}</a>)
|
|
227
|
+
end
|
|
228
|
+
end
|
|
229
|
+
items.empty? ? "" : %(<div class="a">#{items.join(' ')}</div>)
|
|
230
|
+
end
|
|
231
|
+
|
|
232
|
+
# Relative paths inside the results folder only, each segment URL-encoded.
|
|
233
|
+
def safe_href(path)
|
|
234
|
+
path = path.to_s
|
|
235
|
+
segments = path.split("/")
|
|
236
|
+
return nil if path.empty? || path.start_with?("/") || segments.include?("..") || path.include?("\\")
|
|
237
|
+
|
|
238
|
+
segments.map { |segment| ERB::Util.url_encode(segment) }.join("/")
|
|
239
|
+
end
|
|
240
|
+
|
|
241
|
+
def footer(skipped, ignored)
|
|
242
|
+
note = skipped.empty? ? "" : "Skipped unreadable run files: #{h(skipped.join(', '))} · "
|
|
243
|
+
note += "Ignored older run files: #{h(ignored.join(', '))} · " unless ignored.empty?
|
|
244
|
+
"#{note}Generated by raider_reporter #{h(RaiderReporter::VERSION)}"
|
|
245
|
+
end
|
|
246
|
+
|
|
247
|
+
def format_ms(ms)
|
|
248
|
+
return "0ms" if ms.negative?
|
|
249
|
+
return "#{ms}ms" if ms < 1000
|
|
250
|
+
|
|
251
|
+
seconds = (ms / 1000.0).round
|
|
252
|
+
return format("%.1fs", ms / 1000.0) if ms < 59_950
|
|
253
|
+
|
|
254
|
+
"#{seconds / 60}m #{seconds % 60}s"
|
|
255
|
+
end
|
|
256
|
+
|
|
257
|
+
def h(value) = ERB::Util.html_escape(value.to_s)
|
|
258
|
+
|
|
259
|
+
CSS = <<~CSS.gsub(/\s*\n\s*/, "")
|
|
260
|
+
:root{--bg:#fff;--fg:#1f2328;--mute:#656d76;--bdr:#d0d7de;--card:#f6f8fa;--ruby:#C14420;--ok:#1a7f37;--warn:#9a6700}
|
|
261
|
+
@media (prefers-color-scheme:dark){:root{--bg:#0d1117;--fg:#e6edf3;--mute:#8d96a0;--bdr:#30363d;--card:#161b22;--ok:#3fb950;--warn:#d29922}}
|
|
262
|
+
*{box-sizing:border-box}
|
|
263
|
+
body{margin:0;padding:16px;background:var(--bg);color:var(--fg);font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
|
|
264
|
+
header,main,footer{max-width:1100px;margin:0 auto}
|
|
265
|
+
h1{margin:0 0 4px;font-size:20px;color:var(--ruby)}
|
|
266
|
+
.meta,.l,footer,.more{color:var(--mute);font-size:12px}
|
|
267
|
+
.b{display:inline-block;padding:0 6px;border-radius:9px;font-size:11px;font-weight:600;border:1px solid var(--bdr)}
|
|
268
|
+
.b.failed{background:var(--ruby);color:#fff;border-color:var(--ruby)}
|
|
269
|
+
.b.passed{color:var(--ok)}.b.pending,.b.skipped,.b.flaky{color:var(--warn)}
|
|
270
|
+
.rate{font-weight:600;margin-left:6px}
|
|
271
|
+
#q{width:100%;margin:12px 0;padding:8px;border:1px solid var(--bdr);border-radius:6px;background:var(--card);color:var(--fg)}
|
|
272
|
+
details.s{margin:8px 0;border:1px solid var(--bdr);border-radius:8px}
|
|
273
|
+
details.s>summary{padding:8px 12px;font-weight:600;cursor:pointer}
|
|
274
|
+
.r{padding:8px 12px;border-top:1px solid var(--bdr)}
|
|
275
|
+
.r p{margin:2px 0;overflow-wrap:anywhere}.n{font-weight:500}.x{color:var(--ruby);font-weight:600}
|
|
276
|
+
pre{margin:4px 0;padding:8px;background:var(--card);border-radius:6px;overflow-x:auto;white-space:pre-wrap;font-size:12px}
|
|
277
|
+
.a img{max-width:100%;max-height:240px;border:1px solid var(--bdr);border-radius:4px;margin-top:4px}
|
|
278
|
+
a{color:var(--ruby)}.empty{padding:24px 0;color:var(--mute)}
|
|
279
|
+
CSS
|
|
280
|
+
|
|
281
|
+
JS = <<~JS.gsub(/\s*\n\s*/, "")
|
|
282
|
+
(function(){var q=document.getElementById("q");if(!q)return;
|
|
283
|
+
q.addEventListener("input",function(){var v=q.value.toLowerCase();
|
|
284
|
+
document.querySelectorAll("details.s").forEach(function(s){var n=0;
|
|
285
|
+
s.querySelectorAll(".r").forEach(function(r){var hide=v!==""&&r.dataset.k.indexOf(v)<0;r.hidden=hide;if(!hide)n++;});
|
|
286
|
+
if(v===""){s.hidden=false;if(s.dataset.o!==undefined){s.open=s.dataset.o==="1";delete s.dataset.o;}}
|
|
287
|
+
else{if(s.dataset.o===undefined)s.dataset.o=s.open?"1":"0";s.hidden=n===0;s.open=n>0;}});});})();
|
|
288
|
+
JS
|
|
289
|
+
end
|
|
290
|
+
end
|
|
291
|
+
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "fileutils"
|
|
4
4
|
require "json"
|
|
5
|
+
require_relative "html"
|
|
5
6
|
|
|
6
7
|
module RaiderReporter
|
|
7
8
|
module Writers
|
|
@@ -28,6 +29,8 @@ module RaiderReporter
|
|
|
28
29
|
def clean(dir, payload)
|
|
29
30
|
FileUtils.rm_f(File.join(dir, payload.run_file))
|
|
30
31
|
FileUtils.rm_rf(File.join(dir, payload.attachments_dir))
|
|
32
|
+
index = File.join(dir, Html::FILE)
|
|
33
|
+
FileUtils.rm_f(index) if Html.ours?(index)
|
|
31
34
|
end
|
|
32
35
|
end
|
|
33
36
|
end
|
data/lib/raider_reporter.rb
CHANGED
|
@@ -5,11 +5,13 @@ require "rbconfig"
|
|
|
5
5
|
require_relative "raider_reporter/version"
|
|
6
6
|
require_relative "raider_reporter/text"
|
|
7
7
|
require_relative "raider_reporter/models"
|
|
8
|
+
require_relative "raider_reporter/app_frame"
|
|
8
9
|
require_relative "raider_reporter/payload"
|
|
9
10
|
require_relative "raider_reporter/configuration"
|
|
10
11
|
require_relative "raider_reporter/content_types"
|
|
11
12
|
require_relative "raider_reporter/collector"
|
|
12
13
|
require_relative "raider_reporter/writers/local"
|
|
14
|
+
require_relative "raider_reporter/writers/html"
|
|
13
15
|
require_relative "raider_reporter/ci_detector"
|
|
14
16
|
require_relative "raider_reporter/shard_detector"
|
|
15
17
|
require_relative "raider_reporter/writers/uploader"
|
|
@@ -19,7 +21,7 @@ require_relative "raider_reporter/comment/azure_client"
|
|
|
19
21
|
|
|
20
22
|
module RaiderReporter
|
|
21
23
|
class << self
|
|
22
|
-
attr_reader :collector
|
|
24
|
+
attr_reader :collector, :project_root
|
|
23
25
|
|
|
24
26
|
def configuration
|
|
25
27
|
@configuration ||= Configuration.new
|
|
@@ -37,10 +39,12 @@ module RaiderReporter
|
|
|
37
39
|
@collector&.cleanup
|
|
38
40
|
@configuration = nil
|
|
39
41
|
@collector = nil
|
|
42
|
+
@project_root = nil
|
|
40
43
|
end
|
|
41
44
|
|
|
42
45
|
def start_collector!
|
|
43
46
|
@dry_run = false
|
|
47
|
+
@project_root = Dir.pwd
|
|
44
48
|
@collector = Collector.new(
|
|
45
49
|
max_attachment_bytes: configuration.max_attachment_bytes,
|
|
46
50
|
warn: ->(message) { warn(message) }
|
|
@@ -88,6 +92,7 @@ module RaiderReporter
|
|
|
88
92
|
rescue StandardError => e
|
|
89
93
|
warn("could not write #{configuration.output_dir}: #{e.class}: #{e.message}")
|
|
90
94
|
end
|
|
95
|
+
write_html_report
|
|
91
96
|
return nil unless configuration.upload?
|
|
92
97
|
|
|
93
98
|
result = Writers::Uploader.new(configuration).upload(payload)
|
|
@@ -103,6 +108,14 @@ module RaiderReporter
|
|
|
103
108
|
|
|
104
109
|
private
|
|
105
110
|
|
|
111
|
+
def write_html_report
|
|
112
|
+
return unless configuration.html_report
|
|
113
|
+
|
|
114
|
+
Writers::Html.new.write(configuration.output_dir)
|
|
115
|
+
rescue StandardError => e
|
|
116
|
+
warn("HTML report failed: #{e.class}: #{e.message}")
|
|
117
|
+
end
|
|
118
|
+
|
|
106
119
|
def success_line(result)
|
|
107
120
|
link = result.build_url || result.url
|
|
108
121
|
return "Raider report: #{link}" if link
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: raider_reporter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Augustin Gottlieb
|
|
@@ -24,6 +24,7 @@ files:
|
|
|
24
24
|
- README.md
|
|
25
25
|
- exe/raider-reporter
|
|
26
26
|
- lib/raider_reporter.rb
|
|
27
|
+
- lib/raider_reporter/app_frame.rb
|
|
27
28
|
- lib/raider_reporter/azure_env.rb
|
|
28
29
|
- lib/raider_reporter/ci_detector.rb
|
|
29
30
|
- lib/raider_reporter/cli.rb
|
|
@@ -43,6 +44,7 @@ files:
|
|
|
43
44
|
- lib/raider_reporter/shard_detector.rb
|
|
44
45
|
- lib/raider_reporter/text.rb
|
|
45
46
|
- lib/raider_reporter/version.rb
|
|
47
|
+
- lib/raider_reporter/writers/html.rb
|
|
46
48
|
- lib/raider_reporter/writers/local.rb
|
|
47
49
|
- lib/raider_reporter/writers/uploader.rb
|
|
48
50
|
homepage: https://github.com/RaiderHQ/raider_reporter
|