raider_reporter 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6eb732095ea2284078a27ae18b268a3fe33a128e2ec131f95999bd278cd15cd2
-  data.tar.gz: 556f21195f523faabf7785eada01089a5537a7a4f052617152de87a0673c362b
+  metadata.gz: 300255531548132ae2fedcd7f10e8e9832925a204f5790ed8accb1d272ab64c6
+  data.tar.gz: ae60aacce4ecd140c8d771b92e8d83b704ec945a3339917d79fb7aeb50d49da9
 SHA512:
-  metadata.gz: ecf148c6450bdd03be6ed3634a89149ecd117d4b7e02cc65196d91c66cab050bedab950d3db7a2ebf9b66186cab381ff32d91e6783796f72bcb8a7d6a54877d9
-  data.tar.gz: 6dfe2cd5cb2ed735e124f580501c90ed9ef6ace13a21d7c37c7e8f8ed0bb25a6044f0ba248891b4c943898757f58bd077d7b57f9b05c0230cfce697d393f736b
+  metadata.gz: 0213a3b50dbeeedb77ac4e335afc632bf6a25ec815e875cabd4f54bed79fb4164b23770968f38911503055bdb6bc941460f379f08c7e34238f4c379740352f8e
+  data.tar.gz: 6c2c8759eee7dccfb81f93095d4a5c522b6029680ba999ab56073a91a76f5b7e7524da659c9350ea22f8185e1d9c1f294a75ee53aea4ceced898c36b699e0a8d

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.8.0 — 2026-10-24
+
+- Quarantine: with `RAIDER_QUARANTINE=1` (or `config.quarantine = true`) and a token and endpoint, the formatters fetch the project's quarantine list once per run (`GET /api/v1/quarantine`, 3-second timeout) and report matching failures as `quarantined`, keeping their message, backtrace and attachments. Fail-open: any fetch problem prints `[raider_reporter] quarantine list unavailable (<reason>); no tests quarantined` and nothing is quarantined
+- `raider-reporter exec -- <command…>` runs the suite (no shell) and exits 0 when the command exited 1 and every failure the framework counted was quarantined, with no errors outside tests; it prints `raider-reporter: N quarantined failure(s) did not fail the build`. Otherwise it exits with the command's code (128 + signal when killed, 127 when the command cannot start, 2 without a command)
+- Runs with a quarantined result are written as schema 2; every other run stays schema 1, so older Raider servers keep accepting them
+- The HTML report shows quarantined results in their own section and color; `raider-reporter comment` reads `N quarantined` from the headline
+- RSpec and Cucumber exit codes are unchanged without `exec`
+
 ## 0.7.0 — 2026-10-18
 
 - Add a local HTML report: every run now also writes `raider-results/index.html`, one self-contained page (inline CSS and JS, no network requests) built from every `run*.json` in the folder, so shards from one machine share a report. Failures come first with their message, exception class, backtrace and screenshots; a filter box narrows by test name or file. Open it in a browser after a run, or publish it as a CI artifact

data/README.md

@@ -1,6 +1,6 @@
 # raider_reporter
 
-Raider-schema test reporting for RSpec and Cucumber. `raider_reporter` is an RSpec formatter and a Cucumber reporter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
+Raider-schema test reporting for RSpec and Cucumber. `raider_reporter` is an RSpec formatter and a Cucumber reporter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1, or v2 when a run has a quarantined result), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
 
 It has no runtime dependencies and requires Ruby 3.2 or newer.
 
@@ -66,7 +66,7 @@ cucumber --format pretty --format RaiderReporter::Cucumber::Formatter --out raid
 - `identifier` is `"<file_path>:<line>"`: the scenario line, or the Examples row line for outline rows.
 - `tags` are the feature, scenario and Examples tags, merged, sorted, unique, without `@`.
 
-Statuses: passed, failed and ambiguous (reported as failed), pending and undefined (reported as pending), skipped.
+Statuses: passed, failed and ambiguous (reported as failed), pending and undefined (reported as pending), skipped. Failed (and ambiguous) scenarios on the quarantine list are reported as quarantined.
 
 ### Failures
 
@@ -138,13 +138,14 @@ end
 | `output_dir` | `"raider-results"` | Directory for local output. |
 | `clean_output_dir` | `true` | Remove the previous `run.json` and `attachments/` directory (`run-<n>.json` and `attachments-<n>/` when sharded) inside `output_dir` before writing. Nothing else in `output_dir` is touched, except a raider_reporter-written `index.html`. |
 | `html_report` | `true` | Write `index.html`, the local HTML report, into `output_dir` after each run. See [Local report](#local-report). |
+| `quarantine` | `true` when `ENV["RAIDER_QUARANTINE"]` is `1` or `true` | Fetch the quarantine list at run start and report matching failures as `quarantined`. Needs `token` and `endpoint`. See [Quarantine](#quarantine). |
 | `environment` | `{}` | Free-form key/values stored with the run (browser, suite name, and so on). Values are JSON values: strings, booleans or numbers. |
 
 ## Local output
 
 ```
 raider-results/
-  run.json          # the run in schema v1
+  run.json          # the run in schema v1 (v2 with quarantined results)
   attachments/      # copies of attached screenshots and videos
   index.html        # the local report, see below
 ```
@@ -179,6 +180,24 @@ Reporting is fail-open: no error inside the gem, network failure, timeout or non
 
 The hosted Raider service is not public yet; uploads go to a Raider server whose address you provide in `RAIDER_ENDPOINT`.
 
+## Quarantine
+
+A test quarantined in Raider (with a reason and an expiry) still runs and is still reported, but its failure is reported as `quarantined`, and a CI build stays green when quarantined tests are its only failures.
+
+1. Turn it on in CI with `RAIDER_QUARANTINE=1` (or `config.quarantine = true`), next to `RAIDER_TOKEN` and `RAIDER_ENDPOINT`.
+2. Run the suite through `raider-reporter exec`:
+
+   ```
+   bundle exec raider-reporter exec -- bundle exec rspec
+   bundle exec raider-reporter exec -- bundle exec cucumber
+   ```
+
+When the run starts, the formatter fetches the project's quarantine list once (`GET /api/v1/quarantine`, 3-second timeout). On any error it prints `[raider_reporter] quarantine list unavailable (<reason>); no tests quarantined` and nothing is quarantined. Matching failures keep their message, backtrace and screenshots; the run is written as schema 2, and `run.json`, the upload and the HTML report show `quarantined`.
+
+`exec` runs the command without a shell and exits with its status, with one exception: when the command exits 1, every failure the framework counted is quarantined, and nothing went wrong outside tests, it prints `raider-reporter: N quarantined failure(s) did not fail the build` to stderr and exits 0. "Outside tests" means RSpec errors outside examples, a run stopped early (`--fail-fast`, an interrupt, a load error), or a Cucumber run that did not finish every scenario. RSpec `--fail-fast` counts even when the last example triggers it, and interrupts count too, so the build stays red. Under Cucumber `--strict`, a flaky scenario (failed, then passed on retry) counts as a real failure and keeps the build red. A command killed by a signal exits 128 + the signal number; a command that cannot be started exits 127. Several processes under one `exec` (for example `parallel_rspec`) never turn red into green: their summaries conflict and the command's exit code is used.
+
+Without `exec`, RSpec and Cucumber exit codes are unchanged; quarantine only changes how failures are reported.
+
 ## Pull request comments
 
 On pull requests, `raider-reporter comment` posts one comment with the build's results (and sets a `raider/tests` commit status on the PR head commit). Run it in a job that runs after your test jobs, so every shard has uploaded. Use the workflow's `GITHUB_TOKEN` (or a GitHub App token): a personal access token posts as a user, so the step adds a new comment on every run instead of updating the existing one.
@@ -319,7 +338,7 @@ Tip: run `rm -rf raider-results` before a parallel run. Each process only cleans
 
 ## Schema
 
-`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are exactly `passed`, `failed`, `pending` and `skipped`. In both RSpec and Cucumber, a failure may carry `app_frame` (when a project frame is found): the first backtrace line inside your project, for example `spec/pages/login_page.rb:42:in 'LoginPage#submit'`, which Raider uses to group failures with the same error.
+`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are `passed`, `failed`, `pending` and `skipped`, plus `quarantined` in schema 2 (written only when a run has a quarantined result). In both RSpec and Cucumber, a failure may carry `app_frame` (when a project frame is found): the first backtrace line inside your project, for example `spec/pages/login_page.rb:42:in 'LoginPage#submit'`, which Raider uses to group failures with the same error.
 
 ## Contributing
 

data/lib/raider_reporter/cli.rb

@@ -1,8 +1,10 @@
 # frozen_string_literal: true
 
 require "raider_reporter"
+require "raider_reporter/exec"
 
 module RaiderReporter
+  # `raider-reporter exec -- <command…>` runs the test suite and decides the build's exit code (see Exec).
   # `raider-reporter comment` posts the build summary as a PR comment and sets a commit status
   # (fail-open: always exits 0, notices go to stderr). `raider-reporter report [dir]` rebuilds the
   # local HTML report (exits 1 when there are no run files or the report cannot be written).
@@ -24,6 +26,8 @@ module RaiderReporter
     }.freeze
     DEFAULT_API_URL = "https://api.github.com"
     USAGE = <<~TEXT
+      #{Exec::USAGE.chomp}
+
       Usage: raider-reporter report [dir]
 
       Rebuilds <dir>/index.html (default raider-results) from every run*.json in it. Prints its path.
@@ -48,6 +52,20 @@ module RaiderReporter
     end
 
     def run
+      return run_exec if @argv.first == "exec"
+
+      run_reporting
+    end
+
+    private
+
+    # exec decides the build's exit code; Exec owns its own failure handling.
+    def run_exec
+      command = @argv[1] == "--" ? @argv.drop(2) : []
+      Exec.new(command, err: @err).run
+    end
+
+    def run_reporting
       return report if @argv.first == "report" && !@argv.include?("--help") && !@argv.include?("-h")
 
       command = @argv.first
@@ -62,8 +80,6 @@ module RaiderReporter
       0
     end
 
-    private
-
     def report
       dir = File.expand_path(@argv[1] || "raider-results")
       path = Writers::Html.new.write(dir)
@@ -181,7 +197,7 @@ module RaiderReporter
       url.empty? ? DEFAULT_API_URL : url
     end
 
-    # "**Raider: failed** - 2 failed, 148 passed, 3 skipped (1m 12s)" -> "2 failed, 148 passed, 3 skipped".
+    # "**Raider: failed** - 2 failed, 1 quarantined, 148 passed (1m 12s)" -> "2 failed, 1 quarantined, 148 passed".
     # Falls back to a fixed text per state when the Markdown carries no headline counts.
     def description(summary)
       counts = headline_counts(summary.markdown)
@@ -193,7 +209,7 @@ module RaiderReporter
       line = markdown.to_s.lines.find { |l| l.include?("Raider:") }
       return nil unless line
 
-      counts = line.scan(/\b\d+ (?:failed|passed|skipped|pending)\b/)
+      counts = line.scan(/\b\d+ (?:failed|quarantined|passed|skipped|pending)\b/)
       counts.empty? ? nil : counts.join(", ")
     end
 

data/lib/raider_reporter/collector.rb

@@ -3,15 +3,19 @@
 require "securerandom"
 require "tmpdir"
 require "fileutils"
+require "set"
 
 module RaiderReporter
   # Accumulates one suite run. Framework formatters call it; writers read the Run it returns.
   class Collector
-    def initialize(max_attachment_bytes:, warn:, clock: -> { Time.now.utc }, uuid: -> { SecureRandom.uuid })
+    def initialize(max_attachment_bytes:, warn:, clock: -> { Time.now.utc }, uuid: -> { SecureRandom.uuid },
+                   quarantine: -> { Quarantine::EMPTY })
       @max_attachment_bytes = max_attachment_bytes
       @warn = warn
       @clock = clock
       @uuid = uuid
+      @quarantine = quarantine
+      @quarantined = nil
       @results = []
       @pending_attachments = nil
       @staged = 0
@@ -20,6 +24,26 @@ module RaiderReporter
     def start_run
       @started_at = @clock.call
       @results = []
+      @quarantined = load_quarantine
+    end
+
+    # True when the test's key is in the list loaded at run start.
+    def quarantined?(file_path, full_name)
+      return false if @quarantined.nil? || @quarantined.empty?
+
+      @quarantined.include?(RaiderReporter.test_key(file_path, full_name))
+    end
+
+    attr_reader :framework_counts
+
+    # What the framework itself counted (spec §4.4); nil until it reports, and
+    # then no exec summary is written.
+    def record_framework_counts(failures:, errors_outside_tests:)
+      @framework_counts = { failures: Integer(failures), errors_outside_tests: Integer(errors_outside_tests) }
+    end
+
+    def quarantined_count
+      @results.count { |result| result.status == "quarantined" }
     end
 
     def start_test
@@ -81,6 +105,13 @@ module RaiderReporter
 
     private
 
+    def load_quarantine
+      @quarantine.call
+    rescue StandardError => e
+      @warn.call(Quarantine.warning("#{e.class}: #{e.message}"))
+      Quarantine::EMPTY
+    end
+
     def keep?(attachment)
       path = attachment.source_path
       unless File.file?(path)

data/lib/raider_reporter/configuration.rb

@@ -4,7 +4,10 @@ require "raider_reporter/azure_env"
 
 module RaiderReporter
   class Configuration
-    attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :html_report, :environment, :max_attachment_bytes
+    TRUTHY = %w[1 true].freeze
+
+    attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :html_report, :environment, :max_attachment_bytes,
+                  :quarantine, :exec_summary_path
 
     def initialize(env = ENV)
       @token = AzureEnv.value(env, "RAIDER_TOKEN")
@@ -14,12 +17,19 @@ module RaiderReporter
       @html_report = true
       @environment = {}
       @max_attachment_bytes = 5 * 1024 * 1024
+      @quarantine = TRUTHY.include?(AzureEnv.value(env, "RAIDER_QUARANTINE").to_s.downcase)
+      @exec_summary_path = AzureEnv.value(env, "RAIDER_EXEC_SUMMARY")
     end
 
     def upload?
       present?(token) && present?(endpoint)
     end
 
+    # Quarantine needs the server: on, with both token and endpoint (spec §4.1).
+    def quarantine?
+      quarantine == true && upload?
+    end
+
     private
 
     def present?(value)

data/lib/raider_reporter/cucumber/formatter.rb

@@ -84,6 +84,10 @@ module RaiderReporter
         @buffer = {}
         @attempts = Hash.new(0)
         @attempt = nil
+        @outcomes = {}
+        @failed_before = {}
+        @outcomes_broken = false
+        @expected_cases = nil
         guard { setup(config) if self.class.claim(config) }
       end
 
@@ -95,6 +99,8 @@ module RaiderReporter
         @collector = RaiderReporter.start_collector!
         RaiderReporter.dry_run = config.dry_run?
         config.on_event(:test_run_started) { |_event| guard { @collector.start_run } }
+        config.on_event(:test_run_started) { |event| guard { expect_cases(event.test_cases) } }
+        config.on_event(:test_case_finished) { |event| count_outcome(event.test_case, event.result) }
         config.on_event(:test_case_started) { |event| guard { case_started(event.test_case) } }
         config.on_event(:test_step_finished) { |event| guard { step_finished(event.test_step, event.result) } }
         config.on_event(:test_case_finished) { |event| guard { case_finished(event.test_case, event.result) } }
@@ -124,6 +130,7 @@ module RaiderReporter
         # Always buffer: a lookup problem may lose detail, never the scenario.
         failure = status == "failed" ? safely(nil) { failure_for(problem, result, test_case) } : nil
         fields = identity_for(test_case)
+        status = "quarantined" if status == "failed" && quarantined?(fields)
         key = "#{test_case.location}|#{test_case.name}"
         @attempts[key] += 1
         # The newest attempt replaces the older one, taking its attachments with it.
@@ -135,11 +142,42 @@ module RaiderReporter
       def run_finished
         @buffer.each_value { |result| @collector.add_result(result) }
         @buffer.clear
+        safely(nil) { record_framework_counts }
         RaiderReporter.finish!($stderr)
       ensure
         self.class.release(@config)
       end
 
+      def expect_cases(test_cases)
+        @expected_cases = test_cases.map { |test_case| "#{test_case.location}|#{test_case.name}" }.uniq.size
+      end
+
+      # Its own handler, registered before case_finished: if counting ever fails, no
+      # summary is written and exec keeps cucumber's exit code. The newest attempt wins,
+      # except that under strict flaky (--strict, --strict-flaky) a scenario that failed
+      # on an earlier attempt and then passed still fails cucumber's run.
+      def count_outcome(test_case, result)
+        return if @outcomes_broken
+
+        key = "#{test_case.location}|#{test_case.name}"
+        failed = !result.ok?(strict: @config.strict)
+        flaky = !failed && @failed_before[key] && @config.strict.strict?(:flaky)
+        @failed_before[key] ||= failed
+        @outcomes[key] = failed || flaky ? true : false
+      rescue StandardError => e
+        @outcomes_broken = true
+        RaiderReporter.warn("#{e.class}: #{e.message}")
+      end
+
+      # Scenarios that make cucumber's run fail; one error outside tests when the run
+      # stopped before every scenario finished (--fail-fast) (spec §4.4).
+      def record_framework_counts
+        return if @outcomes_broken || @expected_cases.nil?
+
+        aborted = @outcomes.size < @expected_cases ? 1 : 0
+        @collector.record_framework_counts(failures: @outcomes.values.count(true), errors_outside_tests: aborted)
+      end
+
       def identity_for(test_case)
         file_path = relative_path(test_case.location.file)
         line_number = test_case.location.lines.max
@@ -152,6 +190,11 @@ module RaiderReporter
         }
       end
 
+      # A lookup problem must never turn a real failure into a quarantined one.
+      def quarantined?(fields)
+        safely(false) { @collector.quarantined?(fields[:file_path], fields[:full_name]) }
+      end
+
       def full_name(test_case)
         safely(test_case.name) do
           feature = @ast.gherkin_document(test_case.location.file)&.feature

data/lib/raider_reporter/exec.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+require "tmpdir"
+require "raider_reporter/exec_summary"
+
+module RaiderReporter
+  # `raider-reporter exec -- <command…>` (spec §4.5). Runs the command without a
+  # shell, streams its output, and keeps its exit code, except that exit 1 becomes 0
+  # when the summary says every counted failure was quarantined and nothing else
+  # went wrong. Anything uncertain keeps the command's code.
+  class Exec
+    USAGE = <<~TEXT
+      Usage: raider-reporter exec -- <command> [args...]
+
+      Runs the command (no shell) and exits with its status. When the command exits 1, every failure it
+      counted was quarantined in Raider and nothing else went wrong, exits 0 instead and says so on stderr.
+      Quarantine needs RAIDER_QUARANTINE=1 with RAIDER_TOKEN and RAIDER_ENDPOINT.
+    TEXT
+    USAGE_ERROR = 2
+    NOT_STARTED = 127
+    # A terminal Ctrl-C already reaches the command through the process group.
+    FORWARDED = %w[TERM HUP].freeze
+    IGNORED = %w[INT].freeze
+
+    def self.quarantined_only?(summary)
+      return false unless summary
+
+      failures = summary["framework_failures"]
+      failures.positive? && failures == summary["quarantined"] && summary["errors_outside_tests"].zero?
+    end
+
+    def initialize(command, err: $stderr)
+      @command = command
+      @err = err
+      @pid = nil
+      @code = nil
+      @pending = nil
+    end
+
+    def run
+      if @command.empty?
+        @err.puts(USAGE)
+        return USAGE_ERROR
+      end
+
+      Dir.mktmpdir("raider-exec") { |dir| run_with(File.join(dir, "summary.json")) }
+    rescue StandardError => e
+      @err.puts("[raider_reporter] exec failed: #{e.class}: #{e.message}")
+      @code || 1
+    end
+
+    private
+
+    def run_with(summary_path)
+      previous = install_traps
+      @pid = start(summary_path)
+      return NOT_STARTED unless @pid
+
+      forward(@pending) if @pending
+
+      @code = wait
+      summary = ExecSummary.read(summary_path)
+      return @code unless @code == 1 && self.class.quarantined_only?(summary)
+
+      @err.puts("raider-reporter: #{summary['quarantined']} quarantined failure(s) did not fail the build")
+      0
+    ensure
+      previous&.each { |signal, handler| trap(signal, handler || "DEFAULT") }
+    end
+
+    def start(summary_path)
+      program, *args = @command
+      Process.spawn({ ExecSummary::ENV_NAME => summary_path }, [program, program], *args)
+    rescue SystemCallError => e
+      @err.puts("[raider_reporter] exec: cannot run #{program}: #{e.message}")
+      nil
+    end
+
+    def wait
+      _, status = Process.wait2(@pid)
+      status.signaled? ? 128 + status.termsig : status.exitstatus
+    end
+
+    def forward(signal)
+      Process.kill(signal, @pid)
+    rescue SystemCallError
+      nil
+    end
+
+    # Installed before spawning; a signal that arrives before the child exists is kept in
+    # @pending and forwarded right after spawn. A Ruby handler (not "IGNORE") is reset to
+    # the default in the child by exec(2).
+    def install_traps
+      forwarded = FORWARDED.to_h do |signal|
+        handler = trap(signal) { @pid ? forward(signal) : @pending = signal }
+        [signal, handler]
+      end
+      forwarded.merge(IGNORED.to_h { |signal| [signal, trap(signal) {}] })
+    end
+  end
+end

data/lib/raider_reporter/exec_summary.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+require "json"
+
+module RaiderReporter
+  # The file `raider-reporter exec` reads to decide its exit code (spec §4.4). One
+  # process writes it. A second writer (several processes under one exec) turns it
+  # into a conflict, which exec treats as no summary, so the command's code wins.
+  module ExecSummary
+    ENV_NAME = "RAIDER_EXEC_SUMMARY"
+    KEYS = %w[framework_failures quarantined errors_outside_tests].freeze
+    CONFLICT = { "conflict" => true }.freeze
+
+    module_function
+
+    def write(path, framework_failures:, quarantined:, errors_outside_tests:, pid: Process.pid)
+      summary = { "framework_failures" => framework_failures, "quarantined" => quarantined,
+                  "errors_outside_tests" => errors_outside_tests, "pid" => pid }
+      File.open(path, File::RDWR | File::CREAT, 0o600) do |file|
+        file.flock(File::LOCK_EX)
+        existing = file.read
+        same_writer = existing.empty? || parse(existing)&.fetch("pid", nil) == pid
+        file.rewind
+        file.truncate(0)
+        file.write(JSON.generate(same_writer ? summary : CONFLICT))
+      end
+    end
+
+    def read(path)
+      data = parse(File.read(path))
+      return nil if data.nil? || data.key?("conflict")
+      return nil unless KEYS.all? { |key| data[key].is_a?(Integer) && !data[key].negative? }
+
+      data.slice(*KEYS)
+    rescue SystemCallError, IOError
+      nil
+    end
+
+    def parse(text)
+      data = JSON.parse(text)
+      data.is_a?(Hash) ? data : nil
+    rescue JSON::ParserError
+      nil
+    end
+  end
+end

data/lib/raider_reporter/models.rb

@@ -1,8 +1,8 @@
 # frozen_string_literal: true
 
 module RaiderReporter
-  SCHEMA_VERSION = 1
-  STATUSES = %w[passed failed pending skipped].freeze
+  SCHEMA_VERSION = 2
+  STATUSES = %w[passed failed pending skipped quarantined].freeze
 
   Failure = Data.define(:message, :exception_class, :backtrace, :app_frame) do
     def initialize(message:, exception_class:, backtrace:, app_frame: nil)

data/lib/raider_reporter/payload.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module RaiderReporter
-  # Serializes a Run into schema v1. Local mode points attachments at copied
+  # Serializes a Run: schema 1, or schema 2 when a result is quarantined. Local mode points attachments at copied
   # files; upload mode points them at multipart part names.
   class Payload
     MODES = %i[local upload].freeze
@@ -17,7 +17,7 @@ module RaiderReporter
 
       entries = attachment_entries.each
       {
-        "schema_version" => SCHEMA_VERSION,
+        "schema_version" => schema_version,
         "client_run_id" => @run.client_run_id,
         "run" => run_hash,
         "results" => @run.results.map do |result|
@@ -34,6 +34,10 @@ module RaiderReporter
 
     def run_file = shard_index ? "run-#{shard_index}.json" : "run.json"
 
+    # Version 2 only adds the quarantined status; older servers keep accepting
+    # every run that has none (spec §4.3).
+    def schema_version = @run.results.any? { |result| result.status == "quarantined" } ? SCHEMA_VERSION : 1
+
     def attachment_entries
       @attachment_entries ||= @run.results.flat_map(&:attachments).each_with_index.map do |attachment, index|
         {

data/lib/raider_reporter/quarantine.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+require "net/http"
+require "set"
+require "timeout"
+require "uri"
+
+module RaiderReporter
+  NUL = "\u0000"
+  NUL_REPLACEMENT = "\u{FFFD}"
+
+  # The server's test identity: raider_saas TestCase.key_for over the payload's
+  # file_path and full_name, after ingest replaced NUL in each of them.
+  def self.test_key(file_path, full_name)
+    parts = [file_path, full_name].map { |part| part.to_s.tr(NUL, NUL_REPLACEMENT) }
+    Digest::SHA256.hexdigest(parts.join(NUL))
+  end
+
+  # Fetches the project's quarantine list (spec §4.2). Fail-open: any problem
+  # means one warning and an empty set, never an exception.
+  module Quarantine
+    PATH = "/api/v1/quarantine"
+    TIMEOUT = 3
+    EMPTY = Set.new.freeze
+
+    class Unavailable < StandardError; end
+
+    def self.load(config, warn:)
+      return EMPTY unless config.quarantine?
+
+      Set.new(fetch(config)).freeze
+    rescue StandardError => e
+      reason = e.is_a?(Unavailable) ? e.message : "#{e.class}: #{e.message}"
+      warn.call(warning(reason, token: config.token))
+      EMPTY
+    end
+
+    # The one place the fail-open warning text is built (used by load and the Collector).
+    def self.warning(reason, token: nil)
+      "quarantine list unavailable (#{redact(reason, token)}); no tests quarantined"
+    end
+
+    def self.fetch(config)
+      uri = URI.parse("#{config.endpoint.to_s.strip.chomp('/')}#{PATH}")
+      raise Unavailable, "RAIDER_ENDPOINT must start with http:// or https://" unless uri.is_a?(URI::HTTP)
+
+      request = Net::HTTP::Get.new(uri)
+      request["Authorization"] = "Bearer #{config.token}"
+      request["Accept"] = "application/json"
+      request["User-Agent"] = "raider_reporter/#{VERSION}"
+      # The per-step timeouts do not bound a server that trickles bytes; the deadline does.
+      response = Timeout.timeout(TIMEOUT) do
+        Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
+                                            open_timeout: TIMEOUT, read_timeout: TIMEOUT, max_retries: 0) { |http| http.request(request) }
+      end
+      raise Unavailable, "HTTP #{response.code}" unless response.code == "200"
+
+      keys(response.body)
+    end
+
+    def self.keys(body)
+      data = JSON.parse(body.to_s)
+      list = data.is_a?(Hash) ? data["quarantined"] : nil
+      raise Unavailable, "unexpected response" unless list.is_a?(Array)
+
+      list.filter_map { |entry| entry["test_key"] if entry.is_a?(Hash) && entry["test_key"].is_a?(String) }
+    rescue JSON::ParserError
+      raise Unavailable, "invalid JSON"
+    end
+
+    def self.redact(text, token)
+      token.to_s.empty? ? text : text.gsub(token.to_s, "[REDACTED]")
+    end
+
+    private_class_method :fetch, :keys, :redact
+  end
+end

data/lib/raider_reporter/rspec/formatter.rb

@@ -11,7 +11,7 @@ module RaiderReporter
     class Formatter
       ::RSpec::Core::Formatters.register self,
                                          :start, :example_started, :example_passed,
-                                         :example_failed, :example_pending, :close
+                                         :example_failed, :example_pending, :dump_summary, :close
 
       NON_TAG_KEYS = %i[pending skip].freeze
       MAX_MESSAGE = Text::MAX_MESSAGE
@@ -45,6 +45,13 @@ module RaiderReporter
         end
       end
 
+      def dump_summary(notification)
+        guard do
+          @collector.record_framework_counts(failures: notification.failure_count,
+                                             errors_outside_tests: errors_outside_tests(notification))
+        end
+      end
+
       def close(_notification)
         guard { RaiderReporter.finish! }
       end
@@ -54,19 +61,32 @@ module RaiderReporter
       def record(example, status)
         metadata = example.metadata
         execution = example.execution_result
+        file_path = metadata[:file_path].to_s.delete_prefix("./")
+        full_name = utf8(example.full_description)
+        failure = status == "failed" ? failure_for(execution.exception) : nil
+        # The key is computed from the payload's own file_path and full_name (spec §4.2).
+        status = "quarantined" if status == "failed" && @collector.quarantined?(file_path, full_name)
         @collector.finish_test(
           identifier: example.id,
-          full_name: utf8(example.full_description),
-          file_path: metadata[:file_path].to_s.delete_prefix("./"),
+          full_name: full_name,
+          file_path: file_path,
           line_number: metadata[:line_number],
           status: status,
           duration_ms: ((execution.run_time || 0) * 1000).round,
           attempts: metadata.fetch(:retry_attempts, 0).to_i + 1,
           tags: metadata.select { |key, value| value == true && !NON_TAG_KEYS.include?(key) }.keys.map(&:to_s).sort,
-          failure: status == "failed" ? failure_for(execution.exception) : nil
+          failure: failure
         )
       end
 
+      # Errors RSpec reported outside examples, plus one when it stopped early
+      # (--fail-fast, an interrupt, a load error): then not every example ran.
+      def errors_outside_tests(notification)
+        world = ::RSpec.world
+        quitting = world.wants_to_quit || (world.respond_to?(:rspec_is_quitting) && world.rspec_is_quitting)
+        notification.errors_outside_of_examples_count + (quitting ? 1 : 0)
+      end
+
       def failure_for(exception)
         return nil unless exception
 

data/lib/raider_reporter/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RaiderReporter
-  VERSION = "0.7.0"
+  VERSION = "0.8.0"
 end

data/lib/raider_reporter/writers/html.rb

@@ -16,7 +16,7 @@ module RaiderReporter
       MARKER = '<meta name="generator" content="raider_reporter">'
       ROW_LIMIT = 2_000
       RUN_FILE = /\Arun(-[^\/]+)?\.json\z/
-      SECTIONS = [%w[failed Failed], %w[passed Passed], %w[pending Pending], %w[skipped Skipped]].freeze
+      SECTIONS = [%w[failed Failed], %w[quarantined Quarantined], %w[passed Passed], %w[pending Pending], %w[skipped Skipped]].freeze
       PROVIDERS = { "github_actions" => "GitHub Actions", "azure_pipelines" => "Azure Pipelines" }.freeze
 
       def self.ours?(path)
@@ -140,7 +140,11 @@ module RaiderReporter
         flaky = counts["flaky"]
         decided = passed + counts["failed"]
         rate = decided.zero? ? "—" : "#{(passed * 100.0 / decided).round}%"
-        parts = SECTIONS.map { |status, _| %(<span class="b #{status}">#{counts[status]} #{status}</span>) }
+        parts = SECTIONS.filter_map do |status, _|
+          next if status == "quarantined" && counts[status].zero?
+
+          %(<span class="b #{status}">#{counts[status]} #{status}</span>)
+        end
         parts << %(<span class="b flaky">#{flaky} flaky</span>) if flaky.positive?
         "#{parts.join(' ')} <span class=\"rate\">Pass rate #{rate}</span>"
       end
@@ -257,8 +261,8 @@ module RaiderReporter
       def h(value) = ERB::Util.html_escape(value.to_s)
 
       CSS = <<~CSS.gsub(/\s*\n\s*/, "")
-        :root{--bg:#fff;--fg:#1f2328;--mute:#656d76;--bdr:#d0d7de;--card:#f6f8fa;--ruby:#C14420;--ok:#1a7f37;--warn:#9a6700}
-        @media (prefers-color-scheme:dark){:root{--bg:#0d1117;--fg:#e6edf3;--mute:#8d96a0;--bdr:#30363d;--card:#161b22;--ok:#3fb950;--warn:#d29922}}
+        :root{--bg:#fff;--fg:#1f2328;--mute:#656d76;--bdr:#d0d7de;--card:#f6f8fa;--ruby:#C14420;--ok:#1a7f37;--warn:#9a6700;--q:#6B4FA0}
+        @media (prefers-color-scheme:dark){:root{--bg:#0d1117;--fg:#e6edf3;--mute:#8d96a0;--bdr:#30363d;--card:#161b22;--ok:#3fb950;--warn:#d29922;--q:#a78bfa}}
         *{box-sizing:border-box}
         body{margin:0;padding:16px;background:var(--bg);color:var(--fg);font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
         header,main,footer{max-width:1100px;margin:0 auto}
@@ -266,6 +270,7 @@ module RaiderReporter
         .meta,.l,footer,.more{color:var(--mute);font-size:12px}
         .b{display:inline-block;padding:0 6px;border-radius:9px;font-size:11px;font-weight:600;border:1px solid var(--bdr)}
         .b.failed{background:var(--ruby);color:#fff;border-color:var(--ruby)}
+        .b.quarantined{color:var(--q);border-color:var(--q)}
         .b.passed{color:var(--ok)}.b.pending,.b.skipped,.b.flaky{color:var(--warn)}
         .rate{font-weight:600;margin-left:6px}
         #q{width:100%;margin:12px 0;padding:8px;border:1px solid var(--bdr);border-radius:6px;background:var(--card);color:var(--fg)}

data/lib/raider_reporter.rb

@@ -8,6 +8,8 @@ require_relative "raider_reporter/models"
 require_relative "raider_reporter/app_frame"
 require_relative "raider_reporter/payload"
 require_relative "raider_reporter/configuration"
+require_relative "raider_reporter/quarantine"
+require_relative "raider_reporter/exec_summary"
 require_relative "raider_reporter/content_types"
 require_relative "raider_reporter/collector"
 require_relative "raider_reporter/writers/local"
@@ -40,6 +42,7 @@ module RaiderReporter
       @configuration = nil
       @collector = nil
       @project_root = nil
+      @quarantined_keys = nil
     end
 
     def start_collector!
@@ -47,10 +50,16 @@ module RaiderReporter
       @project_root = Dir.pwd
       @collector = Collector.new(
         max_attachment_bytes: configuration.max_attachment_bytes,
-        warn: ->(message) { warn(message) }
+        warn: ->(message) { warn(message) },
+        quarantine: -> { quarantined_keys }
       )
     end
 
+    # Fetched once per process, at the first run start (spec §4.2).
+    def quarantined_keys
+      @quarantined_keys ||= Quarantine.load(configuration, warn: ->(message) { warn(message) })
+    end
+
     def attach(name, path, type: nil)
       unless collector&.in_test?
         warn("attach(#{name.inspect}) called outside a running example; ignored")
@@ -82,6 +91,8 @@ module RaiderReporter
     def finish!(output = $stderr)
       return nil if dry_run?
 
+      write_exec_summary
+
       detector = CiDetector.new
       ci = detector.ci
       shard = ShardDetector.new(job: ci&.fetch("job", nil)).shard
@@ -108,6 +119,19 @@ module RaiderReporter
 
     private
 
+    # First, so a later reporting problem cannot lose it. A failure here only means
+    # no summary, and exec then keeps the command's exit code.
+    def write_exec_summary
+      path = configuration.exec_summary_path
+      counts = collector&.framework_counts
+      return if path.nil? || counts.nil?
+
+      ExecSummary.write(path, framework_failures: counts[:failures], quarantined: collector.quarantined_count,
+                              errors_outside_tests: counts[:errors_outside_tests])
+    rescue StandardError => e
+      warn("could not write the exec summary: #{e.class}: #{e.message}")
+    end
+
     def write_html_report
       return unless configuration.html_report
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: raider_reporter
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Augustin Gottlieb
@@ -36,9 +36,12 @@ files:
 - lib/raider_reporter/content_types.rb
 - lib/raider_reporter/cucumber.rb
 - lib/raider_reporter/cucumber/formatter.rb
+- lib/raider_reporter/exec.rb
+- lib/raider_reporter/exec_summary.rb
 - lib/raider_reporter/github_event.rb
 - lib/raider_reporter/models.rb
 - lib/raider_reporter/payload.rb
+- lib/raider_reporter/quarantine.rb
 - lib/raider_reporter/rspec.rb
 - lib/raider_reporter/rspec/formatter.rb
 - lib/raider_reporter/shard_detector.rb
@@ -68,7 +71,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.6
+rubygems_version: 4.0.21
 specification_version: 4
 summary: RSpec and Cucumber formatters that write Raider reports locally and upload
   them to Raider