raider_reporter 0.7.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6eb732095ea2284078a27ae18b268a3fe33a128e2ec131f95999bd278cd15cd2
-  data.tar.gz: 556f21195f523faabf7785eada01089a5537a7a4f052617152de87a0673c362b
+  metadata.gz: dda0a98d0886eefbfd851be91d57c881e74abee367956b0272fc9b7a464045cf
+  data.tar.gz: 8012de5d41e890b7dae144ab5f208c20ed2ffa94736645a4308901f6abe24174
 SHA512:
-  metadata.gz: ecf148c6450bdd03be6ed3634a89149ecd117d4b7e02cc65196d91c66cab050bedab950d3db7a2ebf9b66186cab381ff32d91e6783796f72bcb8a7d6a54877d9
-  data.tar.gz: 6dfe2cd5cb2ed735e124f580501c90ed9ef6ace13a21d7c37c7e8f8ed0bb25a6044f0ba248891b4c943898757f58bd077d7b57f9b05c0230cfce697d393f736b
+  metadata.gz: fa44978dfb6d93f9bbb316fe36d837b3ecb1ae668c7292b7f7e6ac345646b4c9099dd30375d8881e466f8f21eeb4ae6ae01d53e76652f975ffeaaa83b263939c
+  data.tar.gz: 6a43fa45d7a90997e64e95b9f5e40e978c398343695ce74950809f9cadbafd7420edce162c4560d01dd67892861b61180c5c7938afc2cf28a6191f1d1e18a54b

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 0.9.0 — 2026-10-24
+
+- `raider-reporter gate` waits for the Raider quality gate verdict of the CI build (`GET /api/v1/builds/gate`, every 10 seconds, up to `RAIDER_GATE_TIMEOUT` seconds, default 600, 1–3600) and exits 0 when the gate passed or the project has no gate rules, 1 when it failed (naming the failed rules), and 2 when it gets no verdict (timeout, Raider unreachable, token rejected, build not found, any other error). 429, 5xx and network errors are retried within the time limit
+- It posts a `raider/gate` status: a GitHub commit status on the PR head commit (`pull_request` events) or `GITHUB_SHA`, or an Azure Repos PR status; pending while it waits, then success, failure or error. Best effort: a post that fails prints a warning and never changes the exit code
+- `exec`, uploads and `comment` are unchanged and stay fail-open
+- Needs a Raider server with the gate endpoint
+
+## 0.8.0 — 2026-10-24
+
+- Quarantine: with `RAIDER_QUARANTINE=1` (or `config.quarantine = true`) and a token and endpoint, the formatters fetch the project's quarantine list once per run (`GET /api/v1/quarantine`, 3-second timeout) and report matching failures as `quarantined`, keeping their message, backtrace and attachments. Fail-open: any fetch problem prints `[raider_reporter] quarantine list unavailable (<reason>); no tests quarantined` and nothing is quarantined
+- `raider-reporter exec -- <command…>` runs the suite (no shell) and exits 0 when the command exited 1 and every failure the framework counted was quarantined, with no errors outside tests; it prints `raider-reporter: N quarantined failure(s) did not fail the build`. Otherwise it exits with the command's code (128 + signal when killed, 127 when the command cannot start, 2 without a command)
+- Runs with a quarantined result are written as schema 2; every other run stays schema 1, so older Raider servers keep accepting them
+- The HTML report shows quarantined results in their own section and color; `raider-reporter comment` reads `N quarantined` from the headline
+- RSpec and Cucumber exit codes are unchanged without `exec`
+
 ## 0.7.0 — 2026-10-18
 
 - Add a local HTML report: every run now also writes `raider-results/index.html`, one self-contained page (inline CSS and JS, no network requests) built from every `run*.json` in the folder, so shards from one machine share a report. Failures come first with their message, exception class, backtrace and screenshots; a filter box narrows by test name or file. Open it in a browser after a run, or publish it as a CI artifact

data/README.md

@@ -1,6 +1,6 @@
 # raider_reporter
 
-Raider-schema test reporting for RSpec and Cucumber. `raider_reporter` is an RSpec formatter and a Cucumber reporter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
+Raider-schema test reporting for RSpec and Cucumber. `raider_reporter` is an RSpec formatter and a Cucumber reporter that records every example's result, failure details, screenshots and videos, writes them to a local `run.json` (schema v1, or v2 when a run has a quarantined result), and can optionally upload the run to a Raider server. It replaces Allure for Ruby Raider projects.
 
 It has no runtime dependencies and requires Ruby 3.2 or newer.
 
@@ -66,7 +66,7 @@ cucumber --format pretty --format RaiderReporter::Cucumber::Formatter --out raid
 - `identifier` is `"<file_path>:<line>"`: the scenario line, or the Examples row line for outline rows.
 - `tags` are the feature, scenario and Examples tags, merged, sorted, unique, without `@`.
 
-Statuses: passed, failed and ambiguous (reported as failed), pending and undefined (reported as pending), skipped.
+Statuses: passed, failed and ambiguous (reported as failed), pending and undefined (reported as pending), skipped. Failed (and ambiguous) scenarios on the quarantine list are reported as quarantined.
 
 ### Failures
 
@@ -138,13 +138,14 @@ end
 | `output_dir` | `"raider-results"` | Directory for local output. |
 | `clean_output_dir` | `true` | Remove the previous `run.json` and `attachments/` directory (`run-<n>.json` and `attachments-<n>/` when sharded) inside `output_dir` before writing. Nothing else in `output_dir` is touched, except a raider_reporter-written `index.html`. |
 | `html_report` | `true` | Write `index.html`, the local HTML report, into `output_dir` after each run. See [Local report](#local-report). |
+| `quarantine` | `true` when `ENV["RAIDER_QUARANTINE"]` is `1` or `true` | Fetch the quarantine list at run start and report matching failures as `quarantined`. Needs `token` and `endpoint`. See [Quarantine](#quarantine). |
 | `environment` | `{}` | Free-form key/values stored with the run (browser, suite name, and so on). Values are JSON values: strings, booleans or numbers. |
 
 ## Local output
 
 ```
 raider-results/
-  run.json          # the run in schema v1
+  run.json          # the run in schema v1 (v2 with quarantined results)
   attachments/      # copies of attached screenshots and videos
   index.html        # the local report, see below
 ```
@@ -179,6 +180,24 @@ Reporting is fail-open: no error inside the gem, network failure, timeout or non
 
 The hosted Raider service is not public yet; uploads go to a Raider server whose address you provide in `RAIDER_ENDPOINT`.
 
+## Quarantine
+
+A test quarantined in Raider (with a reason and an expiry) still runs and is still reported, but its failure is reported as `quarantined`, and a CI build stays green when quarantined tests are its only failures.
+
+1. Turn it on in CI with `RAIDER_QUARANTINE=1` (or `config.quarantine = true`), next to `RAIDER_TOKEN` and `RAIDER_ENDPOINT`.
+2. Run the suite through `raider-reporter exec`:
+
+   ```
+   bundle exec raider-reporter exec -- bundle exec rspec
+   bundle exec raider-reporter exec -- bundle exec cucumber
+   ```
+
+When the run starts, the formatter fetches the project's quarantine list once (`GET /api/v1/quarantine`, 3-second timeout). On any error it prints `[raider_reporter] quarantine list unavailable (<reason>); no tests quarantined` and nothing is quarantined. Matching failures keep their message, backtrace and screenshots; the run is written as schema 2, and `run.json`, the upload and the HTML report show `quarantined`.
+
+`exec` runs the command without a shell and exits with its status, with one exception: when the command exits 1, every failure the framework counted is quarantined, and nothing went wrong outside tests, it prints `raider-reporter: N quarantined failure(s) did not fail the build` to stderr and exits 0. "Outside tests" means RSpec errors outside examples, a run stopped early (`--fail-fast`, an interrupt, a load error), or a Cucumber run that did not finish every scenario. RSpec `--fail-fast` counts even when the last example triggers it, and interrupts count too, so the build stays red. Under Cucumber `--strict`, a flaky scenario (failed, then passed on retry) counts as a real failure and keeps the build red. A command killed by a signal exits 128 + the signal number; a command that cannot be started exits 127. Several processes under one `exec` (for example `parallel_rspec`) never turn red into green: their summaries conflict and the command's exit code is used.
+
+Without `exec`, RSpec and Cucumber exit codes are unchanged; quarantine only changes how failures are reported.
+
 ## Pull request comments
 
 On pull requests, `raider-reporter comment` posts one comment with the build's results (and sets a `raider/tests` commit status on the PR head commit). Run it in a job that runs after your test jobs, so every shard has uploaded. Use the workflow's `GITHUB_TOKEN` (or a GitHub App token): a personal access token posts as a user, so the step adds a new comment on every run instead of updating the existing one.
@@ -209,6 +228,42 @@ On pull requests, `raider-reporter comment` posts one comment with the build's r
 - The command never fails your CI: it always exits 0 and prints notices to stderr, prefixed `[raider_reporter]`. It needs `GITHUB_TOKEN`, `GITHUB_REPOSITORY`, `GITHUB_EVENT_PATH`, `RAIDER_TOKEN` and `RAIDER_ENDPOINT`; if one is missing, or the event is not a pull request, it does nothing.
 - It requires a Raider server that supports the summary endpoint and the `git.pr` field. Upgrade the server before using it.
 
+## Quality gate
+
+A Raider project can turn on gate rules (minimum pass rate, no new failures against the default branch, no new flaky tests) under its Gate page. Raider computes one verdict per build once every shard is in, and never changes it. `raider-reporter gate` waits for that verdict in CI and fails the job when the gate fails. Run it once, after every test job:
+
+```yaml
+  raider-gate:
+    needs: [test]   # every test job id
+    if: always() && vars.RAIDER_GATE == '1'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      statuses: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with: { ruby-version: "3.4", bundler-cache: true }
+      - name: Wait for the Raider quality gate
+        run: bundle exec raider-reporter gate
+        env:
+          RAIDER_TOKEN: ${{ secrets.RAIDER_TOKEN }}
+          RAIDER_ENDPOINT: ${{ vars.RAIDER_ENDPOINT }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+
+| Outcome | Exit | Output |
+|---|---|---|
+| passed | 0 | stdout: `raider-reporter gate: passed` and one line per rule |
+| no gate rules | 0 | stdout: `raider-reporter gate: no gate rules for this project` |
+| failed | 1 | stderr: `raider-reporter gate: failed`, one line per failed rule (`  ✗ pass rate: 96% < 98%`) and the build URL |
+| no verdict | 2 | stderr: timeout, Raider unreachable, token rejected, build not found, or another error |
+
+- It polls every 10 seconds (10-second timeouts) for up to `RAIDER_GATE_TIMEOUT` seconds (default 600, 1–3600) and retries 429, 5xx and network errors. While waiting it prints, at most once a minute, `waiting for the verdict (build incomplete: 2 of 4 shards)` or `(analysing results)`.
+- It needs `RAIDER_TOKEN`, `RAIDER_ENDPOINT` and the CI build key that uploads and `comment` use (GitHub Actions or Azure Pipelines); without them it exits 2.
+- It posts a `raider/gate` status: pending while it waits, then success, failure (the failed rules) or error. On GitHub it needs `GITHUB_TOKEN` and `statuses: write`, and uses the PR head commit on `pull_request` events, else `GITHUB_SHA`. On Azure Repos pull requests it needs `SYSTEM_ACCESSTOKEN: $(System.AccessToken)`; other Azure builds get no status. A status that cannot be posted prints a warning and never changes the exit code. Require the `raider/gate` check in branch protection to block merges.
+- Unlike `comment`, `gate` fails closed. Turn it on per repository (the `RAIDER_GATE` variable above) so pull requests from forks and projects without a token are not blocked. The token is never printed.
+
 ## Azure Pipelines
 
 The gem detects Azure Pipelines when `TF_BUILD` is `true`. It reports the build (`provider: azure_pipelines`, build URL, attempt, job), the branch and commit, the pull request and a build key, so shards of one build group together. On a pull request from Azure Repos, `raider-reporter comment` posts one comment thread with the results and sets a `raider/tests` status on the PR.
@@ -319,7 +374,7 @@ Tip: run `rm -rf raider-results` before a parallel run. Each process only cleans
 
 ## Schema
 
-`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are exactly `passed`, `failed`, `pending` and `skipped`. In both RSpec and Cucumber, a failure may carry `app_frame` (when a project frame is found): the first backtrace line inside your project, for example `spec/pages/login_page.rb:42:in 'LoginPage#submit'`, which Raider uses to group failures with the same error.
+`run.json` follows schema v1, documented in [docs/schema-v1.md](docs/schema-v1.md). Statuses are `passed`, `failed`, `pending` and `skipped`, plus `quarantined` in schema 2 (written only when a run has a quarantined result). In both RSpec and Cucumber, a failure may carry `app_frame` (when a project frame is found): the first backtrace line inside your project, for example `spec/pages/login_page.rb:42:in 'LoginPage#submit'`, which Raider uses to group failures with the same error.
 
 ## Contributing
 

data/lib/raider_reporter/azure_env.rb

@@ -26,6 +26,16 @@ module RaiderReporter
       value(env, "BUILD_REASON") == "PullRequest" && value(env, "BUILD_REPOSITORY_PROVIDER") == "TfsGit"
     end
 
+    # The REST base of a pull request (threads, statuses), or nil when a part is unknown.
+    def pull_request_api_url(env, number)
+      base = collection_base(env)
+      project = value(env, "SYSTEM_TEAMPROJECTID")
+      repo = value(env, "BUILD_REPOSITORY_ID")
+      return nil unless base && project && repo && number
+
+      "#{base}#{project}/_apis/git/repositories/#{repo}/pullRequests/#{number}"
+    end
+
     def strip_heads(ref)
       return nil if ref.nil?
       return ref.delete_prefix("refs/heads/") if ref.start_with?("refs/heads/")

data/lib/raider_reporter/cli.rb

@@ -1,8 +1,12 @@
 # frozen_string_literal: true
 
 require "raider_reporter"
+require "raider_reporter/exec"
+require "raider_reporter/gate"
 
 module RaiderReporter
+  # `raider-reporter exec -- <command…>` runs the test suite and decides the build's exit code (see Exec).
+  # `raider-reporter gate` waits for the Raider quality gate verdict and exits 0, 1 or 2 (see Gate).
   # `raider-reporter comment` posts the build summary as a PR comment and sets a commit status
   # (fail-open: always exits 0, notices go to stderr). `raider-reporter report [dir]` rebuilds the
   # local HTML report (exits 1 when there are no run files or the report cannot be written).
@@ -22,8 +26,11 @@ module RaiderReporter
       "failed" => "Raider: tests failed",
       "incomplete" => "Raider: build incomplete"
     }.freeze
-    DEFAULT_API_URL = "https://api.github.com"
     USAGE = <<~TEXT
+      #{Exec::USAGE.chomp}
+
+      #{Gate::USAGE.chomp}
+
       Usage: raider-reporter report [dir]
 
       Rebuilds <dir>/index.html (default raider-results) from every run*.json in it. Prints its path.
@@ -48,6 +55,31 @@ module RaiderReporter
     end
 
     def run
+      return run_exec if @argv.first == "exec"
+      return run_gate if @argv.first == "gate"
+
+      run_reporting
+    end
+
+    private
+
+    # exec decides the build's exit code; Exec owns its own failure handling.
+    def run_exec
+      command = @argv[1] == "--" ? @argv.drop(2) : []
+      Exec.new(command, err: @err).run
+    end
+
+    # gate decides the CI job's exit code; Gate owns its own failure handling.
+    def run_gate
+      if @argv.include?("--help") || @argv.include?("-h")
+        @err.puts(Gate::USAGE)
+        return 0
+      end
+
+      Gate.new(env: @env, out: @out, err: @err).run
+    end
+
+    def run_reporting
       return report if @argv.first == "report" && !@argv.include?("--help") && !@argv.include?("-h")
 
       command = @argv.first
@@ -62,8 +94,6 @@ module RaiderReporter
       0
     end
 
-    private
-
     def report
       dir = File.expand_path(@argv[1] || "raider-results")
       path = Writers::Html.new.write(dir)
@@ -145,8 +175,7 @@ module RaiderReporter
     end
 
     def azure_pr_url(pr)
-      "#{AzureEnv.collection_base(@env)}#{AzureEnv.value(@env, 'SYSTEM_TEAMPROJECTID')}/_apis/git/repositories/" \
-        "#{AzureEnv.value(@env, 'BUILD_REPOSITORY_ID')}/pullRequests/#{pr['number']}"
+      AzureEnv.pull_request_api_url(@env, pr["number"])
     end
 
     def azure_forbidden_message
@@ -177,11 +206,10 @@ module RaiderReporter
     end
 
     def api_url
-      url = @env["GITHUB_API_URL"].to_s.strip
-      url.empty? ? DEFAULT_API_URL : url
+      Comment::GithubClient.api_url(@env)
     end
 
-    # "**Raider: failed** - 2 failed, 148 passed, 3 skipped (1m 12s)" -> "2 failed, 148 passed, 3 skipped".
+    # "**Raider: failed** - 2 failed, 1 quarantined, 148 passed (1m 12s)" -> "2 failed, 1 quarantined, 148 passed".
     # Falls back to a fixed text per state when the Markdown carries no headline counts.
     def description(summary)
       counts = headline_counts(summary.markdown)
@@ -193,7 +221,7 @@ module RaiderReporter
       line = markdown.to_s.lines.find { |l| l.include?("Raider:") }
       return nil unless line
 
-      counts = line.scan(/\b\d+ (?:failed|passed|skipped|pending)\b/)
+      counts = line.scan(/\b\d+ (?:failed|quarantined|passed|skipped|pending)\b/)
       counts.empty? ? nil : counts.join(", ")
     end
 

data/lib/raider_reporter/collector.rb

@@ -3,15 +3,19 @@
 require "securerandom"
 require "tmpdir"
 require "fileutils"
+require "set"
 
 module RaiderReporter
   # Accumulates one suite run. Framework formatters call it; writers read the Run it returns.
   class Collector
-    def initialize(max_attachment_bytes:, warn:, clock: -> { Time.now.utc }, uuid: -> { SecureRandom.uuid })
+    def initialize(max_attachment_bytes:, warn:, clock: -> { Time.now.utc }, uuid: -> { SecureRandom.uuid },
+                   quarantine: -> { Quarantine::EMPTY })
       @max_attachment_bytes = max_attachment_bytes
       @warn = warn
       @clock = clock
       @uuid = uuid
+      @quarantine = quarantine
+      @quarantined = nil
       @results = []
       @pending_attachments = nil
       @staged = 0
@@ -20,6 +24,26 @@ module RaiderReporter
     def start_run
       @started_at = @clock.call
       @results = []
+      @quarantined = load_quarantine
+    end
+
+    # True when the test's key is in the list loaded at run start.
+    def quarantined?(file_path, full_name)
+      return false if @quarantined.nil? || @quarantined.empty?
+
+      @quarantined.include?(RaiderReporter.test_key(file_path, full_name))
+    end
+
+    attr_reader :framework_counts
+
+    # What the framework itself counted (spec §4.4); nil until it reports, and
+    # then no exec summary is written.
+    def record_framework_counts(failures:, errors_outside_tests:)
+      @framework_counts = { failures: Integer(failures), errors_outside_tests: Integer(errors_outside_tests) }
+    end
+
+    def quarantined_count
+      @results.count { |result| result.status == "quarantined" }
     end
 
     def start_test
@@ -81,6 +105,13 @@ module RaiderReporter
 
     private
 
+    def load_quarantine
+      @quarantine.call
+    rescue StandardError => e
+      @warn.call(Quarantine.warning("#{e.class}: #{e.message}"))
+      Quarantine::EMPTY
+    end
+
     def keep?(attachment)
       path = attachment.source_path
       unless File.file?(path)

data/lib/raider_reporter/comment/azure_client.rb

@@ -38,9 +38,9 @@ module RaiderReporter
         end
       end
 
-      def set_status(state:, description:, target_url:)
+      def set_status(state:, description:, target_url:, name: "tests")
         request(:post, "/statuses", { state: state, description: description.to_s[0, DESCRIPTION_LIMIT],
-                                      targetUrl: target_url, context: { genre: "raider", name: "tests" } },
+                                      targetUrl: target_url, context: { genre: "raider", name: name } },
                 version: STATUS_API_VERSION)
         nil
       end

data/lib/raider_reporter/comment/github_client.rb

@@ -15,8 +15,15 @@ module RaiderReporter
       MAX_PAGES = 10
       PER_PAGE = 100
       DESCRIPTION_LIMIT = 140
+      DEFAULT_API_URL = "https://api.github.com"
 
-      def initialize(token:, repository:, api_url: "https://api.github.com", timeout: 30)
+      # GITHUB_API_URL from a CI env, falling back to github.com when unset or blank.
+      def self.api_url(env)
+        url = env["GITHUB_API_URL"].to_s.strip
+        url.empty? ? DEFAULT_API_URL : url
+      end
+
+      def initialize(token:, repository:, api_url: DEFAULT_API_URL, timeout: 30)
         @token = token
         @repository = repository
         @api_url = api_url.to_s.strip.sub(%r{/+\z}, "")

data/lib/raider_reporter/configuration.rb

@@ -4,7 +4,10 @@ require "raider_reporter/azure_env"
 
 module RaiderReporter
   class Configuration
-    attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :html_report, :environment, :max_attachment_bytes
+    TRUTHY = %w[1 true].freeze
+
+    attr_accessor :token, :endpoint, :output_dir, :clean_output_dir, :html_report, :environment, :max_attachment_bytes,
+                  :quarantine, :exec_summary_path
 
     def initialize(env = ENV)
       @token = AzureEnv.value(env, "RAIDER_TOKEN")
@@ -14,12 +17,19 @@ module RaiderReporter
       @html_report = true
       @environment = {}
       @max_attachment_bytes = 5 * 1024 * 1024
+      @quarantine = TRUTHY.include?(AzureEnv.value(env, "RAIDER_QUARANTINE").to_s.downcase)
+      @exec_summary_path = AzureEnv.value(env, "RAIDER_EXEC_SUMMARY")
     end
 
     def upload?
       present?(token) && present?(endpoint)
     end
 
+    # Quarantine needs the server: on, with both token and endpoint (spec §4.1).
+    def quarantine?
+      quarantine == true && upload?
+    end
+
     private
 
     def present?(value)

data/lib/raider_reporter/cucumber/formatter.rb

@@ -84,6 +84,10 @@ module RaiderReporter
         @buffer = {}
         @attempts = Hash.new(0)
         @attempt = nil
+        @outcomes = {}
+        @failed_before = {}
+        @outcomes_broken = false
+        @expected_cases = nil
         guard { setup(config) if self.class.claim(config) }
       end
 
@@ -95,6 +99,8 @@ module RaiderReporter
         @collector = RaiderReporter.start_collector!
         RaiderReporter.dry_run = config.dry_run?
         config.on_event(:test_run_started) { |_event| guard { @collector.start_run } }
+        config.on_event(:test_run_started) { |event| guard { expect_cases(event.test_cases) } }
+        config.on_event(:test_case_finished) { |event| count_outcome(event.test_case, event.result) }
         config.on_event(:test_case_started) { |event| guard { case_started(event.test_case) } }
         config.on_event(:test_step_finished) { |event| guard { step_finished(event.test_step, event.result) } }
         config.on_event(:test_case_finished) { |event| guard { case_finished(event.test_case, event.result) } }
@@ -124,6 +130,7 @@ module RaiderReporter
         # Always buffer: a lookup problem may lose detail, never the scenario.
         failure = status == "failed" ? safely(nil) { failure_for(problem, result, test_case) } : nil
         fields = identity_for(test_case)
+        status = "quarantined" if status == "failed" && quarantined?(fields)
         key = "#{test_case.location}|#{test_case.name}"
         @attempts[key] += 1
         # The newest attempt replaces the older one, taking its attachments with it.
@@ -135,11 +142,42 @@ module RaiderReporter
       def run_finished
         @buffer.each_value { |result| @collector.add_result(result) }
         @buffer.clear
+        safely(nil) { record_framework_counts }
         RaiderReporter.finish!($stderr)
       ensure
         self.class.release(@config)
       end
 
+      def expect_cases(test_cases)
+        @expected_cases = test_cases.map { |test_case| "#{test_case.location}|#{test_case.name}" }.uniq.size
+      end
+
+      # Its own handler, registered before case_finished: if counting ever fails, no
+      # summary is written and exec keeps cucumber's exit code. The newest attempt wins,
+      # except that under strict flaky (--strict, --strict-flaky) a scenario that failed
+      # on an earlier attempt and then passed still fails cucumber's run.
+      def count_outcome(test_case, result)
+        return if @outcomes_broken
+
+        key = "#{test_case.location}|#{test_case.name}"
+        failed = !result.ok?(strict: @config.strict)
+        flaky = !failed && @failed_before[key] && @config.strict.strict?(:flaky)
+        @failed_before[key] ||= failed
+        @outcomes[key] = failed || flaky ? true : false
+      rescue StandardError => e
+        @outcomes_broken = true
+        RaiderReporter.warn("#{e.class}: #{e.message}")
+      end
+
+      # Scenarios that make cucumber's run fail; one error outside tests when the run
+      # stopped before every scenario finished (--fail-fast) (spec §4.4).
+      def record_framework_counts
+        return if @outcomes_broken || @expected_cases.nil?
+
+        aborted = @outcomes.size < @expected_cases ? 1 : 0
+        @collector.record_framework_counts(failures: @outcomes.values.count(true), errors_outside_tests: aborted)
+      end
+
       def identity_for(test_case)
         file_path = relative_path(test_case.location.file)
         line_number = test_case.location.lines.max
@@ -152,6 +190,11 @@ module RaiderReporter
         }
       end
 
+      # A lookup problem must never turn a real failure into a quarantined one.
+      def quarantined?(fields)
+        safely(false) { @collector.quarantined?(fields[:file_path], fields[:full_name]) }
+      end
+
       def full_name(test_case)
         safely(test_case.name) do
           feature = @ast.gherkin_document(test_case.location.file)&.feature

data/lib/raider_reporter/exec.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+require "tmpdir"
+require "raider_reporter/exec_summary"
+
+module RaiderReporter
+  # `raider-reporter exec -- <command…>` (spec §4.5). Runs the command without a
+  # shell, streams its output, and keeps its exit code, except that exit 1 becomes 0
+  # when the summary says every counted failure was quarantined and nothing else
+  # went wrong. Anything uncertain keeps the command's code.
+  class Exec
+    USAGE = <<~TEXT
+      Usage: raider-reporter exec -- <command> [args...]
+
+      Runs the command (no shell) and exits with its status. When the command exits 1, every failure it
+      counted was quarantined in Raider and nothing else went wrong, exits 0 instead and says so on stderr.
+      Quarantine needs RAIDER_QUARANTINE=1 with RAIDER_TOKEN and RAIDER_ENDPOINT.
+    TEXT
+    USAGE_ERROR = 2
+    NOT_STARTED = 127
+    # A terminal Ctrl-C already reaches the command through the process group.
+    FORWARDED = %w[TERM HUP].freeze
+    IGNORED = %w[INT].freeze
+
+    def self.quarantined_only?(summary)
+      return false unless summary
+
+      failures = summary["framework_failures"]
+      failures.positive? && failures == summary["quarantined"] && summary["errors_outside_tests"].zero?
+    end
+
+    def initialize(command, err: $stderr)
+      @command = command
+      @err = err
+      @pid = nil
+      @code = nil
+      @pending = nil
+    end
+
+    def run
+      if @command.empty?
+        @err.puts(USAGE)
+        return USAGE_ERROR
+      end
+
+      Dir.mktmpdir("raider-exec") { |dir| run_with(File.join(dir, "summary.json")) }
+    rescue StandardError => e
+      @err.puts("[raider_reporter] exec failed: #{e.class}: #{e.message}")
+      @code || 1
+    end
+
+    private
+
+    def run_with(summary_path)
+      previous = install_traps
+      @pid = start(summary_path)
+      return NOT_STARTED unless @pid
+
+      forward(@pending) if @pending
+
+      @code = wait
+      summary = ExecSummary.read(summary_path)
+      return @code unless @code == 1 && self.class.quarantined_only?(summary)
+
+      @err.puts("raider-reporter: #{summary['quarantined']} quarantined failure(s) did not fail the build")
+      0
+    ensure
+      previous&.each { |signal, handler| trap(signal, handler || "DEFAULT") }
+    end
+
+    def start(summary_path)
+      program, *args = @command
+      Process.spawn({ ExecSummary::ENV_NAME => summary_path }, [program, program], *args)
+    rescue SystemCallError => e
+      @err.puts("[raider_reporter] exec: cannot run #{program}: #{e.message}")
+      nil
+    end
+
+    def wait
+      _, status = Process.wait2(@pid)
+      status.signaled? ? 128 + status.termsig : status.exitstatus
+    end
+
+    def forward(signal)
+      Process.kill(signal, @pid)
+    rescue SystemCallError
+      nil
+    end
+
+    # Installed before spawning; a signal that arrives before the child exists is kept in
+    # @pending and forwarded right after spawn. A Ruby handler (not "IGNORE") is reset to
+    # the default in the child by exec(2).
+    def install_traps
+      forwarded = FORWARDED.to_h do |signal|
+        handler = trap(signal) { @pid ? forward(signal) : @pending = signal }
+        [signal, handler]
+      end
+      forwarded.merge(IGNORED.to_h { |signal| [signal, trap(signal) {}] })
+    end
+  end
+end

data/lib/raider_reporter/exec_summary.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+require "json"
+
+module RaiderReporter
+  # The file `raider-reporter exec` reads to decide its exit code (spec §4.4). One
+  # process writes it. A second writer (several processes under one exec) turns it
+  # into a conflict, which exec treats as no summary, so the command's code wins.
+  module ExecSummary
+    ENV_NAME = "RAIDER_EXEC_SUMMARY"
+    KEYS = %w[framework_failures quarantined errors_outside_tests].freeze
+    CONFLICT = { "conflict" => true }.freeze
+
+    module_function
+
+    def write(path, framework_failures:, quarantined:, errors_outside_tests:, pid: Process.pid)
+      summary = { "framework_failures" => framework_failures, "quarantined" => quarantined,
+                  "errors_outside_tests" => errors_outside_tests, "pid" => pid }
+      File.open(path, File::RDWR | File::CREAT, 0o600) do |file|
+        file.flock(File::LOCK_EX)
+        existing = file.read
+        same_writer = existing.empty? || parse(existing)&.fetch("pid", nil) == pid
+        file.rewind
+        file.truncate(0)
+        file.write(JSON.generate(same_writer ? summary : CONFLICT))
+      end
+    end
+
+    def read(path)
+      data = parse(File.read(path))
+      return nil if data.nil? || data.key?("conflict")
+      return nil unless KEYS.all? { |key| data[key].is_a?(Integer) && !data[key].negative? }
+
+      data.slice(*KEYS)
+    rescue SystemCallError, IOError
+      nil
+    end
+
+    def parse(text)
+      data = JSON.parse(text)
+      data.is_a?(Hash) ? data : nil
+    rescue JSON::ParserError
+      nil
+    end
+  end
+end

data/lib/raider_reporter/gate.rb

@@ -0,0 +1,193 @@
+# frozen_string_literal: true
+
+require "raider_reporter/azure_env"
+require "raider_reporter/ci_detector"
+require "raider_reporter/gate_client"
+require "raider_reporter/gate_status"
+
+module RaiderReporter
+  # `raider-reporter gate` (spec §5): waits for the Raider server's frozen quality gate
+  # verdict of this CI build. Exits 0 when it passed or the project has no gate rules,
+  # 1 when it failed, and 2 for every way it gets no verdict. Unlike `exec` and
+  # `comment`, it fails closed. The tokens are never printed.
+  class Gate
+    PREFIX = "raider-reporter gate:"
+    PASSED = 0
+    FAILED = 1
+    NO_VERDICT = 2
+    DEFAULT_TIMEOUT = 600
+    TIMEOUT_RANGE = (1..3600)
+    INTERVAL = 10
+    PROGRESS_EVERY = 60
+    LABELS = { "pass_rate" => "pass rate", "new_failures" => "new failures", "new_flaky" => "new flaky" }.freeze
+    MARKS = { "passed" => "✓", "failed" => "✗", "skipped" => "–" }.freeze
+    USAGE = <<~TEXT
+      Usage: raider-reporter gate
+
+      Waits for the Raider quality gate verdict of this CI build and posts a "raider/gate" status.
+      Exits 0 when the gate passed or the project has no gate rules, 1 when it failed, and 2 when no
+      verdict arrived (timeout, Raider unreachable, token rejected, build not found).
+        Environment: RAIDER_TOKEN, RAIDER_ENDPOINT, RAIDER_GATE_TIMEOUT (seconds, 1-3600, default 600).
+        Status: GITHUB_TOKEN (GitHub Actions) or SYSTEM_ACCESSTOKEN (Azure Repos pull requests).
+    TEXT
+
+    def initialize(env: ENV, out: $stdout, err: $stderr, clock: nil, sleeper: nil)
+      @env = env
+      @out = out
+      @err = err
+      @clock = clock || -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }
+      @sleeper = sleeper || ->(seconds) { sleep(seconds) }
+      @status = nil
+      @url = nil
+      @detail = nil
+      @last_progress = nil
+    end
+
+    def run
+      build_key = CiDetector.new(env: @env, git: ->(*) {}).ci&.dig("build_key")
+      return refuse("no CI build key (run this in GitHub Actions or Azure Pipelines)") unless build_key
+
+      token = AzureEnv.value(@env, "RAIDER_TOKEN")
+      endpoint = AzureEnv.value(@env, "RAIDER_ENDPOINT")
+      return refuse("RAIDER_TOKEN and RAIDER_ENDPOINT are required") unless token && endpoint
+
+      timeout = parse_timeout
+      return refuse("RAIDER_GATE_TIMEOUT must be a whole number of seconds from 1 to 3600") unless timeout
+
+      @status = GateStatus.new(env: @env, warn: ->(message) { say(@err, "#{PREFIX} #{message}") })
+      @status.post(:pending, "Waiting for the Raider verdict")
+      finish(poll(GateClient.new(endpoint: endpoint, token: token), build_key, timeout), timeout)
+    rescue StandardError => e
+      no_verdict("#{e.class}: #{e.message}")
+    end
+
+    private
+
+    def parse_timeout
+      raw = AzureEnv.value(@env, "RAIDER_GATE_TIMEOUT")
+      return DEFAULT_TIMEOUT if raw.nil?
+
+      value = Integer(raw, 10, exception: false)
+      value if value && TIMEOUT_RANGE.cover?(value)
+    end
+
+    # The last response that ends the wait, or nil when the time limit ran out.
+    def poll(client, build_key, timeout)
+      deadline = @clock.call + timeout
+      loop do
+        response = client.fetch(build_key)
+        return response unless response.kind == :retry || pending?(response)
+
+        note(response)
+        remaining = deadline - @clock.call
+        return nil if remaining <= 0
+
+        @sleeper.call([INTERVAL, remaining].min)
+      end
+    end
+
+    def pending?(response)
+      response.kind == :verdict && response.data["state"] == "pending"
+    end
+
+    def note(response)
+      if response.kind == :retry
+        @detail = response.reason
+        return
+      end
+
+      remember_url(response.data)
+      @detail = pending_detail(response.data)
+      now = @clock.call
+      return if @last_progress && now - @last_progress < PROGRESS_EVERY
+
+      @last_progress = now
+      say(@out, "#{PREFIX} waiting for the verdict (#{@detail})")
+    end
+
+    def pending_detail(data)
+      shards = data["shards"].is_a?(Hash) ? data["shards"] : {}
+      received = shards["received"]
+      expected = shards["expected"]
+      if received.is_a?(Integer) && expected.is_a?(Integer) && received < expected
+        "build incomplete: #{received} of #{expected} shards"
+      else
+        "analysing results"
+      end
+    end
+
+    def finish(response, timeout)
+      return no_verdict("no verdict after #{timeout}s (#{@detail || 'no response'})") unless response
+
+      case response.kind
+      when :rejected then no_verdict("token rejected (HTTP #{response.code})")
+      when :not_found then no_verdict("build not found (did the tests upload?)")
+      when :error then no_verdict(response.reason)
+      else report(response.data)
+      end
+    end
+
+    def report(data)
+      remember_url(data)
+      rules = data["rules"].select { |rule| rule.is_a?(Hash) }
+      case data["state"]
+      when "none"
+        say(@out, "#{PREFIX} no gate rules for this project")
+        @status.post(:success, "No gate rules", target_url: @url)
+        PASSED
+      when "passed"
+        say(@out, "#{PREFIX} passed")
+        rules.each { |rule| say(@out, rule_line(rule)) }
+        @status.post(:success, "Gate passed", target_url: @url)
+        PASSED
+      else
+        report_failed(rules.select { |rule| rule["outcome"] == "failed" })
+      end
+    end
+
+    def report_failed(failed)
+      say(@err, "#{PREFIX} failed")
+      failed.each { |rule| say(@err, rule_line(rule)) }
+      say(@err, "  #{@url}") if @url
+      description = failed.map { |rule| rule_text(rule) }.join("; ")
+      @status.post(:failure, description.empty? ? "Gate failed" : description, target_url: @url)
+      FAILED
+    end
+
+    def rule_line(rule)
+      "  #{MARKS.fetch(rule['outcome'].to_s, '?')} #{rule_text(rule)}"
+    end
+
+    def rule_text(rule)
+      name = rule["rule"].to_s
+      "#{LABELS.fetch(name, name)}: #{rule['detail']}"
+    end
+
+    def remember_url(data)
+      @url = data["url"] if data["url"].is_a?(String) && !data["url"].empty?
+    end
+
+    # Exit 2 before polling: nothing was posted, so there is no status to clear.
+    def refuse(message)
+      say(@err, "#{PREFIX} #{message}")
+      NO_VERDICT
+    end
+
+    # Exit 2 after the pending status: replace it, so a required check is never left pending.
+    def no_verdict(message)
+      say(@err, "#{PREFIX} #{message}")
+      @status&.post(:error, "No verdict", target_url: @url)
+      NO_VERDICT
+    end
+
+    def say(io, text)
+      io.puts(redact(text))
+    end
+
+    def redact(text)
+      secrets = [AzureEnv.value(@env, "RAIDER_TOKEN"), AzureEnv.value(@env, "GITHUB_TOKEN"),
+                 AzureEnv.value(@env, "SYSTEM_ACCESSTOKEN")].compact
+      secrets.reduce(text.to_s) { |memo, secret| memo.gsub(secret, "[REDACTED]") }
+    end
+  end
+end

data/lib/raider_reporter/gate_client.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+require "cgi/escape"
+require "json"
+require "net/http"
+require "openssl"
+require "timeout"
+require "uri"
+require "raider_reporter/version"
+
+module RaiderReporter
+  # One GET /api/v1/builds/gate call for `raider-reporter gate` (spec §5), classified
+  # so the caller can decide: a verdict, retry (429, 5xx, network), or stop.
+  class GateClient
+    class InvalidResponse < StandardError; end
+
+    PATH = "/api/v1/builds/gate"
+    TIMEOUT = 10
+    STATES = %w[pending none passed failed].freeze
+    NETWORK_ERRORS = [Timeout::Error, SystemCallError, SocketError, IOError, OpenSSL::SSL::SSLError,
+                      Net::ProtocolError, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError].freeze
+
+    # kind: :verdict, :retry, :rejected, :not_found or :error.
+    Response = Data.define(:kind, :data, :code, :reason)
+
+    def initialize(endpoint:, token:)
+      @endpoint = endpoint
+      @token = token
+    end
+
+    def fetch(build_key)
+      classify(request(build_key))
+    rescue *NETWORK_ERRORS => e
+      Response.new(kind: :retry, data: nil, code: nil, reason: redact("#{e.class}: #{e.message}"))
+    end
+
+    private
+
+    def request(build_key)
+      uri = URI.parse("#{@endpoint.to_s.strip.chomp('/')}#{PATH}?build_key=#{CGI.escape(build_key.to_s)}")
+      raise ArgumentError, "RAIDER_ENDPOINT must start with http:// or https://" unless uri.is_a?(URI::HTTP)
+
+      req = Net::HTTP::Get.new(uri)
+      req["Authorization"] = "Bearer #{@token}"
+      req["Accept"] = "application/json"
+      req["User-Agent"] = "raider_reporter/#{VERSION}"
+      Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", open_timeout: TIMEOUT,
+                                          read_timeout: TIMEOUT, max_retries: 0) { |http| http.request(req) }
+    end
+
+    def classify(response)
+      code = response.code.to_i
+      case code
+      when 200 then parse(response.body)
+      when 401, 403 then Response.new(kind: :rejected, data: nil, code: code, reason: nil)
+      when 404 then Response.new(kind: :not_found, data: nil, code: code, reason: nil)
+      when 429, 500..599 then Response.new(kind: :retry, data: nil, code: code, reason: "HTTP #{code}")
+      else error_response(code, "unexpected HTTP #{code}")
+      end
+    end
+
+    def parse(body)
+      data = JSON.parse(body.to_s)
+      return error_response(200, "invalid gate response") unless valid?(data)
+
+      Response.new(kind: :verdict, data: data, code: 200, reason: nil)
+    rescue JSON::ParserError => e
+      Response.new(kind: :error, data: nil, code: 200, reason: redact("#{e.class}: #{e.message}"))
+    end
+
+    def valid?(data)
+      data.is_a?(Hash) && STATES.include?(data["state"]) && data["rules"].is_a?(Array)
+    end
+
+    # Spec §5 "other error" form: <class>: <message>.
+    def error_response(code, message)
+      Response.new(kind: :error, data: nil, code: code, reason: "#{InvalidResponse}: #{message}")
+    end
+
+    def redact(text)
+      @token.to_s.empty? ? text : text.gsub(@token.to_s, "[REDACTED]")
+    end
+  end
+end

data/lib/raider_reporter/gate_status.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+require "raider_reporter/version"
+require "raider_reporter/azure_env"
+require "raider_reporter/ci_detector"
+require "raider_reporter/comment/github_client"
+require "raider_reporter/comment/azure_client"
+
+module RaiderReporter
+  # Posts the "raider/gate" status for `raider-reporter gate` (spec §5): a GitHub commit
+  # status (PR head SHA on pull_request events, else GITHUB_SHA) or an Azure Repos PR
+  # status (genre raider, name gate). Best effort: a failed post warns and never raises.
+  class GateStatus
+    CONTEXT = "raider/gate"
+    AZURE_NAME = "gate"
+    TIMEOUT = 10
+    GITHUB_STATES = { pending: "pending", success: "success", failure: "failure", error: "error" }.freeze
+    AZURE_STATES = { pending: "pending", success: "succeeded", failure: "failed", error: "error" }.freeze
+
+    def initialize(env:, warn:)
+      @env = env
+      @warn = warn
+      @provider = begin
+        detect
+      rescue StandardError
+        nil
+      end
+    end
+
+    def active?
+      !@provider.nil?
+    end
+
+    # state: :pending, :success, :failure or :error.
+    def post(state, description, target_url: nil)
+      case @provider
+      when :github
+        github_client.set_status(@sha, state: GITHUB_STATES.fetch(state), target_url: target_url,
+                                       description: description, context: CONTEXT)
+      when :azure
+        azure_client.set_status(state: AZURE_STATES.fetch(state), description: description, target_url: target_url,
+                                name: AZURE_NAME)
+      end
+      nil
+    rescue StandardError => e
+      @warn.call("could not post the #{CONTEXT} status (#{e.class}: #{e.message})")
+      nil
+    end
+
+    private
+
+    def detect
+      detector = CiDetector.new(env: @env, git: ->(*) {})
+      if @env["GITHUB_ACTIONS"] == "true"
+        @sha = detector.git&.dig("commit_sha")
+        :github if [@env["GITHUB_TOKEN"], @env["GITHUB_REPOSITORY"], @sha].none? { |value| value.to_s.strip.empty? }
+      elsif AzureEnv.azure?(@env)
+        pr = detector.git&.dig("pr")
+        @azure_url = pr && AzureEnv.pull_request_api_url(@env, pr["number"])
+        :azure if @azure_url && AzureEnv.value(@env, "SYSTEM_ACCESSTOKEN")
+      end
+    end
+
+    def github_client
+      Comment::GithubClient.new(token: @env["GITHUB_TOKEN"], repository: @env["GITHUB_REPOSITORY"],
+                                api_url: Comment::GithubClient.api_url(@env), timeout: TIMEOUT)
+    end
+
+    def azure_client
+      Comment::AzureClient.new(token: AzureEnv.value(@env, "SYSTEM_ACCESSTOKEN"), base_url: @azure_url, timeout: TIMEOUT)
+    end
+  end
+end

data/lib/raider_reporter/models.rb

@@ -1,8 +1,8 @@
 # frozen_string_literal: true
 
 module RaiderReporter
-  SCHEMA_VERSION = 1
-  STATUSES = %w[passed failed pending skipped].freeze
+  SCHEMA_VERSION = 2
+  STATUSES = %w[passed failed pending skipped quarantined].freeze
 
   Failure = Data.define(:message, :exception_class, :backtrace, :app_frame) do
     def initialize(message:, exception_class:, backtrace:, app_frame: nil)

data/lib/raider_reporter/payload.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module RaiderReporter
-  # Serializes a Run into schema v1. Local mode points attachments at copied
+  # Serializes a Run: schema 1, or schema 2 when a result is quarantined. Local mode points attachments at copied
   # files; upload mode points them at multipart part names.
   class Payload
     MODES = %i[local upload].freeze
@@ -17,7 +17,7 @@ module RaiderReporter
 
       entries = attachment_entries.each
       {
-        "schema_version" => SCHEMA_VERSION,
+        "schema_version" => schema_version,
         "client_run_id" => @run.client_run_id,
         "run" => run_hash,
         "results" => @run.results.map do |result|
@@ -34,6 +34,10 @@ module RaiderReporter
 
     def run_file = shard_index ? "run-#{shard_index}.json" : "run.json"
 
+    # Version 2 only adds the quarantined status; older servers keep accepting
+    # every run that has none (spec §4.3).
+    def schema_version = @run.results.any? { |result| result.status == "quarantined" } ? SCHEMA_VERSION : 1
+
     def attachment_entries
       @attachment_entries ||= @run.results.flat_map(&:attachments).each_with_index.map do |attachment, index|
         {

data/lib/raider_reporter/quarantine.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+require "net/http"
+require "set"
+require "timeout"
+require "uri"
+
+module RaiderReporter
+  NUL = "\u0000"
+  NUL_REPLACEMENT = "\u{FFFD}"
+
+  # The server's test identity: raider_saas TestCase.key_for over the payload's
+  # file_path and full_name, after ingest replaced NUL in each of them.
+  def self.test_key(file_path, full_name)
+    parts = [file_path, full_name].map { |part| part.to_s.tr(NUL, NUL_REPLACEMENT) }
+    Digest::SHA256.hexdigest(parts.join(NUL))
+  end
+
+  # Fetches the project's quarantine list (spec §4.2). Fail-open: any problem
+  # means one warning and an empty set, never an exception.
+  module Quarantine
+    PATH = "/api/v1/quarantine"
+    TIMEOUT = 3
+    EMPTY = Set.new.freeze
+
+    class Unavailable < StandardError; end
+
+    def self.load(config, warn:)
+      return EMPTY unless config.quarantine?
+
+      Set.new(fetch(config)).freeze
+    rescue StandardError => e
+      reason = e.is_a?(Unavailable) ? e.message : "#{e.class}: #{e.message}"
+      warn.call(warning(reason, token: config.token))
+      EMPTY
+    end
+
+    # The one place the fail-open warning text is built (used by load and the Collector).
+    def self.warning(reason, token: nil)
+      "quarantine list unavailable (#{redact(reason, token)}); no tests quarantined"
+    end
+
+    def self.fetch(config)
+      uri = URI.parse("#{config.endpoint.to_s.strip.chomp('/')}#{PATH}")
+      raise Unavailable, "RAIDER_ENDPOINT must start with http:// or https://" unless uri.is_a?(URI::HTTP)
+
+      request = Net::HTTP::Get.new(uri)
+      request["Authorization"] = "Bearer #{config.token}"
+      request["Accept"] = "application/json"
+      request["User-Agent"] = "raider_reporter/#{VERSION}"
+      # The per-step timeouts do not bound a server that trickles bytes; the deadline does.
+      response = Timeout.timeout(TIMEOUT) do
+        Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
+                                            open_timeout: TIMEOUT, read_timeout: TIMEOUT, max_retries: 0) { |http| http.request(request) }
+      end
+      raise Unavailable, "HTTP #{response.code}" unless response.code == "200"
+
+      keys(response.body)
+    end
+
+    def self.keys(body)
+      data = JSON.parse(body.to_s)
+      list = data.is_a?(Hash) ? data["quarantined"] : nil
+      raise Unavailable, "unexpected response" unless list.is_a?(Array)
+
+      list.filter_map { |entry| entry["test_key"] if entry.is_a?(Hash) && entry["test_key"].is_a?(String) }
+    rescue JSON::ParserError
+      raise Unavailable, "invalid JSON"
+    end
+
+    def self.redact(text, token)
+      token.to_s.empty? ? text : text.gsub(token.to_s, "[REDACTED]")
+    end
+
+    private_class_method :fetch, :keys, :redact
+  end
+end

data/lib/raider_reporter/rspec/formatter.rb

@@ -11,7 +11,7 @@ module RaiderReporter
     class Formatter
       ::RSpec::Core::Formatters.register self,
                                          :start, :example_started, :example_passed,
-                                         :example_failed, :example_pending, :close
+                                         :example_failed, :example_pending, :dump_summary, :close
 
       NON_TAG_KEYS = %i[pending skip].freeze
       MAX_MESSAGE = Text::MAX_MESSAGE
@@ -45,6 +45,13 @@ module RaiderReporter
         end
       end
 
+      def dump_summary(notification)
+        guard do
+          @collector.record_framework_counts(failures: notification.failure_count,
+                                             errors_outside_tests: errors_outside_tests(notification))
+        end
+      end
+
       def close(_notification)
         guard { RaiderReporter.finish! }
       end
@@ -54,19 +61,32 @@ module RaiderReporter
       def record(example, status)
         metadata = example.metadata
         execution = example.execution_result
+        file_path = metadata[:file_path].to_s.delete_prefix("./")
+        full_name = utf8(example.full_description)
+        failure = status == "failed" ? failure_for(execution.exception) : nil
+        # The key is computed from the payload's own file_path and full_name (spec §4.2).
+        status = "quarantined" if status == "failed" && @collector.quarantined?(file_path, full_name)
         @collector.finish_test(
           identifier: example.id,
-          full_name: utf8(example.full_description),
-          file_path: metadata[:file_path].to_s.delete_prefix("./"),
+          full_name: full_name,
+          file_path: file_path,
           line_number: metadata[:line_number],
           status: status,
           duration_ms: ((execution.run_time || 0) * 1000).round,
           attempts: metadata.fetch(:retry_attempts, 0).to_i + 1,
           tags: metadata.select { |key, value| value == true && !NON_TAG_KEYS.include?(key) }.keys.map(&:to_s).sort,
-          failure: status == "failed" ? failure_for(execution.exception) : nil
+          failure: failure
         )
       end
 
+      # Errors RSpec reported outside examples, plus one when it stopped early
+      # (--fail-fast, an interrupt, a load error): then not every example ran.
+      def errors_outside_tests(notification)
+        world = ::RSpec.world
+        quitting = world.wants_to_quit || (world.respond_to?(:rspec_is_quitting) && world.rspec_is_quitting)
+        notification.errors_outside_of_examples_count + (quitting ? 1 : 0)
+      end
+
       def failure_for(exception)
         return nil unless exception
 

data/lib/raider_reporter/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RaiderReporter
-  VERSION = "0.7.0"
+  VERSION = "0.9.0"
 end

data/lib/raider_reporter/writers/html.rb

@@ -16,7 +16,7 @@ module RaiderReporter
       MARKER = '<meta name="generator" content="raider_reporter">'
       ROW_LIMIT = 2_000
       RUN_FILE = /\Arun(-[^\/]+)?\.json\z/
-      SECTIONS = [%w[failed Failed], %w[passed Passed], %w[pending Pending], %w[skipped Skipped]].freeze
+      SECTIONS = [%w[failed Failed], %w[quarantined Quarantined], %w[passed Passed], %w[pending Pending], %w[skipped Skipped]].freeze
       PROVIDERS = { "github_actions" => "GitHub Actions", "azure_pipelines" => "Azure Pipelines" }.freeze
 
       def self.ours?(path)
@@ -140,7 +140,11 @@ module RaiderReporter
         flaky = counts["flaky"]
         decided = passed + counts["failed"]
         rate = decided.zero? ? "—" : "#{(passed * 100.0 / decided).round}%"
-        parts = SECTIONS.map { |status, _| %(<span class="b #{status}">#{counts[status]} #{status}</span>) }
+        parts = SECTIONS.filter_map do |status, _|
+          next if status == "quarantined" && counts[status].zero?
+
+          %(<span class="b #{status}">#{counts[status]} #{status}</span>)
+        end
         parts << %(<span class="b flaky">#{flaky} flaky</span>) if flaky.positive?
         "#{parts.join(' ')} <span class=\"rate\">Pass rate #{rate}</span>"
       end
@@ -257,8 +261,8 @@ module RaiderReporter
       def h(value) = ERB::Util.html_escape(value.to_s)
 
       CSS = <<~CSS.gsub(/\s*\n\s*/, "")
-        :root{--bg:#fff;--fg:#1f2328;--mute:#656d76;--bdr:#d0d7de;--card:#f6f8fa;--ruby:#C14420;--ok:#1a7f37;--warn:#9a6700}
-        @media (prefers-color-scheme:dark){:root{--bg:#0d1117;--fg:#e6edf3;--mute:#8d96a0;--bdr:#30363d;--card:#161b22;--ok:#3fb950;--warn:#d29922}}
+        :root{--bg:#fff;--fg:#1f2328;--mute:#656d76;--bdr:#d0d7de;--card:#f6f8fa;--ruby:#C14420;--ok:#1a7f37;--warn:#9a6700;--q:#6B4FA0}
+        @media (prefers-color-scheme:dark){:root{--bg:#0d1117;--fg:#e6edf3;--mute:#8d96a0;--bdr:#30363d;--card:#161b22;--ok:#3fb950;--warn:#d29922;--q:#a78bfa}}
         *{box-sizing:border-box}
         body{margin:0;padding:16px;background:var(--bg);color:var(--fg);font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
         header,main,footer{max-width:1100px;margin:0 auto}
@@ -266,6 +270,7 @@ module RaiderReporter
         .meta,.l,footer,.more{color:var(--mute);font-size:12px}
         .b{display:inline-block;padding:0 6px;border-radius:9px;font-size:11px;font-weight:600;border:1px solid var(--bdr)}
         .b.failed{background:var(--ruby);color:#fff;border-color:var(--ruby)}
+        .b.quarantined{color:var(--q);border-color:var(--q)}
         .b.passed{color:var(--ok)}.b.pending,.b.skipped,.b.flaky{color:var(--warn)}
         .rate{font-weight:600;margin-left:6px}
         #q{width:100%;margin:12px 0;padding:8px;border:1px solid var(--bdr);border-radius:6px;background:var(--card);color:var(--fg)}

data/lib/raider_reporter.rb

@@ -8,6 +8,8 @@ require_relative "raider_reporter/models"
 require_relative "raider_reporter/app_frame"
 require_relative "raider_reporter/payload"
 require_relative "raider_reporter/configuration"
+require_relative "raider_reporter/quarantine"
+require_relative "raider_reporter/exec_summary"
 require_relative "raider_reporter/content_types"
 require_relative "raider_reporter/collector"
 require_relative "raider_reporter/writers/local"
@@ -40,6 +42,7 @@ module RaiderReporter
       @configuration = nil
       @collector = nil
       @project_root = nil
+      @quarantined_keys = nil
     end
 
     def start_collector!
@@ -47,10 +50,16 @@ module RaiderReporter
       @project_root = Dir.pwd
       @collector = Collector.new(
         max_attachment_bytes: configuration.max_attachment_bytes,
-        warn: ->(message) { warn(message) }
+        warn: ->(message) { warn(message) },
+        quarantine: -> { quarantined_keys }
       )
     end
 
+    # Fetched once per process, at the first run start (spec §4.2).
+    def quarantined_keys
+      @quarantined_keys ||= Quarantine.load(configuration, warn: ->(message) { warn(message) })
+    end
+
     def attach(name, path, type: nil)
       unless collector&.in_test?
         warn("attach(#{name.inspect}) called outside a running example; ignored")
@@ -82,6 +91,8 @@ module RaiderReporter
     def finish!(output = $stderr)
       return nil if dry_run?
 
+      write_exec_summary
+
       detector = CiDetector.new
       ci = detector.ci
       shard = ShardDetector.new(job: ci&.fetch("job", nil)).shard
@@ -108,6 +119,19 @@ module RaiderReporter
 
     private
 
+    # First, so a later reporting problem cannot lose it. A failure here only means
+    # no summary, and exec then keeps the command's exit code.
+    def write_exec_summary
+      path = configuration.exec_summary_path
+      counts = collector&.framework_counts
+      return if path.nil? || counts.nil?
+
+      ExecSummary.write(path, framework_failures: counts[:failures], quarantined: collector.quarantined_count,
+                              errors_outside_tests: counts[:errors_outside_tests])
+    rescue StandardError => e
+      warn("could not write the exec summary: #{e.class}: #{e.message}")
+    end
+
     def write_html_report
       return unless configuration.html_report
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: raider_reporter
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.9.0
 platform: ruby
 authors:
 - Augustin Gottlieb
@@ -36,9 +36,15 @@ files:
 - lib/raider_reporter/content_types.rb
 - lib/raider_reporter/cucumber.rb
 - lib/raider_reporter/cucumber/formatter.rb
+- lib/raider_reporter/exec.rb
+- lib/raider_reporter/exec_summary.rb
+- lib/raider_reporter/gate.rb
+- lib/raider_reporter/gate_client.rb
+- lib/raider_reporter/gate_status.rb
 - lib/raider_reporter/github_event.rb
 - lib/raider_reporter/models.rb
 - lib/raider_reporter/payload.rb
+- lib/raider_reporter/quarantine.rb
 - lib/raider_reporter/rspec.rb
 - lib/raider_reporter/rspec/formatter.rb
 - lib/raider_reporter/shard_detector.rb