rspec-mergify 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2f6831a7105994678c61d541f3ca3840a618f93ed59561cecd23063d91063d7e
-  data.tar.gz: e6b1528a39efa7de31ef4ded5d44b2479f78521666cf48acb26edc6135580067
+  metadata.gz: 2d222991a085c33d95515483a7a41823b3db8c978d71e9e34a5c2ba8d4ca1777
+  data.tar.gz: a555d7a13c32f2bc1205c2060b5bd353b06f2b72c3460119d2800549f68a1c89
 SHA512:
-  metadata.gz: acc68c52030c7cf29760e293f99a46faae5c012745f0cd6d9b81c36cbb5c29505b7d16662a1785fd641f3f4ba8e424b592782308ef06356c3694e9bbfe09bddc
-  data.tar.gz: c95fc1c14925f61b63d1888e51e47f88af3ced37565a4964e6c30b913f0cbc6261a212562578f41faef937c1a0665fc0a150b302fd83e270bc991bcefb76b663
+  metadata.gz: 3908a8fb75af4875c54edfd828e1d638ca85730597d8283c8a558a06e50023b62345881a6bfbbcd37f1c69aef4e0d6043feb8bb027f00e83b492752295b2a8e2
+  data.tar.gz: ea86ab73dbab04394f1cd4c5718a4347592adf940aa1b82c2c54ccc4a56fa540a120127c54eb244531cfd90a94c38907bf22ec4f7c49129defeb57d5594fe2ac

data/README.md

@@ -7,6 +7,7 @@ RSpec plugin for [Mergify Test Insights](https://docs.mergify.com/ci-insights/).
 - **Test tracing** — Sends OpenTelemetry traces for every test to Mergify's API
 - **Flaky test detection** — Intelligently reruns tests to detect flakiness with budget constraints
 - **Test quarantine** — Quarantines failing tests so they don't block CI
+- **Test selection** — Runs only the previously-failing examples when Mergify's merge queue reruns a job
 
 ## Installation
 
@@ -18,6 +19,23 @@ gem 'rspec-mergify'
 
 Then run `bundle install`.
 
+### Supported platforms
+
+The gem ships precompiled, so nothing is built on your machine. Each platform gem carries an extension for Ruby 3.1 through 4.0:
+
+| Platform | Requirement |
+|---|---|
+| Linux x86_64 / aarch64 (glibc) | **glibc 2.30 or newer** — Debian 11+, Ubuntu 20.04+, RHEL 9+, Amazon Linux 2023 |
+| Linux x86_64 / aarch64 (musl) | Alpine; on Ruby 3.1 also `apk add libgcc` |
+| macOS arm64 / x86_64 | — |
+| Windows x64 (UCRT) | — |
+
+Anywhere else, and on glibc older than 2.30 — RHEL, Rocky, AlmaLinux and Oracle Linux 8, Amazon Linux 2, CentOS 7 — the extension cannot load. The gem still installs and your suite still runs, but nothing is reported to Mergify. Pin the last pure-Ruby release on those systems:
+
+```ruby
+gem 'rspec-mergify', '0.1.4'
+```
+
 ## Configuration
 
 Set the `MERGIFY_TOKEN` environment variable with your Mergify API token.
@@ -34,6 +52,54 @@ The plugin activates automatically when running in CI (detected via the `CI` env
 | `RSPEC_MERGIFY_DEBUG` | Print spans to console | `false` |
 | `MERGIFY_TRACEPARENT` | W3C distributed trace context | — |
 | `MERGIFY_TEST_JOB_NAME` | Mergify test job name | — |
+| `MERGIFY_TEST_SELECTION_ENABLE` | Opt this job into test selection (see below) | `false` |
+
+### Test selection
+
+When Mergify's merge queue reruns a job — a retry, or a step of a batch
+bisection — only the examples that failed on the previous attempt are
+informative. The gem asks Mergify whether the current run is such a rerun and,
+if so, runs only those examples.
+
+**This is off until you turn it on, per job.** Set
+`MERGIFY_TEST_SELECTION_ENABLE=true` on the job you want reduced. Installing the
+gem is not enough — a feature that decides not to run tests starts only where
+you wrote that it should. Anything that is not a recognised yes (unset, empty,
+`false`, unparsable) means no, and a job that has not opted in never asks.
+
+Once opted in, it only ever removes work:
+
+- the answer is applied **after** RSpec's own filters — file and line
+  arguments, `--tag`, `-e`, `--only-failures` all still narrow the run, never
+  widen it;
+- examples are matched by their id (`./spec/models/user_spec.rb[1:2]`), the
+  identity RSpec itself reruns failures by;
+- if Mergify says the previous attempt already ran every one of these examples
+  and they passed, the run executes none and exits green;
+- if any example Mergify asks for is not collected here, the full suite runs;
+- any error, timeout, or unrecognised answer runs the full suite;
+- if Mergify refuses to pick a previous attempt (several runs of this job
+  report under one name), the run **fails** with Mergify's explanation: give
+  each run its own `MERGIFY_TEST_JOB_NAME`;
+- at the end of the run the gem sends Mergify what it concluded (the counts
+  and the failing examples) — unless the run failed outside any example (a
+  failing `after(:context)` or suite hook), in which case a retry runs the
+  full suite.
+
+Each parallel_tests worker, and each CI job handed a slice of the spec files,
+is matched to the same slice of the previous attempt, by the examples it
+collects. This needs the slices to be the same on both attempts, which is the
+case for a split by file size or by a committed runtime log; Knapsack Pro's
+Queue Mode hands files out dynamically and is not supported.
+
+### Parallel runs
+
+[parallel_tests](https://github.com/grosser/parallel_tests) and [turbo_tests](https://github.com/serpapi/turbo_tests) need no setup. Each worker is a separate RSpec process, and each reports to Mergify as its own session:
+
+- flaky detection sizes each worker's rerun budget from the tests that worker runs;
+- each worker prints its own Mergify report, so a report covers only that worker's tests.
+
+Suites split across CI jobs (knapsack, `circleci tests split`, Buildkite `parallelism`) behave the same way, one session per job.
 
 For detailed documentation, see the [official guide](https://docs.mergify.com/ci-insights/test-frameworks/rspec/).
 

data/lib/mergify/rspec/ci_insights.rb

@@ -1,40 +1,172 @@
 # frozen_string_literal: true
 
 require 'securerandom'
-require 'opentelemetry-sdk'
+require_relative 'trace'
 require_relative 'utils'
 require_relative 'native'
-require_relative 'synchronous_batch_span_processor'
 require_relative 'resources/rspec'
+require_relative 'test_selection'
+require_relative 'session_verdict'
 
 module Mergify
   module RSpec
-    # Central orchestrator for Mergify Test Insights: sets up OpenTelemetry tracing,
-    # manages the tracer provider, and coordinates flaky detection and quarantine.
+    # Central orchestrator for Mergify Test Insights: records the run's spans,
+    # uploads them, and coordinates flaky detection, quarantine and test
+    # selection.
     # rubocop:disable-next Metrics/ClassLength
     class CIInsights
+      # The resource attributes pytest-mergify reports, under the same keys.
+      # The fingerprint and count describe what this run collected, on every
+      # run that took one; the selection keys echo what Mergify answered, as
+      # sent, and are absent on a run it never answered.
+      TEST_COLLECTION_FINGERPRINT = 'test.collection.fingerprint'
+      TEST_COLLECTION_COUNT = 'test.collection.count'
+      TEST_SELECTION_ANSWER = 'test.selection.answer'
+      TEST_SELECTION_REASON = 'test.selection.reason'
+      TEST_SELECTION_KEPT_COUNT = 'test.selection.kept_count'
+      TEST_SELECTION_NOT_APPLIED_REASON = 'test.selection.not_applied_reason'
+
+      # The largest value the engine's counters take (a signed 32-bit int).
+      MAX_COUNT = (2**31) - 1
+
       attr_reader :token, :repo_name, :api_url, :test_run_id,
-                  :tracer_provider, :tracer, :exporter,
+                  :recorder,
                   :branch_name,
-                  :flaky_detector, :flaky_detector_error_message, :quarantined_tests
+                  :flaky_detector, :flaky_detector_error_message, :quarantined_tests,
+                  :test_selection, :session_verdict, :session_verdict_result,
+                  :captured_session_verdict, :session_verdict_withheld
 
-      # rubocop:disable-next Metrics/MethodLength
+      # rubocop:disable-next Metrics/MethodLength,Metrics/AbcSize
       def initialize
         @token = ENV.fetch('MERGIFY_TOKEN', nil)
         @repo_name = Native.detect_repository_name
         @api_url = ENV.fetch('MERGIFY_API_URL', 'https://api.mergify.com')
         @test_run_id = SecureRandom.hex(8)
-        @tracer_provider = nil
-        @tracer = nil
-        @exporter = nil
+        @recorder = nil
+        @uploads = false
         @branch_name = nil
         @flaky_detector = nil
         @flaky_detector_error_message = nil
         @quarantined_tests = nil
+        @test_selection = nil
+        @selection_echo = nil
+        @session_verdict = SessionVerdict.new
+        @session_verdict_result = SessionVerdict::Result.new(sent: false, truncated: false)
+        @captured_session_verdict = nil
+        @session_verdict_withheld = false
 
         setup_tracing if Utils.in_ci?
       end
 
+      # Take the identity of what this run is about to execute, and ask
+      # Mergify whether part of it is enough.
+      #
+      # Called once RSpec has loaded the spec files and applied every filter
+      # of its own -- file and line arguments, `--tag`, `-e`, `--only-failures`
+      # -- so the fingerprint describes this process's examples and nothing
+      # else. A parallel_tests worker, or a CI job handed a slice of the spec
+      # files, collects its own slice and so reports its own identity: the
+      # engine matches each against the previous attempt of the same slice.
+      #
+      # The fingerprint is reported on every run that can take one, whether
+      # or not this job opted into the selection, as pytest-mergify does.
+      def on_examples_collected(example_ids)
+        return unless @recorder && Native.available?
+
+        fingerprint = Native.test_collection_fingerprint(example_ids)
+        @recorder.resource_attributes[TEST_COLLECTION_FINGERPRINT] = fingerprint
+        @recorder.resource_attributes[TEST_COLLECTION_COUNT] = example_ids.size
+        # A process with nothing to run has nothing to reduce, and asking would
+        # cost the job its next retry: every parallel worker left empty by a
+        # filter reports the same empty-set fingerprint under the same job, and
+        # Mergify refuses to choose between sessions it cannot tell apart.
+        load_test_selection(fingerprint) unless example_ids.empty?
+      end
+
+      # Report what Mergify answered and what this run made of it, once the
+      # answer has met the collection. Nothing is echoed unless Mergify
+      # actually answered: a run that never asked, or whose question went
+      # unanswered, was offered nothing.
+      # rubocop:disable-next Metrics/MethodLength
+      def on_selection_resolved(kept_count)
+        return unless @recorder && @test_selection&.served?
+
+        @selection_echo = {
+          'answer' => @test_selection.selection,
+          'reason' => @test_selection.reason,
+          'kept_count' => kept_count
+        }
+        not_applied = @test_selection.not_applied_reason
+        @selection_echo['not_applied_reason'] = not_applied if not_applied
+
+        resource = @recorder.resource_attributes
+        resource[TEST_SELECTION_ANSWER] = @selection_echo['answer']
+        resource[TEST_SELECTION_REASON] = @selection_echo['reason']
+        resource[TEST_SELECTION_KEPT_COUNT] = kept_count
+        resource[TEST_SELECTION_NOT_APPLIED_REASON] = not_applied if not_applied
+      end
+
+      # Write what this session concluded to Mergify, before the spans go.
+      #
+      # Sent exactly when the selection was asked for -- including when that
+      # request failed: the verdict is what the NEXT rerun of this job needs.
+      # Withheld when the run failed outside any example (a failing
+      # `after(:context)` or suite hook): its examples then read as passed,
+      # and a verdict naming no failure would have the retry run nothing and
+      # turn green on the same breakage. Without a verdict, the retry runs the
+      # whole suite. Never raises.
+      # rubocop:disable-next Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
+      def send_session_verdict(failed_outside_examples:)
+        return if @test_selection.nil?
+
+        if failed_outside_examples
+          @session_verdict_withheld = true
+          return
+        end
+
+        body = session_verdict_body
+        return if body.nil?
+
+        if test_mode?
+          @captured_session_verdict = body
+          @session_verdict_result = SessionVerdict::Result.new(sent: true, truncated: false)
+          return
+        end
+
+        if debug_mode?
+          puts "MERGIFY SESSION VERDICT: #{body}"
+          @session_verdict_result = SessionVerdict::Result.new(sent: true, truncated: false)
+          return
+        end
+
+        receipt = api_client.send_session_verdict(body)
+        # Dormant: the feature is not enabled for this repository, which the
+        # selection block already says.
+        return if receipt.nil?
+
+        @session_verdict_result = SessionVerdict::Result.new(sent: true, truncated: receipt['truncated'])
+      rescue StandardError => e
+        # Broader than ApiError on purpose: a value the binding cannot marshal
+        # raises something else, and a verdict must never cost the run.
+        message = e.is_a?(Native::ApiError) ? e.message : "#{e.class}: #{e.message}"
+        @session_verdict_result = SessionVerdict::Result.new(sent: false, truncated: false, error: message)
+      end
+
+      # Send the run, once, at the end. Failing to report a run is worth
+      # saying out loud but never worth failing a suite that just passed,
+      # so this answers with a message instead of raising.
+      def flush
+        return nil unless @recorder && @uploads
+        return nil if @recorder.finished_spans.empty?
+
+        owner, repo = Utils.split_full_repo_name(@repo_name)
+        client = Native::Client.new(@api_url, @token, owner, repo, Mergify::RSpec::VERSION)
+        client.upload_trace(@recorder.resource_attributes, @recorder.finished_spans.map(&:to_h))
+        nil
+      rescue Native::ApiError, Utils::InvalidRepositoryFullNameError => e
+        e.message
+      end
+
       def mark_test_as_quarantined_if_needed(example_id) # rubocop:disable Naming/PredicateMethod
         return false unless @quarantined_tests&.include?(example_id)
 
@@ -44,30 +176,104 @@ module Mergify
 
       private
 
-      def setup_tracing
-        processor, exp = build_processor
-        return unless processor
+      # Opt-in, per job, and asked for only where the run's identity is
+      # complete: the head branch and revision (a merge-queue draft branch on a
+      # rerun) and the job coordinates, the very values each uploaded example
+      # carries, so the server can match its records.
+      # rubocop:disable-next Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity
+      def load_test_selection(fingerprint)
+        return unless TestSelection.enabled?
+        # An empty token is what a fork or a bot pull request gets for a secret.
+        return if @token.to_s.empty? || @repo_name.to_s.empty?
 
-        @exporter = exp
-        resource = build_resource
-        @tracer_provider = OpenTelemetry::SDK::Trace::TracerProvider.new(resource: resource)
-        @tracer_provider.add_span_processor(processor)
-        @tracer = @tracer_provider.tracer('rspec-mergify', Mergify::RSpec::VERSION)
-        @branch_name = extract_branch_name(resource)
-        load_flaky_detector
-        load_quarantine
+        resource = @recorder.resource_attributes
+        branch = resource['vcs.ref.head.name']
+        head_sha = resource['vcs.ref.head.revision']
+        pipeline_name = resource['cicd.pipeline.name']
+        job_name = job_name_from(resource)
+        return if [branch, head_sha, pipeline_name, job_name].any? { |value| value.to_s.empty? }
+
+        answer = api_client.fetch_test_selection(branch.to_s, head_sha.to_s, pipeline_name.to_s,
+                                                 job_name.to_s, fingerprint)
+        @test_selection = answer.nil? ? TestSelection.unanswered : TestSelection.served(answer)
+      rescue Native::ApiError, Utils::InvalidRepositoryFullNameError => e
+        @test_selection = TestSelection.unanswered(init_error_msg: e.message)
+      end
+
+      # `mergify.test.job.name` is the operator-set override; the provider's
+      # own task name is the fallback. Same precedence as the other clients.
+      def job_name_from(resource)
+        override = resource['mergify.test.job.name']
+        override.to_s.empty? ? resource['cicd.pipeline.task.name'] : override
+      end
+
+      # The verdict body, keyed on exactly what the selection call was keyed
+      # on, so a verdict is found by what the asking run knows.
+      # rubocop:disable-next Metrics/MethodLength,Metrics/AbcSize
+      def session_verdict_body
+        resource = @recorder.resource_attributes
+        fingerprint = resource[TEST_COLLECTION_FINGERPRINT]
+        head_sha = resource['vcs.ref.head.revision']
+        pipeline_name = resource['cicd.pipeline.name']
+        job_name = job_name_from(resource)
+        return nil if [fingerprint, head_sha, pipeline_name, job_name].any? { |value| value.to_s.empty? }
+
+        body = {
+          'test_run_id' => @test_run_id,
+          'head_sha' => head_sha.to_s,
+          'pipeline_name' => pipeline_name.to_s,
+          'job_name' => job_name.to_s,
+          'collection_fingerprint' => fingerprint,
+          'collection_count' => resource[TEST_COLLECTION_COUNT],
+          'total_test_runtime_ms' => @session_verdict.total_test_runtime_ms,
+          'failing_tests' => @session_verdict.failing_tests,
+          'quarantined_failing_tests' => @session_verdict.quarantined_failing_tests
+        }.merge(@session_verdict.counts)
+        head_branch = resource['vcs.ref.head.name']
+        body['head_branch'] = head_branch.to_s unless head_branch.to_s.empty?
+        add_run_identity(body, resource)
+        body['selection'] = @selection_echo.dup if @selection_echo
+        body
       end
 
-      def build_processor
-        if debug_mode? || test_mode?
-          build_in_memory_processor
-        elsif @token && @repo_name
-          build_otlp_processor
-        else
-          [nil, nil]
+      # `run_attempt` only next to a `run_id`: the engine refuses an attempt of
+      # nothing. An attempt the engine's counter cannot hold is left out
+      # rather than refused.
+      def add_run_identity(body, resource)
+        run_id = resource['cicd.pipeline.run.id']
+        return if run_id.to_s.empty?
+
+        body['run_id'] = run_id.to_s
+        attempt = resource['cicd.pipeline.run.attempt']
+        body['run_attempt'] = attempt if attempt.is_a?(Integer) && attempt.between?(0, MAX_COUNT)
+      end
+
+      def api_client
+        @api_client ||= begin
+          owner, repo = Utils.split_full_repo_name(@repo_name)
+          Native::Client.new(@api_url, @token, owner, repo, Mergify::RSpec::VERSION)
         end
       end
 
+      # Recording is unconditional; whether the run is *uploaded* is what the
+      # token and repository decide. Debug and test runs keep their spans and
+      # send nothing, which is what they always did -- the difference is that
+      # the collector is the same object either way instead of two processors.
+      def setup_tracing
+        resource = build_resource
+        @recorder = Trace::Recorder.new(resource_attributes: resource,
+                                        traceparent: ENV.fetch('MERGIFY_TRACEPARENT', nil))
+        @uploads = uploadable?
+        # Only a pull request has a base branch, and that is what puts flaky
+        # detection in 'new' mode. GitHub Actions still sets GITHUB_BASE_REF on
+        # every other event, to an empty string, so empty counts as absent.
+        base_branch_name = resource['vcs.ref.base.name']
+        @base_branch_name = base_branch_name unless base_branch_name.to_s.empty?
+        @branch_name = @base_branch_name || resource['vcs.ref.head.name']
+        load_flaky_detector
+        load_quarantine
+      end
+
       def debug_mode?
         ENV.key?('RSPEC_MERGIFY_DEBUG')
       end
@@ -76,59 +282,23 @@ module Mergify
         ENV['_RSPEC_MERGIFY_TEST'] == 'true'
       end
 
-      def build_in_memory_processor
-        exp = OpenTelemetry::SDK::Trace::Export::InMemorySpanExporter.new
-        processor = OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(exp)
-        [processor, exp]
-      end
+      # A run is uploaded when there is somewhere to upload it to and nothing
+      # asking us not to: debug and test runs record and keep.
+      def uploadable?
+        return false if debug_mode? || test_mode?
+        return false unless @token && @repo_name && Native.available?
 
-      def build_otlp_processor
-        owner, repo = Utils.split_full_repo_name(@repo_name)
-        endpoint = "#{@api_url}/v1/ci/#{owner}/repositories/#{repo}/traces"
-        exp = create_otlp_exporter(endpoint)
-        processor = SynchronousBatchSpanProcessor.new(exp)
-        [processor, exp]
+        true
       end
 
       # The cicd.* and vcs.* attributes come from the Rust core, which every
-      # Mergify test client shares, so a provider gains them everywhere at once.
-      # What stays here is what only Ruby knows: the test framework, and the id
-      # this run invented for itself.
+      # Mergify test client shares, so a CI provider added there reaches every
+      # client at once. What stays here is what only Ruby knows: the test
+      # framework and its language, and the id this run invented for itself.
       def build_resource
-        resources = [
-          OpenTelemetry::SDK::Resources::Resource.create(Native.detect_attributes),
-          Resources::RSpec.detect,
-          OpenTelemetry::SDK::Resources::Resource.create('test.run.id' => @test_run_id)
-        ]
-        resources.reduce(OpenTelemetry::SDK::Resources::Resource.create({})) do |merged, r|
-          merged.merge(r)
-        end
-      end
-
-      def extract_branch_name(resource)
-        attrs = resource.attribute_enumerator.to_h
-        @base_branch_name = attrs['vcs.ref.base.name']
-        @base_branch_name || attrs['vcs.ref.head.name']
-      end
-
-      # rubocop:disable-next Metrics/MethodLength
-      def create_otlp_exporter(endpoint)
-        require 'opentelemetry-exporter-otlp'
-        original_env = ENV.fetch('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT', nil)
-        ENV['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'] = endpoint
-        begin
-          OpenTelemetry::Exporter::OTLP::Exporter.new(
-            endpoint: endpoint,
-            headers: { 'Authorization' => "Bearer #{@token}" },
-            compression: 'gzip'
-          )
-        ensure
-          if original_env
-            ENV['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'] = original_env
-          else
-            ENV.delete('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT')
-          end
-        end
+        Native.detect_attributes
+              .merge(Resources::RSpec.detect)
+              .merge('test.run.id' => @test_run_id)
       end
 
       # rubocop:disable-next Metrics/MethodLength