ci-queue 0.96.0 → 0.98.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8c3ced29dc916918cd28d47f312be55e014fd5836ab0c91d1e2eab17c9484951
-  data.tar.gz: 3ac890363a4a861ca2c3420c24490dff1ffd89dfb890b8d0fd71062ccf812003
+  metadata.gz: 020b8bf0b6887aa814e3f6b310bc62888e9a5e0af94c6ed652efc0af2b947ef7
+  data.tar.gz: 11e86bbe7b3e6bacfe96b84ae131c40222fb61a1afa9788f953c3a85db02d751
 SHA512:
-  metadata.gz: 27f198e8d9201f56112cc29bb3fcaa8518e7d5ddedc629d5eba13de06e0b936a8e809d47162d088f57d5ff614d3c4f92f003f786de62310ae421498e9913d8a9
-  data.tar.gz: 2c5f925d28012585afa8212cbb14afb1966cb0a2e5dbeaa9927486b2e96fb2c96d595f06ef7aa5ab6b97e5c238e5bca01e3572273a8a5fa9e8be357f54c44bf5
+  metadata.gz: b02b9a927e82321c3792c5a479487ef2a80e1093e23d4614e1e3b9191b9a5b5f9691cbdc78bd2a2a566e97dcbcf6231c3d9df19521bc2408099b6e905c38ecfa
+  data.tar.gz: 39cbdb4ef28553b81373f5836a8267b1919815fcab5959893bf002a307b7341e6337e045b00dd8a06e4c93dba2a1b3bd05ca9b96672292b5e25b8f50cc507fbf

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ci-queue (0.96.0)
+    ci-queue (0.98.0)
       logger
 
 GEM

data/README.md

@@ -36,7 +36,7 @@ Assuming you use one of the supported CI providers, the command can be as simple
 minitest-queue --queue redis://example.com run -Itest test/**/*_test.rb
 ```
 
-Additionally you can configure the requeue settings (see main README) with `--max-requeues` and `--requeue-tolerance`.
+Additionally you can configure the requeue settings (see main README) with `--max-requeues` and `--requeue-tolerance`. Worker-health circuit breakers are available through `--max-consecutive-failures` and `--max-consecutive-requeues`.
 
 #### Lazy loading (opt-in)
 
@@ -138,6 +138,57 @@ The runner also comes with a tool to investigate leaky tests:
 minitest-queue --queue path/to/test_order.log --failing-test 'SomeTest#test_something' bisect -Itest test/**/*_test.rb
 ```
 
+#### Parallel worker metadata
+
+Each result is stamped, as it is recorded in the process that ran the test, with:
+
+- `parallel_worker_pid`: the pid of the process that ran the test.
+- `parallel_worker_test_index`: a 0-based monotonic counter of results recorded by that process (requeued executions get their own index). Restarts at 0 in each forked process.
+- `parallel_worker_id`: an identifier injected by the embedding environment (e.g. a Rails parallel-testing worker number); `nil` when not applicable.
+
+All three fields are included (nil-safe) in `log/test_data.json` emitted by the test data reporter, making per-worker-process execution order reconstructable downstream (`PARTITION BY parallel_worker_id, parallel_worker_pid ORDER BY parallel_worker_test_index`).
+
+The worker id can be provided either programmatically or via the environment:
+
+```ruby
+Minitest::Queue.parallel_worker_id = 3
+```
+
+| Variable | Description |
+|---|---|
+| `CI_QUEUE_PARALLEL_WORKER_ID=N` | Sets `parallel_worker_id` for results produced by this process. Read once per process; the setter takes precedence. |
+
+Stamping is first-writer-wins. When ci-queue runs tests in-process (the normal
+`minitest-queue` flow), results are stamped automatically as they are recorded.
+Embedding environments that run tests in forked workers and transport results to
+another process (e.g. over DRb to a central reporting server) must stamp in the
+worker **before** sending:
+
+```ruby
+# in the forked worker, after running the test and before the DRb send
+Minitest::Queue.stamp_parallel_worker_metadata(result)
+```
+
+Otherwise the automatic stamp during reporting would capture the reporting
+process's pid and an arrival-order index interleaved across workers. Pre-stamped
+results pass through reporting untouched.
+
+Notes:
+
+- `parallel_worker_id` identifies a forked test process *inside* one queue worker.
+  It is unrelated to ci-queue's own `--worker` / `CI::Queue::Configuration#worker_id`,
+  which identifies the whole queue worker (typically one CI job).
+- The payload intentionally carries no build/job identity — like every other field
+  in `log/test_data.json`, scoping to a build/job (e.g. a `job_id` column making
+  `(job_id, parallel_worker_id, parallel_worker_pid)` unique across a build) is
+  expected to be attached by whatever pipeline ingests the file.
+- Stamping is mutex-guarded, so indexes are unique and gap-free even when results
+  are recorded from multiple threads. However, per-worker order reconstruction is
+  only meaningful with process-based (forked) workers: under thread-based
+  parallelization all threads share one `(parallel_worker_id, parallel_worker_pid)`
+  partition and the index reflects record order across threads.
+- This applies to minitest-queue only; rspec-queue does not emit these fields.
+
 ### RSpec [DEPRECATED]
 
 The rspec-queue runner is deprecated. The minitest-queue runner continues to be supported and is actively being improved. At Shopify, we strongly recommend that new projects set up their test suite using Minitest rather than RSpec.
@@ -158,6 +209,30 @@ rspec-queue --queue redis://example.com --timeout 600 --report
 
 Because of how `ci-queue` executes the examples, `before(:all)` and `after(:all)` hooks are not supported. `rspec-queue` will explicitly reject them.
 
+### Worker circuit breakers
+
+Both runners support two independent, disabled-by-default circuit breakers:
+
+- `--max-consecutive-failures MAX` stops a worker after `MAX` consecutive final test failures.
+- `--max-consecutive-requeues MAX` stops a worker after `MAX` consecutive failures were accepted for requeueing.
+
+The requeue count is local to each worker and resets whenever that worker produces any non-requeued result, including a pass, skip, ignored flaky result, or final failure. An accepted requeue neither increments nor resets the consecutive-final-failure count. Requeue attempts rejected because of an ownership race or an exhausted retry budget follow the final-failure path and do not increment the requeue count.
+
+For example:
+
+```bash
+minitest-queue --max-consecutive-requeues 10 --queue redis://example.com run test/**/*_test.rb
+rspec-queue --max-consecutive-requeues 10 --queue redis://example.com
+```
+
+The breaker can also be configured programmatically:
+
+```ruby
+CI::Queue::Configuration.new(max_consecutive_requeues: 10)
+# or
+config.max_consecutive_requeues = 10
+```
+
 ## Releasing a New Version
 
 After merging changes to `main`, follow these steps to release and propagate the update:

data/lib/ci/queue/circuit_breaker.rb

@@ -8,6 +8,9 @@ module CI
         def report_failure!
         end
 
+        def report_requeue!
+        end
+
         def report_success!
         end
 
@@ -32,6 +35,9 @@ module CI
         def report_failure!
         end
 
+        def report_requeue!
+        end
+
         def report_success!
         end
 
@@ -60,6 +66,9 @@ module CI
           @consecutive_failures += 1
         end
 
+        def report_requeue!
+        end
+
         def report_success!
           @consecutive_failures = 0
         end
@@ -72,6 +81,33 @@ module CI
           'This worker is exiting early because it encountered too many consecutive test failures, probably because of some corrupted state.'
         end
       end
+
+      class MaxConsecutiveRequeues
+        def initialize(max_consecutive_requeues:)
+          @max = max_consecutive_requeues
+          @consecutive_requeues = 0
+        end
+
+        def report_failure!
+          @consecutive_requeues = 0
+        end
+
+        def report_requeue!
+          @consecutive_requeues += 1
+        end
+
+        def report_success!
+          @consecutive_requeues = 0
+        end
+
+        def open?
+          @consecutive_requeues >= @max
+        end
+
+        def message
+          'This worker is exiting early because it requeued too many consecutive tests, probably because of some corrupted state.'
+        end
+      end
     end
   end
 end

data/lib/ci/queue/common.rb

@@ -27,6 +27,10 @@ module CI
         config.circuit_breakers.each(&:report_failure!)
       end
 
+      def report_requeue!
+        config.circuit_breakers.each(&:report_requeue!)
+      end
+
       def report_success!
         config.circuit_breakers.each(&:report_success!)
       end

data/lib/ci/queue/configuration.rb

@@ -55,7 +55,7 @@ module CI
       def initialize(
         timeout: 30, build_id: nil, worker_id: nil, max_requeues: 0, requeue_tolerance: 0,
         namespace: nil, seed: nil, flaky_tests: [], statsd_endpoint: nil, max_consecutive_failures: nil,
-        grind_count: nil, max_duration: nil, failure_file: nil, max_test_duration: nil,
+        max_consecutive_requeues: nil, grind_count: nil, max_duration: nil, failure_file: nil, max_test_duration: nil,
         max_test_duration_percentile: 0.5, track_test_duration: false, max_test_failed: nil,
         queue_init_timeout: nil, redis_ttl: 8 * 60 * 60, report_timeout: nil, inactive_workers_timeout: nil,
         export_flaky_tests_file: nil, warnings_file: nil, debug_log: nil, max_missed_heartbeat_seconds: nil, heartbeat_max_test_duration: nil,
@@ -79,6 +79,7 @@ module CI
         @track_test_duration = track_test_duration
         @worker_id = worker_id
         self.max_consecutive_failures = max_consecutive_failures
+        self.max_consecutive_requeues = max_consecutive_requeues
         self.max_duration = max_duration
         @redis_ttl = redis_ttl
         @report_timeout = report_timeout
@@ -165,6 +166,12 @@ module CI
         end
       end
 
+      def max_consecutive_requeues=(max)
+        if max
+          @circuit_breakers << CircuitBreaker::MaxConsecutiveRequeues.new(max_consecutive_requeues: max)
+        end
+      end
+
       def max_duration=(duration)
         if duration
           @circuit_breakers << CircuitBreaker::Timeout.new(duration: duration)

data/lib/ci/queue/version.rb

@@ -2,7 +2,7 @@
 
 module CI
   module Queue
-    VERSION = '0.96.0'
+    VERSION = '0.98.0'
     DEV_SCRIPTS_ROOT = ::File.expand_path('../../../../../redis', __FILE__)
     RELEASE_SCRIPTS_ROOT = ::File.expand_path('../redis', __FILE__)
   end

data/lib/minitest/queue/runner.rb

@@ -692,6 +692,15 @@ module Minitest
             queue_config.max_consecutive_failures = max
           end
 
+          help = <<~EOS
+            Defines after how many consecutive accepted requeues the worker will be considered unhealthy and stop reserving tests.
+            Defaults to disabled.
+          EOS
+          opts.separator ""
+          opts.on('--max-consecutive-requeues MAX', Integer, help) do |max|
+            queue_config.max_consecutive_requeues = max
+          end
+
           help = <<~EOS
             Must set this option in report and report_grind command if you set --max-test-duration in the report_grind
           EOS

data/lib/minitest/queue/test_data.rb

@@ -49,7 +49,7 @@ module Minitest
       end
 
       def test_duration
-        @test.time
+        @test.time || 0.0
       end
 
       def test_start_timestamp
@@ -73,6 +73,22 @@ module Minitest
         @test.source_location.last
       end
 
+      # The parallel_worker_* fields are stamped by
+      # Minitest::Queue.stamp_parallel_worker_metadata in the worker process
+      # that ran the test. They are nil for embedders that don't produce
+      # them (results lacking the accessors, or no worker id configured).
+      def parallel_worker_id
+        @test.parallel_worker_id if @test.respond_to?(:parallel_worker_id)
+      end
+
+      def parallel_worker_test_index
+        @test.parallel_worker_test_index if @test.respond_to?(:parallel_worker_test_index)
+      end
+
+      def parallel_worker_pid
+        @test.parallel_worker_pid if @test.respond_to?(:parallel_worker_pid)
+      end
+
       # Error class only considers failures wheras the other error fields also consider skips
       def error_class
         return nil unless @test.failure
@@ -119,6 +135,9 @@ module Minitest
           test_finish_timestamp: test_finish_timestamp,
           test_file_path: test_file_path,
           test_file_line_number: test_file_line_number,
+          parallel_worker_id: parallel_worker_id,
+          parallel_worker_test_index: parallel_worker_test_index,
+          parallel_worker_pid: parallel_worker_pid,
           error_class: error_class,
           error_message: error_message,
           error_file_path: error_file_path,

data/lib/minitest/queue.rb

@@ -111,10 +111,22 @@ module Minitest
     attr_accessor :queue_id, :queue_entry
   end
 
+  # Per-worker-process execution metadata, stamped on each result in the
+  # process that ran the test (before any DRb send in embedding
+  # environments), so that per-process execution order is reconstructable
+  # downstream: PARTITION BY parallel_worker_id, parallel_worker_pid
+  # ORDER BY parallel_worker_test_index.
+  module ParallelWorkerMetadata
+    attr_accessor :parallel_worker_id, :parallel_worker_test_index, :parallel_worker_pid
+  end
+
   module Queue
     extend ::CI::Queue::OutputHelpers
     attr_writer :run_command_formatter, :project_root
 
+    PARALLEL_WORKER_METADATA_MUTEX = Mutex.new
+    private_constant :PARALLEL_WORKER_METADATA_MUTEX
+
     def run_command_formatter
       @run_command_formatter ||= if defined?(Rails) && defined?(Rails::TestUnitRailtie)
         RAILS_RUN_COMMAND_FORMATTER
@@ -155,6 +167,60 @@ module Minitest
     end
 
     class << self
+      # Identifies which parallel worker this process is (e.g. Rails'
+      # parallel testing worker number). Injected by the embedding
+      # environment via this setter or the CI_QUEUE_PARALLEL_WORKER_ID
+      # environment variable; nil when not applicable.
+      attr_writer :parallel_worker_id
+
+      def parallel_worker_id
+        return @parallel_worker_id if @parallel_worker_id
+
+        # Memoized per process: re-read after a fork, but not on every test.
+        if @parallel_worker_id_env_pid != Process.pid
+          @parallel_worker_id_env_pid = Process.pid
+          @parallel_worker_id_from_env = parallel_worker_id_from_env
+        end
+        @parallel_worker_id_from_env
+      end
+
+      # Stamps per-process execution metadata on a result. Must be called
+      # in the process that ran the test so pid and per-process test index
+      # are captured worker-side.
+      #
+      # First writer wins: results that were already stamped are left
+      # untouched. Embedding environments that run tests in forked workers
+      # and transport results to another process (e.g. over DRb) MUST call
+      # this in the worker before sending, otherwise the stamp applied
+      # during reporting would carry the reporting process's pid and an
+      # arrival-order index interleaved across workers.
+      #
+      # The counter is mutex-guarded: results recorded from multiple threads
+      # (e.g. a DRb server dispatching each call on its own thread) get
+      # unique, gap-free indexes reflecting record order in this process.
+      # Returns the result when it was stamped, nil when it was skipped.
+      def stamp_parallel_worker_metadata(result)
+        return unless result.respond_to?(:parallel_worker_pid=)
+        return if result.parallel_worker_pid # already stamped worker-side
+
+        PARALLEL_WORKER_METADATA_MUTEX.synchronize do
+          pid = Process.pid
+          if @parallel_worker_metadata_pid != pid
+            # Restart the per-process counter in forked workers so that
+            # (parallel_worker_id, parallel_worker_pid, parallel_worker_test_index)
+            # always reflects execution order within a single process.
+            @parallel_worker_metadata_pid = pid
+            @parallel_worker_next_test_index = 0
+          end
+
+          result.parallel_worker_id = parallel_worker_id
+          result.parallel_worker_pid = pid
+          result.parallel_worker_test_index = @parallel_worker_next_test_index
+          @parallel_worker_next_test_index += 1
+        end
+        result
+      end
+
       def queue
         Minitest.queue
       end
@@ -179,6 +245,8 @@ module Minitest
       end
 
       def handle_test_result(reporter, example, result)
+        stamp_parallel_worker_metadata(result)
+
         if result.respond_to?(:queue_id=)
           result.queue_id = example.id
           result.queue_entry = example.queue_entry if result.respond_to?(:queue_entry=)
@@ -198,6 +266,7 @@ module Minitest
         end
 
         if failed && CI::Queue.requeueable?(result) && queue.requeue(example.queue_entry)
+          queue.report_requeue!
           result.requeue!
           if CI::Queue.debug?
             $stderr.puts "[ci-queue][requeue] test_id=#{example.id} error_class=#{result.failures.first&.class} error=#{result.failures.first&.message&.lines&.first&.chomp}"
@@ -212,6 +281,17 @@ module Minitest
 
       private
 
+      def parallel_worker_id_from_env
+        value = ENV['CI_QUEUE_PARALLEL_WORKER_ID']
+        return nil if value.nil? || value.empty?
+
+        begin
+          Integer(value)
+        rescue ArgumentError
+          value
+        end
+      end
+
       def report_load_stats(queue)
         return unless CI::Queue.debug?
         return unless queue.respond_to?(:file_loader)
@@ -440,6 +520,7 @@ module Minitest
         if result
           result.start_timestamp = start_timestamp
           result.finish_timestamp = current_timestamp
+          result.time ||= (result.finish_timestamp - result.start_timestamp).to_f
         end
       end
 
@@ -576,7 +657,7 @@ module Minitest
           Queue.run(*args)
 
           if queue.config.circuit_breakers.any?(&:open?)
-            STDERR.puts queue.config.circuit_breakers.map(&:message).join(' ').strip
+            STDERR.puts queue.config.circuit_breakers.select(&:open?).map(&:message).join(' ').strip
           end
 
           if queue.max_test_failed?
@@ -607,11 +688,13 @@ if defined? Minitest::Result
   Minitest::Result.prepend(Minitest::Flakiness)
   Minitest::Result.prepend(Minitest::WithTimestamps)
   Minitest::Result.prepend(Minitest::ResultMetadata)
+  Minitest::Result.prepend(Minitest::ParallelWorkerMetadata)
 else
   Minitest::Test.prepend(Minitest::Requeueing)
   Minitest::Test.prepend(Minitest::Flakiness)
   Minitest::Test.prepend(Minitest::WithTimestamps)
   Minitest::Test.prepend(Minitest::ResultMetadata)
+  Minitest::Test.prepend(Minitest::ParallelWorkerMetadata)
 
   module MinitestBackwardCompatibility
     def source_location

data/lib/rspec/queue.rb

@@ -158,6 +158,15 @@ module RSpec
           queue_config.max_consecutive_failures = Integer(max)
         end
 
+        help = <<~EOS
+          Defines after how many consecutive accepted requeues the worker will be considered unhealthy and stop reserving tests.
+          Defaults to disabled.
+        EOS
+        parser.separator ""
+        parser.on('--max-consecutive-requeues MAX', *help) do |max|
+          queue_config.max_consecutive_requeues = Integer(max)
+        end
+
         help = <<~EOS
           Defines how long the test report remain after the test run, in seconds.
           Defaults to 28,800 (8 hours)
@@ -215,22 +224,21 @@ module RSpec
 
       def finish(reporter, acknowledge: true)
         if acknowledge && reporter.respond_to?(:requeue)
-          if @exception
-            reporter.report_failure!
-          else
-            reporter.report_success!
-          end
-
           if @exception && CI::Queue.requeueable?(@exception) && reporter.requeue
+            reporter.report_requeue!
             reporter.cancel_run!
             dup.mark_as_requeued!(reporter)
             return true
+          end
+
+          if @exception
+            reporter.report_failure!
           else
-            super(reporter)
+            reporter.report_success!
           end
-        else
-          super(reporter)
         end
+
+        super(reporter)
       end
 
       def reset!
@@ -417,6 +425,10 @@ module RSpec
         @queue.report_failure!
       end
 
+      def report_requeue!
+        @queue.report_requeue!
+      end
+
       def requeue
         @queue.requeue(@example.queue_entry)
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ci-queue
 version: !ruby/object:Gem::Version
-  version: 0.96.0
+  version: 0.98.0
 platform: ruby
 authors:
 - Jean Boussier
@@ -319,7 +319,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.11
+rubygems_version: 4.0.16
 specification_version: 4
 summary: Distribute tests over many workers using a queue
 test_files: []