specbandit 1.0.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9444599b748d46d177475926a3b5f3740a6534c54a0e540c7663ee4eb4208d40
-  data.tar.gz: f7e5787ee93b270c2ba121d7edae8a2ccac5881f1bc873d65e5afcb481958e35
+  metadata.gz: 24148d640548adc9f3bd6714ad39f4bfd380eeaf8e904f0fffd4b777a9b13d4c
+  data.tar.gz: '082673e08f9b40e9c8babae73f658d91b7627a40002b1e326d62fb49c5242d4f'
 SHA512:
-  metadata.gz: 147b0eb446363ece19f086bbdedc375f6f6a15afc8d128628cdb64881396bd7c0ad01b8fc04b2af72b8db47aa89d83af0b48c86772ea57615582e412ec80a0d6
-  data.tar.gz: ae4357cd2cfcb1a12f539c5a7f43dacfa838a5810540ca41270c8e8fb1efb672189d898d0b4182106bb2719308b0f0ecf74495aade0f904f25d7253c48c96ed9
+  metadata.gz: 06ced4b5b9530b9f836d90a0e05a663acf88d16f008c6f181acc19a1711ee6260465f42b7e27c76d3e16ec8d28ae621bfd4a614d2ec2aae26312927cb40fd3fb
+  data.tar.gz: 6f1578c3db2a446bce8677d16020014e4fa1c30e2483789f41311c1d18ba75567875a65728fca9f48136951b722b91631d674e995d2a66b1ebe26b99b910e128

data/README.md

@@ -319,7 +319,7 @@ The full decision table:
 | Yes | Drained / empty | Empty | **OK, exit 0.** Worker arriving late -- everything was already taken by peers and this runner has no re-run memory. |
 | Yes | Has data | Empty | **Steal.** Classic run: pop batches from the shared queue (and record them to the rerun key if one is configured). |
 | Yes | Drained / empty | Has data | **Replay.** Classic re-run: ignore the shared queue and re-run exactly the recorded files. |
-| Yes | Has data | Has data | **Crash** (exit 1). Inconsistent state -- refuses to run to avoid double-executing. |
+| Yes | Has data | Has data | **Full rerun.** The queue was re-pushed while this runner still holds rerun memory from a previous run. The stale rerun key is deleted and the runner steals from the shared queue like a classic run, re-recording as it goes. |
 
 > **Empty key names count as "not provided".** A `--key-rerun` whose value is an empty string -- e.g. `--key-rerun "$VAR"` where `$VAR` is unset in CI -- is treated exactly like `--key-rerun` being absent. The same applies to `--key-failed`: an empty/unset name is treated as "not configured" and no failed files are recorded.
 
@@ -351,7 +351,7 @@ The full decision table:
 
 Key details:
 
-- **Replay reads non-destructively** (`LRANGE`, not `LPOP`). The rerun key is never consumed. If you re-run the same runner multiple times, it replays the same files every time.
+- **Replay reads non-destructively** (`LRANGE`, not `LPOP`). Replay never consumes the rerun key, so re-running the same runner multiple times replays the same files every time. The one case where the rerun key is deleted (`DEL`) is a **full rerun** -- the shared queue has data again while the rerun key still holds files from a previous run -- where the stale memory is reset and rebuilt from the newly stolen batches.
 - **The shared queue is never touched in replay mode**. Other runners are unaffected.
 - **Each runner has its own rerun key**. Only the re-run runner enters replay mode; runners that aren't re-run don't start at all.
 
@@ -429,6 +429,7 @@ specbandit push --key "pr-42-run-100" --key-ttl 259200 --pattern 'spec/**/*_spec
 - **Steal** uses `LPOP key count` (Redis 6.2+), which atomically pops up to N elements. No Lua scripts, no locks, no race conditions.
 - **Record** (when `--key-rerun` is set): after each steal, the batch is also `RPUSH`ed to the per-runner rerun key.
 - **Replay** (when the rerun key has data and the shared queue is drained): reads all files from the rerun key via `LRANGE` (non-destructive), splits into batches, and runs them locally. The shared queue is never touched.
+- **Full rerun** (when both the shared queue and the rerun key have data): the stale rerun key is removed with `DEL`, then the runner steals from the shared queue and re-records as in a classic run.
 - **Mode selection** is derived entirely from Redis (published marker + queue/rerun state) -- see the decision table above. There is no re-run flag or environment variable.
 - **Run** delegates to the configured adapter:
   - **CLI adapter**: spawns a shell command per batch via `Open3`, appending file paths as arguments. Works with any test runner.

data/lib/specbandit/cli.rb

@@ -131,6 +131,10 @@ module Specbandit
           Specbandit.configuration.report = v
         end
 
+        opts.on('--redis-max-attempts N', Integer, 'Redis connection retry attempts (default: 5)') do |v|
+          Specbandit.configuration.redis_max_attempts = v
+        end
+
         opts.on('--verbose', 'Show per-batch file list and full command output (default: quiet)') do
           Specbandit.configuration.verbose = true
         end

data/lib/specbandit/configuration.rb

@@ -5,7 +5,9 @@ module Specbandit
     attr_accessor :redis_url, :batch_size, :key, :rspec_opts, :key_ttl,
                   :key_rerun, :verbose,
                   :adapter, :command, :command_opts,
-                  :key_failed, :report
+                  :key_failed, :report,
+                  :redis_max_attempts, :redis_connect_timeout, :redis_timeout,
+                  :redis_reconnect_attempts
 
     DEFAULT_REDIS_URL = 'redis://localhost:6379'
     DEFAULT_BATCH_SIZE = 5
@@ -16,6 +18,17 @@ module Specbandit
     DEFAULT_KEY_TTL = 604_800 # 1 week in seconds
     DEFAULT_ADAPTER = 'cli'
 
+    # Redis connection resilience. Redis is a best-effort coordination store for
+    # a distributed test run, and CI runners can sit a WAN hop away from it
+    # (e.g. cross-datacenter mesh), so a transient blip must not red the build.
+    # We retry a handful of times with capped exponential backoff, and give the
+    # underlying client explicit connect/read/write timeouts + reconnects rather
+    # than relying on library defaults.
+    DEFAULT_REDIS_MAX_ATTEMPTS = 5
+    DEFAULT_REDIS_CONNECT_TIMEOUT = 3.0
+    DEFAULT_REDIS_TIMEOUT = 5.0
+    DEFAULT_REDIS_RECONNECT_ATTEMPTS = 3
+
     def initialize
       @redis_url = ENV.fetch('SPECBANDIT_REDIS_URL', DEFAULT_REDIS_URL)
       @batch_size = Integer(ENV.fetch('SPECBANDIT_BATCH_SIZE', DEFAULT_BATCH_SIZE))
@@ -29,12 +42,18 @@ module Specbandit
       @command_opts = parse_space_separated(ENV.fetch('SPECBANDIT_COMMAND_OPTS', nil))
       @key_failed = ENV.fetch('SPECBANDIT_KEY_FAILED', nil)
       @report = ENV.fetch('SPECBANDIT_REPORT', nil)
+
+      @redis_max_attempts = Integer(ENV.fetch('SPECBANDIT_REDIS_MAX_ATTEMPTS', DEFAULT_REDIS_MAX_ATTEMPTS))
+      @redis_connect_timeout = Float(ENV.fetch('SPECBANDIT_REDIS_CONNECT_TIMEOUT', DEFAULT_REDIS_CONNECT_TIMEOUT))
+      @redis_timeout = Float(ENV.fetch('SPECBANDIT_REDIS_TIMEOUT', DEFAULT_REDIS_TIMEOUT))
+      @redis_reconnect_attempts = Integer(ENV.fetch('SPECBANDIT_REDIS_RECONNECT_ATTEMPTS', DEFAULT_REDIS_RECONNECT_ATTEMPTS))
     end
 
     def validate!
       raise Error, 'key is required (set via --key or SPECBANDIT_KEY)' if key.nil? || key.empty?
       raise Error, 'batch_size must be a positive integer' unless batch_size.positive?
       raise Error, 'key_ttl must be a positive integer' unless key_ttl.positive?
+      raise Error, 'redis_max_attempts must be a positive integer' unless redis_max_attempts.positive?
     end
 
     private

data/lib/specbandit/redis_queue.rb

@@ -6,12 +6,28 @@ module Specbandit
   class RedisQueue
     attr_reader :redis
 
-    def initialize(redis_url: Specbandit.configuration.redis_url)
-      @redis = Redis.new(url: redis_url)
+    # Cap the exponential backoff so a real outage degrades/fails within a
+    # bounded window instead of sleeping for minutes on the last attempts.
+    MAX_BACKOFF_SECONDS = 10
+
+    def initialize(
+      redis_url: Specbandit.configuration.redis_url,
+      connect_timeout: Specbandit.configuration.redis_connect_timeout,
+      read_timeout: Specbandit.configuration.redis_timeout,
+      write_timeout: Specbandit.configuration.redis_timeout,
+      reconnect_attempts: Specbandit.configuration.redis_reconnect_attempts,
+      max_attempts: Specbandit.configuration.redis_max_attempts
+    )
+      @max_attempts = max_attempts
+      @redis = Redis.new(
+        url: redis_url,
+        connect_timeout: connect_timeout,
+        read_timeout: read_timeout,
+        write_timeout: write_timeout,
+        reconnect_attempts: reconnect_attempts
+      )
     end
 
-    MAX_ATTEMPTS = 3
-
     # Push file paths onto the queue and set an expiry on the key.
     # Returns the new length of the list.
     def push(key, files, ttl: nil)
@@ -72,6 +88,12 @@ module Specbandit
       with_retries { redis.lrange(key, 0, -1) }
     end
 
+    # Remove a key entirely. Used to discard stale rerun memory when a
+    # full rerun starts over from the shared queue.
+    def delete(key)
+      with_retries { redis.del(key) }
+    end
+
     def close
       redis.close
     end
@@ -83,7 +105,7 @@ module Specbandit
       "#{key}:published"
     end
 
-    def with_retries(attempts: MAX_ATTEMPTS)
+    def with_retries(attempts: @max_attempts)
       retries = 0
       begin
         yield
@@ -91,7 +113,7 @@ module Specbandit
         retries += 1
         raise if retries >= attempts
 
-        delay = 2**retries
+        delay = [2**retries, MAX_BACKOFF_SECONDS].min
         warn "[specbandit] Redis connection failed (attempt #{retries}/#{attempts}): #{e.message}. Retrying in #{delay}s..."
         sleep(delay)
         retry

data/lib/specbandit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Specbandit
-  VERSION = '1.0.1'
+  VERSION = '1.2.0'
 end

data/lib/specbandit/worker.rb

@@ -76,7 +76,7 @@ module Specbandit
     #   Yes       | empty/drained  | empty       | OK: worker arriving late (0)
     #   Yes       | has data       | empty       | Steal (record if rerun key set)
     #   Yes       | empty/drained  | has data    | Replay recorded files
-    #   Yes       | has data       | has data    | Crash: inconsistent (weird case)
+    #   Yes       | has data       | has data    | Full rerun: reset rerun key, steal
     #
     # "Published" is a durable marker written by `specbandit push`; it is the
     # only reliable signal that work was ever enqueued, because Redis
@@ -89,7 +89,7 @@ module Specbandit
       rerun_files = key_present?(key_rerun) ? queue.read_all(key_rerun) : []
 
       if key_has_data && rerun_files.any?
-        fail_inconsistent_state
+        run_full_rerun
       elsif rerun_files.any?
         run_replay(rerun_files)
       elsif key_has_data
@@ -119,14 +119,16 @@ module Specbandit
       1
     end
 
-    # Both the shared queue and this runner's rerun key hold files at once.
-    # That should never happen: a fresh run has no rerun memory yet, and a
-    # re-run reads from a drained queue. Crash instead of double-executing.
-    def fail_inconsistent_state
-      output.puts "[specbandit] ERROR: inconsistent state — shared queue '#{key}' still has files " \
-                  "while rerun key '#{key_rerun}' also has recorded files."
-      output.puts '[specbandit] Refusing to run to avoid double-execution / undefined behavior.'
-      1
+    # Both the shared queue and this runner's rerun key hold files: the queue
+    # was re-pushed while this runner still carries rerun memory from a
+    # previous run (a full rerun). The stored memory is stale -- discard it
+    # and steal from the shared queue like a classic run, re-recording each
+    # stolen batch as we go.
+    def run_full_rerun
+      output.puts "[specbandit] Shared queue '#{key}' and rerun key '#{key_rerun}' both have files. " \
+                  "Full rerun: resetting '#{key_rerun}' and working from '#{key}'."
+      queue.delete(key_rerun)
+      run_steal(record: true)
     end
 
     # Replay mode: run a known list of files in local batches.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: specbandit
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.2.0
 platform: ruby
 authors:
 - Ferran Basora
@@ -109,7 +109,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.0.16
 specification_version: 4
 summary: Distributed test runner using Redis as a work queue
 test_files: []