specbandit 1.2.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 24148d640548adc9f3bd6714ad39f4bfd380eeaf8e904f0fffd4b777a9b13d4c
-  data.tar.gz: '082673e08f9b40e9c8babae73f658d91b7627a40002b1e326d62fb49c5242d4f'
+  metadata.gz: d9934a98dc6ad0d278306da5cb4fdc818f56dceee9118e8581b49393f9d26150
+  data.tar.gz: a783e737d7d478397c689e82952c8c6c67e0fbab71c4ddc3ccb6ea34fee2e8b6
 SHA512:
-  metadata.gz: 06ced4b5b9530b9f836d90a0e05a663acf88d16f008c6f181acc19a1711ee6260465f42b7e27c76d3e16ec8d28ae621bfd4a614d2ec2aae26312927cb40fd3fb
-  data.tar.gz: 6f1578c3db2a446bce8677d16020014e4fa1c30e2483789f41311c1d18ba75567875a65728fca9f48136951b722b91631d674e995d2a66b1ebe26b99b910e128
+  metadata.gz: 5ab66609668cfd0d41cab6c243d94268da69f4659f9640f2391707dc74381e3566c459a6646010bc7c2197c7552c3818e78d4b066ba7c9f5e7727da2e0ffe50d
+  data.tar.gz: f352d8beb6637eabd22b3e11c0d2c62bce44498ba5da20bdd050785b4588913ecdbb9dfe53c799471bd7b75b9e74722e1efb6f4aeb89e6f9779beb4cbb149f35

data/README.md

@@ -109,6 +109,14 @@ specbandit push --key pr-123-run-456 spec/models/user_spec.rb spec/models/order_
 
 File input priority: **stdin > --pattern > direct args**.
 
+Add `--reset` when the producer can run more than once for the same key:
+
+```bash
+specbandit push --key pr-123-run-456 --reset --pattern 'spec/**/*_spec.rb'
+```
+
+See [Repeated pushes: `--reset`](#repeated-pushes---reset).
+
 ### 2. Steal and run from multiple workers
 
 Each CI runner steals batches and runs them. Start as many runners as you want -- they'll divide the work automatically.
@@ -137,6 +145,11 @@ specbandit push [options] [files...]
   --pattern PATTERN      Glob pattern for file discovery
   --redis-url URL        Redis URL (default: redis://localhost:6379)
   --key-ttl SECONDS      TTL for all Redis keys (default: 604800 / 1 week)
+  --reset                Empty the key before pushing (see below)
+
+specbandit reset [options]
+  --key KEY              Redis queue key (required)
+  --redis-url URL        Redis URL (default: redis://localhost:6379)
 
 specbandit work [options] [-- extra-opts...]
   --key KEY              Redis queue key (required)
@@ -423,6 +436,34 @@ Override via `--key-ttl` or `SPECBANDIT_KEY_TTL` (set it on `push`, which is whe
 specbandit push --key "pr-42-run-100" --key-ttl 259200 --pattern 'spec/**/*_spec.rb'
 ```
 
+## Repeated pushes: `--reset`
+
+The queue key is scoped by CI run, not by CI attempt. It has to be: re-running a single failed runner does not re-run the job that pushed, so that runner must still find the queue and the published marker the first attempt created.
+
+The cost is that the producer is not idempotent. A producer that pushes and then fails, or that is re-run as part of the whole workflow, appends a second copy of the work list to the same key. Every file is then enqueued twice, so the suite runs twice, and any two copies that reach the same worker are loaded twice in one process. For test files that define constants at file scope, the second load is fatal.
+
+`--reset` makes the push idempotent:
+
+```bash
+specbandit push --key "pr-42-run-100" --reset --pattern 'spec/**/*_spec.rb'
+```
+
+The queue and its `<key>:published` marker are deleted, then the work list is pushed, so the key holds exactly one copy however many times the producer runs. The leftover count is logged, and a non-zero one tells you an earlier attempt pushed a list nobody consumed:
+
+```
+[specbandit] Reset key 'pr-42-run-100': discarded 4213 queued files from a previous push.
+```
+
+There is also a standalone command, for callers that clean up separately from the push:
+
+```bash
+specbandit reset --key "pr-42-run-100"
+```
+
+Both leave the per-runner rerun keys and the failed keys alone, so a single-runner re-run can still replay its own files. A runner that finds data in both the shared queue and its rerun key is the **full rerun** case in the table above, and it resets its own memory.
+
+A reset with nothing to reset is not an error. Neither command clears the key when there is nothing to push in its place: dropping the marker on its own would make every worker on that key crash as "never published".
+
 ## How it works
 
 - **Push** uses `RPUSH` to append all file paths to a Redis list in a single command, sets `EXPIRE` on the key, and writes a durable `<key>:published` marker (with the same TTL). Empty Redis lists are auto-deleted, so this marker is what lets `work` tell "never pushed" (crash) apart from "drained, arriving late" (exit 0).
@@ -430,6 +471,7 @@ specbandit push --key "pr-42-run-100" --key-ttl 259200 --pattern 'spec/**/*_spec
 - **Record** (when `--key-rerun` is set): after each steal, the batch is also `RPUSH`ed to the per-runner rerun key.
 - **Replay** (when the rerun key has data and the shared queue is drained): reads all files from the rerun key via `LRANGE` (non-destructive), splits into batches, and runs them locally. The shared queue is never touched.
 - **Full rerun** (when both the shared queue and the rerun key have data): the stale rerun key is removed with `DEL`, then the runner steals from the shared queue and re-records as in a classic run.
+- **Reset** (`push --reset` or `specbandit reset`) removes the queue and its `<key>:published` marker in a single `DEL`, so a producer that runs twice cannot enqueue the work list twice. Rerun and failed keys are untouched.
 - **Mode selection** is derived entirely from Redis (published marker + queue/rerun state) -- see the decision table above. There is no re-run flag or environment variable.
 - **Run** delegates to the configured adapter:
   - **CLI adapter**: spawns a shell command per batch via `Open3`, appending file paths as arguments. Works with any test runner.

data/lib/specbandit/cli.rb

@@ -4,7 +4,7 @@ require 'optparse'
 
 module Specbandit
   class CLI
-    COMMANDS = %w[push work].freeze
+    COMMANDS = %w[push work reset].freeze
 
     def self.run(argv = ARGV)
       new(argv).execute
@@ -24,6 +24,8 @@ module Specbandit
         run_push
       when 'work'
         run_work
+      when 'reset'
+        run_reset
       when nil, '-h', '--help'
         print_usage
         0
@@ -46,7 +48,7 @@ module Specbandit
     private
 
     def run_push
-      options = { pattern: nil }
+      options = { pattern: nil, reset: false }
 
       parser = OptionParser.new do |opts|
         opts.banner = 'Usage: specbandit push [options] [files...]'
@@ -67,6 +69,10 @@ module Specbandit
           Specbandit.configuration.key_ttl = v
         end
 
+        opts.on('--reset', 'Empty the key before pushing, so a re-run cannot enqueue a second copy') do
+          options[:reset] = true
+        end
+
         opts.on('-h', '--help', 'Show this help') do
           puts opts
           return 0
@@ -78,11 +84,51 @@ module Specbandit
 
       publisher = Publisher.new
       files_arg = argv.empty? ? [] : argv
-      count = publisher.publish(files: files_arg, pattern: options[:pattern])
+      count = publisher.publish(files: files_arg, pattern: options[:pattern], reset: options[:reset])
 
       count.positive? ? 0 : 1
     end
 
+    # Empty a queue key and its published marker, leaving it as if nothing had
+    # ever been pushed. Standalone counterpart to `push --reset`, for callers
+    # that clean up separately from the push.
+    def run_reset
+      parser = OptionParser.new do |opts|
+        opts.banner = 'Usage: specbandit reset [options]'
+
+        opts.on('--key KEY', 'Redis queue key (required, or set SPECBANDIT_KEY)') do |v|
+          Specbandit.configuration.key = v
+        end
+
+        opts.on('--redis-url URL', 'Redis URL (default: redis://localhost:6379)') do |v|
+          Specbandit.configuration.redis_url = v
+        end
+
+        opts.on('-h', '--help', 'Show this help') do
+          puts opts
+          return 0
+        end
+      end
+
+      parser.parse!(argv)
+      Specbandit.configuration.validate!
+
+      key = Specbandit.configuration.key
+      queue = RedisQueue.new
+
+      begin
+        stale = queue.length(key)
+        queue.clear(key)
+        puts "[specbandit] Reset key '#{key}': discarded #{stale} queued files."
+      ensure
+        queue.close
+      end
+
+      # Removing nothing is a valid outcome: the caller asked for an empty key
+      # and got one.
+      0
+    end
+
     def run_work
       parser = OptionParser.new do |opts|
         opts.banner = 'Usage: specbandit work [options] [-- extra-opts...]'
@@ -200,12 +246,22 @@ module Specbandit
         Usage:
           specbandit push [options] [files...]           Enqueue test files into Redis
           specbandit work [options] [-- extra-opts...]   Steal and run test file batches
+          specbandit reset [options]                     Empty a queue key and its published marker
 
         Push options:
           --key KEY              Redis queue key (required, or set SPECBANDIT_KEY)
           --pattern PATTERN      Glob pattern for file discovery (e.g. 'spec/**/*_spec.rb')
           --redis-url URL        Redis URL (default: redis://localhost:6379)
           --key-ttl SECONDS      TTL for all Redis keys (default: 604800 / 1 week)
+          --reset                Empty the key before pushing (see Reset options)
+
+        Reset options:
+          --key KEY              Redis queue key (required, or set SPECBANDIT_KEY)
+          --redis-url URL        Redis URL (default: redis://localhost:6379)
+
+          Deletes the queue list and its ':published' marker, leaving the key as if
+          nothing had ever been pushed. Per-runner rerun and failed keys are left
+          alone, so a re-run of a single shard can still replay its own files.
 
         Work options:
           --key KEY              Redis queue key (required, or set SPECBANDIT_KEY)

data/lib/specbandit/publisher.rb

@@ -17,8 +17,11 @@ module Specbandit
     # Resolve files from the three input sources (priority: stdin > pattern > args)
     # and push them onto the Redis queue.
     #
+    # With `reset: true` the key is emptied first, so the queue ends up holding
+    # exactly this work list even if an earlier attempt already pushed one.
+    #
     # Returns the number of files enqueued.
-    def publish(files: [], pattern: nil)
+    def publish(files: [], pattern: nil, reset: false)
       resolved = resolve_files(files: files, pattern: pattern)
 
       if resolved.empty?
@@ -26,6 +29,11 @@ module Specbandit
         return 0
       end
 
+      # Only reset once there is something to put in its place. Clearing on an
+      # empty push would drop the marker too and leave workers crashing on a
+      # key that looks like it was never published.
+      reset_key if reset
+
       push_ms = measure { queue.push(key, resolved, ttl: key_ttl) }
       # Record a durable "published" marker so workers can tell a drained
       # queue ("worker arriving late", OK) apart from one that was never
@@ -40,6 +48,20 @@ module Specbandit
 
     private
 
+    # Empty the key before pushing. The leftover count is reported because a
+    # non-zero one means an earlier attempt pushed a list that no worker ever
+    # consumed, which is worth seeing in the producer's log.
+    def reset_key
+      stale = queue.length(key)
+      queue.clear(key)
+
+      if stale.positive?
+        output.puts "[specbandit] Reset key '#{key}': discarded #{stale} queued files from a previous push."
+      else
+        output.puts "[specbandit] Reset key '#{key}': nothing left over."
+      end
+    end
+
     # Run the block and return the wall-clock time it took, in milliseconds.
     # Used to surface how long the Redis round-trips took in the push log.
     def measure

data/lib/specbandit/redis_queue.rb

@@ -94,6 +94,17 @@ module Specbandit
       with_retries { redis.del(key) }
     end
 
+    # Remove a queue and its published marker together, so the key is back to
+    # the state it had before anything was ever pushed to it. A producer calls
+    # this before pushing: the queue key is not scoped by run attempt, so a
+    # second attempt would otherwise stack another copy of the work list on
+    # top of whatever the first attempt left behind.
+    #
+    # Returns the number of keys removed (0, 1 or 2).
+    def clear(key)
+      with_retries { redis.del(key, published_marker(key)) }
+    end
+
     def close
       redis.close
     end

data/lib/specbandit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Specbandit
-  VERSION = '1.2.0'
+  VERSION = '1.3.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: specbandit
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.0
 platform: ruby
 authors:
 - Ferran Basora