rspecq-instructure 0.8.1 → 0.8.2.pre.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1df575298a210ef26b7f057dcfd500287519e2ec6f8d32b2296f6eea26bc05fc
-  data.tar.gz: fe66df1684e18af2501fb67cf26ce475a9db89e65717f83e0ff1f3bb9455041c
+  metadata.gz: dc24b763464c7e4c08422ac93f921cc37dc3d19a099be47a2cbb3a0503993002
+  data.tar.gz: 7a31691e1f3736806edb4d1f6fda4e3085d87738cf795e8be5c49a5b7f2dcb70
 SHA512:
-  metadata.gz: 52b16ee95c80f6a7da5246c6335910dde1a897d7ce37ac7d5f99b508aa306d3cdce350d47fc7d641d1c4947f894345eea3d49844df04aab5aa0ae7de97c51bc1
-  data.tar.gz: 31ec697c4f6b54c39a8dad7527d6151e177e45a81ad55e74e6a07ad459ae3c430bf02d35a8674817488b9e56008941875e371e73a293d69e8646ac446aefc7b2
+  metadata.gz: ed0e52a1d7e547c1099fe18941597aafe574d6feac020726d15f7ad6072c31c449e8d392b7a7964fccf5f030469e74fca39f1b1b579ac1e0544306502b4c1778
+  data.tar.gz: 1979f5b9851690d2bda1a6469491f9e8b4873225a620b59bd46739b6117c3fdcec5ec26b7bb49f9cc5874e9774fa24f01dddc235ca74331df6e20bad7a3fb814

data/CHANGELOG.md

@@ -2,7 +2,30 @@
 
 Breaking changes are prefixed with a "[BREAKING]" label.
 
-## 0.8.1 (unreleased)
+Fixes accumulate under the unreleased heading without touching
+`lib/rspecq/version.rb`. Bumping that file is what publishes, so a batch is cut
+by one deliberate bump: `X.Y.Z.pre.N` for a build to soak in canvas, then
+`X.Y.Z` once it holds. See the header of `.github/workflows/release.yml`.
+
+## 0.8.2 (unreleased)
+
+- A raise out of the RSpec runner no longer pins a job. The run is wrapped so
+  the error is recorded against the job and the job is acknowledged in an
+  `ensure`; previously it sat in the running hash until `requeue_lost_job`
+  reclaimed it a full `worker_liveness_sec` later, and the worker died with it.
+- Requeuing a flaky example no longer sleeps. The worker used to block 0.5s on
+  every requeue so a peer could claim the job first; `reserve_job` now skips a
+  job this worker requeued, provided a live peer exists and something else is
+  queued. Alone, or as the last job, it runs it rather than delay the retry.
+- A master that dies between winning the election and publishing no longer
+  hangs the build. The election is now a lease that expires, so a surviving
+  worker stands again instead of every worker blocking until
+  `wait_until_published` gives up. The master extends the lease while it
+  publishes, and a worker that has lost it will not publish over the new one.
+  `--queue-wait-timeout` now waits up to that long plus `worker_liveness_sec`,
+  restarting for each new master.
+
+## 0.8.1 (2026-09-23)
 
 - [BREAKING] The gem is published to rubygems.org as `rspecq-instructure`,
   not to Artifactory as `rspecq`. Change the dependency to
@@ -13,11 +36,50 @@ Breaking changes are prefixed with a "[BREAKING]" label.
   name, so no code has to change: `require "rspecq"`, the `RSpecQ` namespace,
   and the `rspecq` executable all still work as before.
 
-## 0.8.0.instructure2 (unreleased)
+## 0.8.0 (2026-09-17)
+
+Reconciles the Instructure fork with upstream skroutz and is the first release
+published to Artifactory rather than a git ref.
 
 - [BREAKING] Require Ruby >= 3.2 and raise the redis-rb floor to `>= 5.0`
   (cap `< 7.0`). Verified compatible with redis-rb 6.0 / RESP3.
+- New `--tag` option filters the specs a build runs; prefix with `~` to exclude.
+  Repeatable, and passed through to rspec.
+- Two-tier job timings. Workers always record per-build timings; the reporter
+  promotes them to the global key only with `--update-timings`, and
+  `--timings-key` can redirect that promotion. Whole-file timings are
+  reconstructed from per-example entries so a split file is still recognised as
+  slow. The global key is explicitly persisted after promotion; previously it
+  inherited the build key's 24h TTL and expired silently, dropping later builds
+  to random ordering.
+- Build outcome (`success`/`failure`) is written to redis when the build
+  finishes, so callers can read the result at the source instead of scraping
+  output.
+- Lost jobs are tracked and recovered: a job reserved by a worker that died is
+  requeued, counted, and reported. The reporter summary now also shows requeue,
+  flaky and lost-job counts, and flaky detail carries per-job timing.
+- Flaky failures are recorded in a structured key rather than only printed.
+- Build-time reporting reworked: workers stamp elected-master, queue-ready and
+  finished times, and the summary breaks total runtime into time-from-election,
+  time-from-queue-ready and total worker execution time.
+- The rerun command printed for a flaky job is trimmed to that job.
+- `--seed` is parsed as base 10, so a leading zero no longer becomes octal, and
+  the seed feeds the worker's own RNG as well as rspec's.
+- Workers back off when no job is available instead of spinning.
+- Redis connection settings (timeouts, reconnect backoff) are tunable via CLI
+  flags and env vars.
+- Lua scripts run via plain `EVAL` instead of `EVALSHA`. `SCRIPT LOAD` is
+  unavailable behind some Redis proxies, and the overhead saved is not worth
+  the incompatibility.
+- Fixed `Queue#publish` returning the wrong value, which made the published
+  queue size log as `1`.
+- Fixed a race where a build's duration went unrecorded if the reporter
+  observed exhaustion before any worker stamped `finished_at`, and a
+  string-comparison bug that capped `max_requeues` at single digits.
+- Removed dead code: `requeueable_job?`, its Lua script, and its stats key.
 - `REDIS_GEM` env var pins the redis-rb version in CI (mirrors `RSPEC_CORE`).
+- `RSPECQ_TEST_REDIS_URL` points the test suite at a specific redis, so it no
+  longer assumes a default local instance.
 
 ## 0.7.1 (2021-04-08)
 

data/lib/rspecq/formatters/failure_recorder.rb

@@ -38,10 +38,7 @@ module RSpecQ
         msg = presenter.fully_formatted(nil, @colorizer)
 
         if @queue.requeue_job(example, @max_requeues, @worker_id)
-          # HACK: try to avoid picking the job we just requeued; we want it
-          # to be picked up by a different worker
           @queue.record_flaky_failure(notification.example.id, msg)
-          sleep 0.5
           return
         end
 

data/lib/rspecq/queue.rb

@@ -16,15 +16,30 @@ module RSpecQ
     RESERVE_JOB = <<~LUA.freeze
       local queue = KEYS[1]
       local queue_running = KEYS[2]
+      local requeued_job_original_worker = KEYS[3]
+      local worker_heartbeats = KEYS[4]
       local worker_id = ARGV[1]
+      local time_now = ARGV[2]
+      local timeout = ARGV[3]
 
       local job = redis.call('lpop', queue)
-      if job then
-        redis.call('hset', queue_running, worker_id, job)
-        return job
-      else
+      if not job then
         return nil
       end
+
+      -- A requeue sits at the head to be retried promptly; leave our own for a
+      -- live peer. Alone, deferring it only delays the retry (and fail-fast).
+      if redis.call('hget', requeued_job_original_worker, job) == worker_id and
+         redis.call('zcount', worker_heartbeats, time_now - timeout, '+inf') > 1 then
+        local other = redis.call('lpop', queue)
+        if other then
+          redis.call('lpush', queue, job)
+          job = other
+        end
+      end
+
+      redis.call('hset', queue_running, worker_id, job)
+      return job
     LUA
 
     # Scans for dead workers and puts their reserved jobs back to the queue.
@@ -74,6 +89,37 @@ module RSpecQ
       return true
     LUA
 
+    BECOME_MASTER = <<~LUA.freeze
+      local key_queue_status = KEYS[1]
+      local key_master_id = KEYS[2]
+      local status_initializing = ARGV[1]
+      local lease = ARGV[2]
+      local worker_id = ARGV[3]
+
+      if redis.call('set', key_queue_status, status_initializing, 'NX', 'EX', lease) then
+        redis.call('set', key_master_id, worker_id)
+        return 1
+      end
+
+      return false
+    LUA
+
+    REFRESH_ELECTION_LEASE = <<~LUA.freeze
+      local key_queue_status = KEYS[1]
+      local key_master_id = KEYS[2]
+      local worker_id = ARGV[1]
+      local lease = ARGV[2]
+      local status_initializing = ARGV[3]
+
+      -- Once published, status must not expire, even if a refresh races the publish.
+      if redis.call('get', key_master_id) == worker_id and
+         redis.call('get', key_queue_status) == status_initializing then
+        return redis.call('expire', key_queue_status, lease)
+      end
+
+      return 0
+    LUA
+
     STATUS_INITIALIZING = "initializing".freeze
     STATUS_READY = "ready".freeze
     STATUS_SUCCESS = "success".freeze
@@ -102,19 +148,27 @@ module RSpecQ
     # NOTE: jobs will be processed from head to tail (lpop)
     def publish(jobs, fail_fast = 0)
       time = current_time
-      @redis.multi do |pipeline|
-        pipeline.hset(key_queue_config, "fail_fast", fail_fast)
-        pipeline.rpush(key_queue_unprocessed, jobs)
-        pipeline.setnx(key_queue_ready_at, time)
-        pipeline.set(key_queue_status, STATUS_READY)
+      committed = @redis.watch(key_master_id) do |client|
+        master = client.get(key_master_id)
+        if master && master != @worker_id
+          client.unwatch
+          next
+        end
+
+        client.multi do |pipeline|
+          pipeline.hset(key_queue_config, "fail_fast", fail_fast)
+          pipeline.rpush(key_queue_unprocessed, jobs)
+          pipeline.setnx(key_queue_ready_at, time)
+          pipeline.set(key_queue_status, STATUS_READY)
+        end
       end
 
-      jobs.size
+      jobs.size if committed
     end
 
     # Records when a master worker was elected (start of the whole build).
     def mark_elected_master_at
-      @redis.set(key_elected_master_at, current_time)
+      @redis.setnx(key_elected_master_at, current_time)
     end
 
     TRY_MARK_FINISHED = <<~LUA.freeze
@@ -199,8 +253,10 @@ module RSpecQ
         keys: [
           key_queue_unprocessed,
           key_queue_running,
+          key("requeued_job_original_worker"),
+          key_worker_heartbeats,
         ],
-        argv: [@worker_id]
+        argv: [@worker_id, current_time, @worker_liveness_sec]
       )
     end
 
@@ -373,8 +429,22 @@ module RSpecQ
       @redis.hgetall(key_requeues).transform_values(&:to_i)
     end
 
+    # A lease, not a permanent claim: a master that dies before publishing
+    # would otherwise block every other worker until their wait times out.
     def become_master
-      @redis.setnx(key_queue_status, STATUS_INITIALIZING)
+      eval_script(
+        BECOME_MASTER,
+        keys: [key_queue_status, key_master_id],
+        argv: [STATUS_INITIALIZING, @worker_liveness_sec, @worker_id]
+      )
+    end
+
+    def refresh_election_lease
+      eval_script(
+        REFRESH_ELECTION_LEASE,
+        keys: [key_queue_status, key_master_id],
+        argv: [@worker_id, @worker_liveness_sec, STATUS_INITIALIZING]
+      )
     end
 
     # Global timings for scheduling, ordered by execution time desc. Whole-file
@@ -418,14 +488,33 @@ module RSpecQ
       [STATUS_READY, STATUS_SUCCESS, STATUS_FAILURE].include?(@redis.get(key_queue_status))
     end
 
+    # Each election gets the whole wait plus one lease, so a dead master's lease
+    # lapses before its waiters give up, and its replacement isn't left the remainder.
     def wait_until_published(timeout = 30)
-      (timeout * 10).times do
-        return if published?
+      limit = timeout + @worker_liveness_sec
+      master = deadline = nil
+
+      loop do
+        status = @redis.get(key_queue_status)
+        return if [STATUS_READY, STATUS_SUCCESS, STATUS_FAILURE].include?(status)
+
+        if status.nil? && block_given?
+          yield
+          next
+        end
+
+        current = @redis.get(key_master_id)
+        if deadline.nil? || current != master
+          master = current
+          deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + limit
+        end
+
+        if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline
+          raise "Queue not yet published after #{limit} seconds"
+        end
 
         sleep 0.1
       end
-
-      raise "Queue not yet published after #{timeout} seconds"
     end
 
     def build_successful?
@@ -480,6 +569,11 @@ module RSpecQ
       key("queue", "status")
     end
 
+    # redis: STRING <worker_id> — holder of the election lease on queue:status
+    def key_master_id
+      key("queue", "master_id")
+    end
+
     # redis:  HASH<config_key => config_value>
     def key_queue_config
       key("queue", "config")

data/lib/rspecq/version.rb

@@ -1,3 +1,3 @@
 module RSpecQ
-  VERSION = "0.8.1".freeze
+  VERSION = "0.8.2.pre.1".freeze
 end

data/lib/rspecq/worker.rb

@@ -115,9 +115,10 @@ module RSpecQ
       recovered = queue.recover_own_job
       puts "Recovered abandoned job from previous crash: #{recovered}" if recovered
 
-      q_size = try_publish_queue!(queue)
-      puts "Published queue (size=#{q_size})" if q_size
-      queue.wait_until_published(queue_wait_timeout)
+      queue.wait_until_published(queue_wait_timeout) do
+        q_size = try_publish_queue!(queue)
+        puts "Published queue (size=#{q_size})" if q_size
+      end
       queue.save_worker_seed(@worker_id, seed)
 
       # Use `--seed` to deterministically reproduce test failures
@@ -190,9 +191,18 @@ module RSpecQ
         tags.each { |tag| args.push("--tag", tag) }
         opts = RSpec::Core::ConfigurationOptions.new(args)
 
-        _result = RSpec::Core::Runner.new(opts).run($stderr, $stdout)
+        begin
+          RSpec::Core::Runner.new(opts).run($stderr, $stdout)
+        rescue StandardError => e
+          message = "#{e.class}: #{e.message}\n#{Array(e.backtrace).join("\n")}"
+          puts "Error running #{job}: #{message}"
+          queue.record_non_example_error(job, message)
+        ensure
+          # Without this the job stays pinned in the running hash until
+          # requeue_lost_job reclaims it, a full worker_liveness_sec later.
+          queue.acknowledge_job(job)
+        end
 
-        queue.acknowledge_job(job)
         idx += 1
       end
     end
@@ -214,7 +224,30 @@ module RSpecQ
       return if !queue.become_master
 
       queue.mark_elected_master_at
+      stop = Thread::Queue.new
+      refresher = start_lease_refresher(queue, stop)
+
+      begin
+        publish_queue!(queue)
+      ensure
+        stop << true
+        refresher.join
+      end
+    end
+
+    def start_lease_refresher(queue, stop)
+      Thread.new do
+        until stop.pop(timeout: [@heartbeat_frequency, 1].max)
+          begin
+            queue.refresh_election_lease
+          rescue StandardError => e
+            puts "Failed to refresh election lease: #{e.message}"
+          end
+        end
+      end
+    end
 
+    def publish_queue!(queue)
       if reproduction
         q_size = queue.publish(files_or_dirs_to_run, fail_fast)
         log_event(

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rspecq-instructure
 version: !ruby/object:Gem::Version
-  version: 0.8.1
+  version: 0.8.2.pre.1
 platform: ruby
 authors:
 - Agis Anastasopoulos