rspecq 0.2.2 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 900fd0a0742c08ed5a99f922cecb585017109b875d18dbd013d1ed93ea879791
-  data.tar.gz: 67537632a20c43c7da743571c30f5df6eeb826164ed6d9dc7b20ddb4d8761bf3
+  metadata.gz: 4337137edc0f3ee2c22d4f9cd2547ab31d4ddc0b043b49848c7b513ae98094a9
+  data.tar.gz: 6de02e1ce31cb7dff6af9d4862f1dceebfbfbbd10063329c9d3d058d32bf8f06
 SHA512:
-  metadata.gz: 42d517b6bfb37eb6bf12bef093e6db9bbfdd207155c7a30b7bf64d6dc114cde27f503323aa9b7e940a0a783db40fbb396c0567a780ee93f303291c083df29ddd
-  data.tar.gz: a6fe68d291eb761202d366c9fccb7aeee387b6dad048ec2efb9fb81b705e8cd0826fc339cf447d2ff272d9cbd1e8b21d060181b3cc7d74539f8c911ee466533a
+  metadata.gz: beb1b013a1d78e8c18b473ef33d286224a92ef46ae3944f99296cc3d2d13193309b0efb020307fa3e9139b7fd88eb9a9111ad526026762ccf0b17ba892b3ddb4
+  data.tar.gz: 178481fe5b707cefa25426ec6b2ead691dd69494a6b9869eaf8e438b78e100c5dd8c0c9caab2dd4b792c18e5c931e4e7231cc971b77fa048c14420f33978fe2a

data/CHANGELOG.md

@@ -4,12 +4,52 @@ Breaking changes are prefixed with a "[BREAKING]" label.
 
 ## master (unreleased)
 
+## 0.7.0 (2021-04-1)
+
+- New cli parameter `reproduction`.
+  When passed, primary worker publishes the queue in the same order as passed
+  in the command.
+- Reporter now includes a reproduction command for flaky tests.
+
+## 0.6.0 (2021-03-23)
+
+- New cli parameter `seed`.
+  The seed is passed to the RSpec command.
+
+## 0.5.0 (2021-02-05)
+
+### Added
+
+- New cli parameter `queue_wait_timeout`.
+  It configured the time a queue can wait to be ready. The env equivalent
+  is `RSPECQ_QUEUE_WAIT_TIMEOUT`. [#51](https://github.com/skroutz/rspecq/pull/51)
+
+## 0.4.0 (2020-10-07)
+
+### Added
+
+- Builds can be configured to terminate after a specified number of failures,
+  using the `--fail-fast` option.
+
+
+## 0.3.0 (2020-10-05)
+
+### Added
+
+- Providing a Redis URL is now possible using the `--redis-url` option
+  [[#40](https://github.com/skroutz/rspecq/pull/40)]
+
+### Changed
+
+- [DEPRECATION] The `--redis` option is now deprecated. Use `--redis-host`
+  instead [[#40](https://github.com/skroutz/rspecq/pull/40)]
+
 ## 0.2.2 (2020-09-10)
 
 ### Fixed
-- Worker would fail if application code was writing to stderr 
+- Worker would fail if application code was writing to stderr
  [[#35](https://github.com/skroutz/rspecq/pull/35)]
-  
+
 ## 0.2.1 (2020-09-09)
 
 ### Changed

data/README.md

@@ -26,8 +26,8 @@ and [ci-queue](https://github.com/Shopify/ci-queue).
 - Handles intermittent worker failures (e.g. network hiccups, faulty hardware etc.)
   by detecting non-responsive workers and requeing their jobs. See [*Worker failures*](#worker-failures)
 - Sentry integration for monitoring build-level events. See [*Sentry integration*](#sentry-integration).
-- [PLANNED] StatsD integration for various build-level metrics and insights.
   See [#2](https://github.com/skroutz/rspecq/issues/2).
+- Automatic termination of builds after a certain amount of failures. See [*Fail-fast*](#fail-fast).
 
 ## Usage
 
@@ -64,17 +64,46 @@ USAGE:
 OPTIONS:
     -b, --build ID                   A unique identifier for the build. Should be common among workers participating in the same build.
     -w, --worker ID                  An identifier for the worker. Workers participating in the same build should have distinct IDs.
-    -r, --redis HOST                 Redis host to connect to (default: 127.0.0.1).
+        --seed SEED                  The RSpec seed. Passing the seed can be helpful in many ways i.e reproduction and testing.
+    -r, --redis HOST                 --redis is deprecated. Use --redis-host or --redis-url instead. Redis host to connect to (default: 127.0.0.1).
+        --redis-host HOST            Redis host to connect to (default: 127.0.0.1).
+        --redis-url URL              Redis URL to connect to (e.g.: redis://127.0.0.1:6379/0).
         --update-timings             Update the global job timings key with the timings of this build. Note: This key is used as the basis for job scheduling.
         --file-split-threshold N     Split spec files slower than N seconds and schedule them as individual examples.
         --report                     Enable reporter mode: do not pull tests off the queue; instead print build progress and exit when it's finished.
                                      Exits with a non-zero status code if there were any failures.
         --report-timeout N           Fail if build is not finished after N seconds. Only applicable if --report is enabled (default: 3600).
         --max-requeues N             Retry failed examples up to N times before considering them legit failures (default: 3).
+        --queue-wait-timeout N       Time to wait for a queue to be ready before considering it failed (default: 30).
+        --fail-fast N                Abort build with a non-zero status code after N failed examples.
+        --reproduction               Enable reproduction mode: Publish files and examples in the exact order given in the command. Incompatible with --timings.
     -h, --help                       Show this message.
     -v, --version                    Print the version and exit.
 ```
 
+You can set most options using ENV variables:
+
+```shell
+$ RSPECQ_BUILD=123 RSPECQ_WORKDER=foo1 rspecq spec/
+```
+
+### Supported ENV variables
+
+| Name | Desc |
+| --- | --- |
+| `RSPECQ_BUILD` | Build ID |
+| `RSPECQ_WORKER` | Worker ID |
+| `RSPECQ_SEED` | RSpec seed |
+| `RSPECQ_REDIS` | Redis HOST |
+| `RSPECQ_UPDATE_TIMINGS` | Timings |
+| `RSPECQ_FILE_SPLIT_THRESHOLD` | File split threshold |
+| `RSPECQ_REPORT` | Report |
+| `RSPECQ_REPORT_TIMEOUT` | Report Timeout |
+| `RSPECQ_MAX_REQUEUES` | Max requests |
+| `RSPECQ_QUEUE_WAIT_TIMEOUT` | Queue wait timeout |
+| `RSPECQ_REDIS_URL` | Redis URL |
+| `RSPECQ_FAIL_FAST` | Fail fast |
+
 ### Sentry integration
 
 RSpecQ can optionally emit build events to a
@@ -86,7 +115,6 @@ This is convenient for monitoring important warnings/errors that may impact
 build times, such as the fact that no previous timings were found and
 therefore job scheduling was effectively random for a particular build.
 
-
 ## How it works
 
 The core design is almost identical to ci-queue so please refer to its
@@ -113,7 +141,7 @@ For example, a single file may need 10 minutes to run while all other
 files finish after 8 minutes. This would cause all but one workers to be
 sitting idle for 2 minutes.
 
-To overcome this issue, RSpecQ can splits files which their execution time is
+To overcome this issue, RSpecQ can split files which their execution time is
 above a certain threshold (set with the `--file-split-threshold` option)
 and instead schedule them as individual examples.
 
@@ -131,6 +159,16 @@ final report.
 Flaky tests are also detected and printed as such in the final report. They are
 also emitted to Sentry (see [Sentry integration](#sentry-integration)).
 
+### Fail-fast
+
+In order to prevent large suites running for a long time with a lot of
+failures, a threshold can be set to control the number of failed examples that
+will render the build unsuccessful. This is in par with RSpec's
+[--fail-fast](https://relishapp.com/rspec/rspec-core/docs/command-line/fail-fast-option).
+
+This feature is disabled by default, and can be controlled via the
+`--fail-fast` command line option.
+
 ### Worker failures
 
 It's not uncommon for CI processes to encounter unrecoverable failures for
@@ -202,6 +240,13 @@ To enable verbose output in the tests:
 $ RSPECQ_DEBUG=1 bundle exec rake
 ```
 
+## Redis
+
+RSpecQ by design doesn't expire its keys from Redis. It is left to the user
+to configure the Redis server to do so; see
+[Using Redis as an LRU cache](https://redis.io/topics/lru-cache) for more info.
+
+You can do this from a configuration file or with `redis-cli`.
 
 ## License
 

data/Rakefile

@@ -2,7 +2,7 @@ require "rake/testtask"
 
 Rake::TestTask.new do |t|
   t.libs << "test"
-  t.test_files = FileList['test/test_*.rb']
+  t.test_files = FileList["test/test_*.rb"]
   t.verbose = true
 end
 

data/bin/rspecq

@@ -2,9 +2,11 @@
 require "optparse"
 require "rspecq"
 
-DEFAULT_REDIS_HOST = "127.0.0.1"
+DEFAULT_REDIS_HOST = "127.0.0.1".freeze
 DEFAULT_REPORT_TIMEOUT = 3600 # 1 hour
 DEFAULT_MAX_REQUEUES = 3
+DEFAULT_QUEUE_WAIT_TIMEOUT = 30
+DEFAULT_FAIL_FAST = 0
 
 def env_set?(var)
   ["1", "true"].include?(ENV[var])
@@ -36,11 +38,27 @@ OptionParser.new do |o|
     opts[:worker] = v
   end
 
+  o.on("--seed SEED", "The RSpec seed. Passing the seed can be helpful in " \
+    "many ways i.e reproduction and testing.") do |v|
+    opts[:seed] = v
+  end
+
   o.on("-r", "--redis HOST", "Redis host to connect to " \
        "(default: #{DEFAULT_REDIS_HOST}).") do |v|
+    puts "DEPRECATION: --redis is deprecated. Use --redis-host or --redis-url instead"
     opts[:redis_host] = v
   end
 
+  o.on("--redis-host HOST", "Redis host to connect to " \
+       "(default: #{DEFAULT_REDIS_HOST}).") do |v|
+    opts[:redis_host] = v
+  end
+
+  o.on("--redis-url URL", "The URL of the Redis host to connect to " \
+       "(e.g.: redis://127.0.0.1:6379/0).") do |v|
+    opts[:redis_url] = v
+  end
+
   o.on("--update-timings", "Update the global job timings key with the "     \
        "timings of this build. Note: This key is used as the basis for job " \
        "scheduling.") do |v|
@@ -54,7 +72,7 @@ OptionParser.new do |o|
 
   o.on("--report", "Enable reporter mode: do not pull tests off the queue; " \
                    "instead print build progress and exit when it's "        \
-                   "finished.\n#{o.summary_indent*9} "                       \
+                   "finished.\n#{o.summary_indent * 9} "                       \
                    "Exits with a non-zero status code if there were any "    \
                    "failures.") do |v|
     opts[:report] = v
@@ -72,6 +90,23 @@ OptionParser.new do |o|
     opts[:max_requeues] = v
   end
 
+  o.on("--queue-wait-timeout N", Integer, "Time to wait for a queue to be "   \
+       "ready before considering it failed "                                  \
+       "(default: #{DEFAULT_QUEUE_WAIT_TIMEOUT}).") do |v|
+    opts[:queue_wait_timeout] = v
+  end
+
+  o.on("--fail-fast N", Integer, "Abort build with a non-zero status code " \
+       "after N failed examples.") do |v|
+    opts[:fail_fast] = v
+  end
+
+  o.on("--reproduction", "Enable reproduction mode: run rspec on the given files " \
+       "and examples in the exact order they are given. Incompatible with " \
+       "--timings.") do |v|
+    opts[:reproduction] = v
+  end
+
   o.on_tail("-h", "--help", "Show this message.") do
     puts o
     exit
@@ -85,21 +120,37 @@ end.parse!
 
 opts[:build] ||= ENV["RSPECQ_BUILD"]
 opts[:worker] ||= ENV["RSPECQ_WORKER"]
+opts[:seed] ||= ENV["RSPECQ_SEED"]
 opts[:redis_host] ||= ENV["RSPECQ_REDIS"] || DEFAULT_REDIS_HOST
 opts[:timings] ||= env_set?("RSPECQ_UPDATE_TIMINGS")
-opts[:file_split_threshold] ||= Integer(ENV["RSPECQ_FILE_SPLIT_THRESHOLD"] || 9999999)
+opts[:file_split_threshold] ||= Integer(ENV["RSPECQ_FILE_SPLIT_THRESHOLD"] || 9_999_999)
 opts[:report] ||= env_set?("RSPECQ_REPORT")
 opts[:report_timeout] ||= Integer(ENV["RSPECQ_REPORT_TIMEOUT"] || DEFAULT_REPORT_TIMEOUT)
 opts[:max_requeues] ||= Integer(ENV["RSPECQ_MAX_REQUEUES"] || DEFAULT_MAX_REQUEUES)
+opts[:queue_wait_timeout] ||= Integer(ENV["RSPECQ_QUEUE_WAIT_TIMEOUT"] || DEFAULT_QUEUE_WAIT_TIMEOUT)
+opts[:redis_url] ||= ENV["RSPECQ_REDIS_URL"]
+opts[:fail_fast] ||= Integer(ENV["RSPECQ_FAIL_FAST"] || DEFAULT_FAIL_FAST)
+opts[:reproduction] ||= env_set?("RSPECQ_REPRODUCTION")
 
+# rubocop:disable Style/RaiseArgs, Layout/EmptyLineAfterGuardClause
 raise OptionParser::MissingArgument.new(:build) if opts[:build].nil?
 raise OptionParser::MissingArgument.new(:worker) if !opts[:report] && opts[:worker].nil?
+# rubocop:enable Style/RaiseArgs, Layout/EmptyLineAfterGuardClause
+
+redis_opts = {}
+
+if opts[:redis_url]
+  redis_opts[:url] = opts[:redis_url]
+else
+  redis_opts[:host] = opts[:redis_host]
+end
 
 if opts[:report]
   reporter = RSpecQ::Reporter.new(
     build_id: opts[:build],
     timeout: opts[:report_timeout],
-    redis_host: opts[:redis_host],
+    redis_opts: redis_opts,
+    queue_wait_timeout: opts[:queue_wait_timeout]
   )
 
   reporter.report
@@ -107,12 +158,16 @@ else
   worker = RSpecQ::Worker.new(
     build_id: opts[:build],
     worker_id: opts[:worker],
-    redis_host: opts[:redis_host]
+    redis_opts: redis_opts
   )
 
-  worker.files_or_dirs_to_run = ARGV[0] if ARGV[0]
+  worker.files_or_dirs_to_run = ARGV if ARGV.any?
   worker.populate_timings = opts[:timings]
   worker.file_split_threshold = opts[:file_split_threshold]
   worker.max_requeues = opts[:max_requeues]
+  worker.queue_wait_timeout = opts[:queue_wait_timeout]
+  worker.fail_fast = opts[:fail_fast]
+  worker.seed = Integer(opts[:seed]) if opts[:seed]
+  worker.reproduction = opts[:reproduction]
   worker.work
 end

data/lib/rspecq/formatters/failure_recorder.rb

@@ -1,12 +1,18 @@
 module RSpecQ
   module Formatters
+    # Persists failed examples information (i.e. message and backtrace), so
+    # that they can be reported to the end user by the Reporter.
+    #
+    # Also persists non-example error information (e.g. a syntax error that
+    # in a spec file).
     class FailureRecorder
-      def initialize(queue, job, max_requeues)
+      def initialize(queue, job, max_requeues, worker_id)
         @queue = queue
         @job = job
         @colorizer = RSpec::Core::Formatters::ConsoleCodes
         @non_example_error_recorded = false
         @max_requeues = max_requeues
+        @worker_id = worker_id
       end
 
       # Here we're notified about errors occuring outside of examples.
@@ -25,7 +31,7 @@ module RSpecQ
       def example_failed(notification)
         example = notification.example
 
-        if @queue.requeue_job(example.id, @max_requeues)
+        if @queue.requeue_job(example, @max_requeues, @worker_id)
           # HACK: try to avoid picking the job we just requeued; we want it
           # to be picked up by a different worker
           sleep 0.5
@@ -33,16 +39,19 @@ module RSpecQ
         end
 
         presenter = RSpec::Core::Formatters::ExceptionPresenter.new(
-          example.exception, example)
+          example.exception, example
+        )
 
         msg = presenter.fully_formatted(nil, @colorizer)
         msg << "\n"
         msg << @colorizer.wrap(
-          "bin/rspec #{example.location_rerun_argument}",
-          RSpec.configuration.failure_color)
+          "bin/rspec --seed #{RSpec.configuration.seed} #{example.location_rerun_argument}",
+          RSpec.configuration.failure_color
+        )
 
         msg << @colorizer.wrap(
-          " # #{example.full_description}", RSpec.configuration.detail_color)
+          " # #{example.full_description}", RSpec.configuration.detail_color
+        )
 
         @queue.record_example_failure(notification.example.id, msg)
       end

data/lib/rspecq/formatters/job_timing_recorder.rb

@@ -1,5 +1,8 @@
 module RSpecQ
   module Formatters
+    # Persists each job's timing (in seconds). Those timings are used when
+    # determining the ordering in which jobs are scheduled (slower jobs will
+    # be enqueued first).
     class JobTimingRecorder
       def initialize(queue, job)
         @queue = queue

data/lib/rspecq/formatters/worker_heartbeat_recorder.rb

@@ -14,4 +14,3 @@ module RSpecQ
     end
   end
 end
-

data/lib/rspecq/queue.rb

@@ -51,8 +51,12 @@ module RSpecQ
     REQUEUE_JOB = <<~LUA.freeze
       local key_queue_unprocessed = KEYS[1]
       local key_requeues = KEYS[2]
+      local key_requeued_job_original_worker = KEYS[3]
+      local key_job_location = KEYS[4]
       local job = ARGV[1]
       local max_requeues = ARGV[2]
+      local original_worker = ARGV[3]
+      local location = ARGV[4]
 
       local requeued_times = redis.call('hget', key_requeues, job)
       if requeued_times and requeued_times >= max_requeues then
@@ -60,7 +64,9 @@ module RSpecQ
       end
 
       redis.call('lpush', key_queue_unprocessed, job)
+      redis.call('hset', key_requeued_job_original_worker, job, original_worker)
       redis.call('hincrby', key_requeues, job, 1)
+      redis.call('hset', key_job_location, job, location)
 
       return true
     LUA
@@ -70,15 +76,16 @@ module RSpecQ
 
     attr_reader :redis
 
-    def initialize(build_id, worker_id, redis_host)
+    def initialize(build_id, worker_id, redis_opts)
       @build_id = build_id
       @worker_id = worker_id
-      @redis = Redis.new(host: redis_host, id: worker_id)
+      @redis = Redis.new(redis_opts.merge(id: worker_id))
     end
 
     # NOTE: jobs will be processed from head to tail (lpop)
-    def publish(jobs)
+    def publish(jobs, fail_fast = 0)
       @redis.multi do
+        @redis.hset(key_queue_config, "fail_fast", fail_fast)
         @redis.rpush(key_queue_unprocessed, jobs)
         @redis.set(key_queue_status, STATUS_READY)
       end.first
@@ -116,6 +123,7 @@ module RSpecQ
       @redis.multi do
         @redis.hdel(key_queue_running, @worker_id)
         @redis.sadd(key_queue_processed, job)
+        @redis.rpush(key("queue", "jobs_per_worker", @worker_id), job)
       end
     end
 
@@ -124,16 +132,41 @@ module RSpecQ
     #
     # Returns nil if the job hit the requeue limit and therefore was not
     # requeued and should be considered a failure.
-    def requeue_job(job, max_requeues)
+    def requeue_job(example, max_requeues, original_worker_id)
       return false if max_requeues.zero?
 
+      job = example.id
+      location = example.location_rerun_argument
+
       @redis.eval(
         REQUEUE_JOB,
-        keys: [key_queue_unprocessed, key_requeues],
-        argv: [job, max_requeues],
+        keys: [key_queue_unprocessed, key_requeues, key("requeued_job_original_worker"), key("job_location")],
+        argv: [job, max_requeues, original_worker_id, location]
       )
     end
 
+    def save_worker_seed(worker, seed)
+      @redis.hset(key("worker_seed"), worker, seed)
+    end
+
+    def job_location(job)
+      @redis.hget(key("job_location"), job)
+    end
+
+    def failed_job_worker(job)
+      redis.hget(key("requeued_job_original_worker"), job)
+    end
+
+    def job_rerun_command(job)
+      worker = failed_job_worker(job)
+      jobs = redis.lrange(key("queue", "jobs_per_worker", worker), 0, -1)
+      seed = redis.hget(key("worker_seed"), worker)
+
+      "DISABLE_SPRING=1 DISABLE_BOOTSNAP=1 bin/rspecq --build 1 " \
+        "--worker foo --seed #{seed} --max-requeues 0 --fail-fast 1 " \
+        "--reproduction #{jobs.join(' ')}"
+    end
+
     def record_example_failure(example_id, message)
       @redis.hset(key_failures, example_id, message)
     end
@@ -209,9 +242,10 @@ module RSpecQ
       @redis.get(key_queue_status) == STATUS_READY
     end
 
-    def wait_until_published(timeout=30)
+    def wait_until_published(timeout = 30)
       (timeout * 10).times do
         return if published?
+
         sleep 0.1
       end
 
@@ -232,7 +266,9 @@ module RSpecQ
     # after being retried). Must be called after the build is complete,
     # otherwise an exception will be raised.
     def flaky_jobs
-      raise "Queue is not yet exhausted" if !exhausted?
+      if !exhausted? && !build_failed_fast?
+        raise "Queue is not yet exhausted"
+      end
 
       requeued = @redis.hkeys(key_requeues)
 
@@ -241,11 +277,38 @@ module RSpecQ
       requeued - @redis.hkeys(key_failures)
     end
 
+    # Returns the number of failures that will trigger the build to fail-fast.
+    # Returns 0 if this feature is disabled and nil if the Queue is not yet
+    # published
+    def fail_fast
+      return nil unless published?
+
+      @fail_fast ||= Integer(@redis.hget(key_queue_config, "fail_fast"))
+    end
+
+    # Returns true if the number of failed tests, has surpassed the threshold
+    # to render the run unsuccessful and the build should be terminated.
+    def build_failed_fast?
+      if fail_fast.nil? || fail_fast.zero?
+        return false
+      end
+
+      @redis.multi do
+        @redis.hlen(key_failures)
+        @redis.hlen(key_errors)
+      end.inject(:+) >= fail_fast
+    end
+
     # redis: STRING [STATUS_INITIALIZING, STATUS_READY]
     def key_queue_status
       key("queue", "status")
     end
 
+    # redis:  HASH<config_key => config_value>
+    def key_queue_config
+      key("queue", "config")
+    end
+
     # redis: LIST<job>
     def key_queue_unprocessed
       key("queue", "unprocessed")

data/lib/rspecq/reporter.rb

@@ -9,18 +9,19 @@ module RSpecQ
   #
   # Reporters are readers of the queue.
   class Reporter
-    def initialize(build_id:, timeout:, redis_host:)
+    def initialize(build_id:, timeout:, redis_opts:, queue_wait_timeout: 30)
       @build_id = build_id
       @timeout = timeout
-      @queue = Queue.new(build_id, "reporter", redis_host)
+      @queue = Queue.new(build_id, "reporter", redis_opts)
+      @queue_wait_timeout = queue_wait_timeout
 
       # We want feedback to be immediattely printed to CI users, so
       # we disable buffering.
-      STDOUT.sync = true
+      $stdout.sync = true
     end
 
     def report
-      @queue.wait_until_published
+      @queue.wait_until_published(@queue_wait_timeout)
 
       finished = false
 
@@ -28,7 +29,7 @@ module RSpecQ
       failure_heading_printed = false
 
       tests_duration = measure_duration do
-        @timeout.times do |i|
+        @timeout.times do
           @queue.example_failures.each do |job, rspec_output|
             next if reported_failures[job]
 
@@ -41,7 +42,7 @@ module RSpecQ
             puts failure_formatted(rspec_output)
           end
 
-          if !@queue.exhausted?
+          unless @queue.exhausted? || @queue.build_failed_fast?
             sleep 1
             next
           end
@@ -83,6 +84,13 @@ module RSpecQ
       end
 
       summary = ""
+      if @queue.build_failed_fast?
+        summary << "\n\n"
+        summary << "The limit of #{@queue.fail_fast} failures has been reached\n"
+        summary << "Aborting..."
+        summary << "\n"
+      end
+
       summary << failed_examples_section if !failures.empty?
 
       errors.each { |_job, msg| summary << msg }
@@ -99,7 +107,14 @@ module RSpecQ
       if !flaky_jobs.empty?
         summary << "\n\n"
         summary << "Flaky jobs detected (count=#{flaky_jobs.count}):\n"
-        flaky_jobs.each { |j| summary << "  #{j}\n" }
+        flaky_jobs.each do |j|
+          summary << RSpec::Core::Formatters::ConsoleCodes.wrap(
+            "#{@queue.job_location(j)} @ #{@queue.failed_job_worker(j)}\n",
+            RSpec.configuration.pending_color
+          )
+
+          summary << "#{@queue.job_rerun_command(j)}\n\n\n"
+        end
       end
 
       summary
@@ -117,16 +132,15 @@ module RSpecQ
       return if jobs.empty?
 
       jobs.each do |job|
-        filename = job.sub(/\[.+\]/, '')
+        filename = job.sub(/\[.+\]/, "")[%r{spec/.+}].split(":")[0]
 
         extra = {
           build: @build_id,
           build_timeout: @timeout,
-          queue: @queue.inspect,
-          object: self.inspect,
-          pid: Process.pid,
-          job_path: job,
-          build_duration: build_duration
+          build_duration: build_duration,
+          location: @queue.job_location(job),
+          rerun_command: @queue.job_rerun_command(job),
+          worker: @queue.failed_job_worker(job)
         }
 
         tags = {
@@ -136,7 +150,7 @@ module RSpecQ
 
         Raven.capture_message(
           "Flaky test in #{filename}",
-          level: 'warning',
+          level: "warning",
           extra: extra,
           tags: tags
         )

data/lib/rspecq/version.rb

@@ -1,3 +1,3 @@
 module RSpecQ
-  VERSION = "0.2.2".freeze
+  VERSION = "0.7.0".freeze
 end

data/lib/rspecq/worker.rb

@@ -40,17 +40,39 @@ module RSpecQ
     # Defaults to 3
     attr_accessor :max_requeues
 
+    # Stop the execution after N failed tests. Do not stop at any point when
+    # set to 0.
+    #
+    # Defaults to 0
+    attr_accessor :fail_fast
+
+    # Time to wait for a queue to be published.
+    #
+    # Defaults to 30
+    attr_accessor :queue_wait_timeout
+
+    # The RSpec seed
+    attr_accessor :seed
+
+    # Reproduction flag. If true, worker will publish files in the exact order
+    # given in the command.
+    attr_accessor :reproduction
+
     attr_reader :queue
 
-    def initialize(build_id:, worker_id:, redis_host:)
+    def initialize(build_id:, worker_id:, redis_opts:)
       @build_id = build_id
       @worker_id = worker_id
-      @queue = Queue.new(build_id, worker_id, redis_host)
+      @queue = Queue.new(build_id, worker_id, redis_opts)
+      @fail_fast = 0
       @files_or_dirs_to_run = "spec"
       @populate_timings = false
-      @file_split_threshold = 999999
+      @file_split_threshold = 999_999
       @heartbeat_updated_at = nil
       @max_requeues = 3
+      @queue_wait_timeout = 30
+      @seed = srand && srand % 0xFFFF
+      @reproduction = false
 
       RSpec::Core::Formatters.register(Formatters::JobTimingRecorder, :dump_summary)
       RSpec::Core::Formatters.register(Formatters::ExampleCountRecorder, :dump_summary)
@@ -62,13 +84,16 @@ module RSpecQ
       puts "Working for build #{@build_id} (worker=#{@worker_id})"
 
       try_publish_queue!(queue)
-      queue.wait_until_published
+      queue.wait_until_published(queue_wait_timeout)
+      queue.save_worker_seed(@worker_id, seed)
 
       loop do
         # we have to bootstrap this so that it can be used in the first call
         # to `requeue_lost_job` inside the work loop
         update_heartbeat
 
+        return if queue.build_failed_fast?
+
         lost = queue.requeue_lost_job
         puts "Requeued lost job: #{lost}" if lost
 
@@ -88,9 +113,9 @@ module RSpecQ
 
         # reconfigure rspec
         RSpec.configuration.detail_color = :magenta
-        RSpec.configuration.seed = srand && srand % 0xFFFF
-        RSpec.configuration.backtrace_formatter.filter_gem('rspecq')
-        RSpec.configuration.add_formatter(Formatters::FailureRecorder.new(queue, job, max_requeues))
+        RSpec.configuration.seed = seed
+        RSpec.configuration.backtrace_formatter.filter_gem("rspecq")
+        RSpec.configuration.add_formatter(Formatters::FailureRecorder.new(queue, job, max_requeues, @worker_id))
         RSpec.configuration.add_formatter(Formatters::ExampleCountRecorder.new(queue))
         RSpec.configuration.add_formatter(Formatters::WorkerHeartbeatRecorder.new(self))
 
@@ -116,12 +141,21 @@ module RSpecQ
     def try_publish_queue!(queue)
       return if !queue.become_master
 
+      if reproduction
+        q_size = queue.publish(files_or_dirs_to_run, fail_fast)
+        log_event(
+          "Reproduction mode. Published queue as given (size=#{q_size})",
+          "info"
+        )
+        return
+      end
+
       RSpec.configuration.files_or_directories_to_run = files_or_dirs_to_run
       files_to_run = RSpec.configuration.files_to_run.map { |j| relative_path(j) }
 
       timings = queue.timings
       if timings.empty?
-        q_size = queue.publish(files_to_run.shuffle)
+        q_size = queue.publish(files_to_run.shuffle, fail_fast)
         log_event(
           "No timings found! Published queue in random order (size=#{q_size})",
           "warning"
@@ -146,7 +180,7 @@ module RSpecQ
         jobs.concat(files_to_run)
       end
 
-      default_timing = timings.values[timings.values.size/2]
+      default_timing = timings.values[timings.values.size / 2]
 
       # assign timings (based on previous runs) to all jobs
       jobs = jobs.each_with_object({}) do |j, h|
@@ -160,7 +194,7 @@ module RSpecQ
       # sort jobs based on their timings (slowest to be processed first)
       jobs = jobs.sort_by { |_j, t| -t }.map(&:first)
 
-      puts "Published queue (size=#{queue.publish(jobs)})"
+      puts "Published queue (size=#{queue.publish(jobs, fail_fast)})"
     end
 
     private
@@ -171,7 +205,8 @@ module RSpecQ
       # see https://github.com/rspec/rspec-core/pull/2723
       if Gem::Version.new(RSpec::Core::Version::STRING) <= Gem::Version.new("3.9.1")
         RSpec.world.instance_variable_set(
-          :@example_group_counts_by_spec_file, Hash.new(0))
+          :@example_group_counts_by_spec_file, Hash.new(0)
+        )
       end
 
       # RSpec.clear_examples does not reset those, which causes issues when
@@ -195,17 +230,17 @@ module RSpecQ
 
       if !cmd_result.success?
         rspec_output = begin
-                         JSON.parse(out)
-                       rescue JSON::ParserError
-                         out
-                       end
+          JSON.parse(out)
+        rescue JSON::ParserError
+          out
+        end
 
         log_event(
           "Failed to split slow files, falling back to regular scheduling.\n #{err}",
           "error",
           rspec_stdout: rspec_output,
           rspec_stderr: err,
-          cmd_result: cmd_result.inspect,
+          cmd_result: cmd_result.inspect
         )
 
         pp rspec_output
@@ -227,7 +262,7 @@ module RSpecQ
 
     # Prints msg to standard output and emits an event to Sentry, if the
     # SENTRY_DSN environment variable is set.
-    def log_event(msg, level, additional={})
+    def log_event(msg, level, additional = {})
       puts msg
 
       Raven.capture_message(msg, level: level, extra: {
@@ -238,8 +273,8 @@ module RSpecQ
         populate_timings: populate_timings,
         file_split_threshold: file_split_threshold,
         heartbeat_updated_at: @heartbeat_updated_at,
-        object: self.inspect,
-        pid: Process.pid,
+        object: inspect,
+        pid: Process.pid
       }.merge(additional))
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rspecq
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.7.0
 platform: ruby
 authors:
 - Agis Anastasopoulos
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-09-10 00:00:00.000000000 Z
+date: 2021-04-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec-core
@@ -53,7 +53,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -81,7 +81,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: minitest
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -108,6 +108,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.93.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.93.0
 description: 
 email: agis.anast@gmail.com
 executables: