paraspec 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 709786de26ffdac4d238bb0a1c531a654b7183386b0f85f0c523e74958f66f33
+  data.tar.gz: aa00e1665689abead274c8259a1ab55e1fc049b9e16029723e8c19269248d884
+SHA512:
+  metadata.gz: 70f1e59627cc4d63ae6653b0f945831d9d7abac73cc998c76b1f8c36100e5b1a1f4578731c091904086a159be9ff4d3a16f24702f7847eb46cffac2c39947d4b
+  data.tar.gz: 06d5f5746820455ebe7fc0d440cc8f335a8c8ca6177a5d1a36afc0691d0854a9b2df31eb27b87726470be35139e7fdece0d2315e3bbbec07966ba4e7a0ae7801

data/LICENSE

@@ -0,0 +1,23 @@
+The MIT License (MIT)
+=====================
+
+Copyright (c) 2018 Oleg Pudeyev
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,97 @@
+# Paraspec
+
+Paraspec is a parallel RSpec test runner.
+
+It is built with a producer/consumer architecture. A master process loads
+the entire test suite and sets up a queue to feed the tests to the workers.
+Each worker requests a test from the master, runs it, reports the results
+back to the master and requests the next test until there are no more left.
+
+This producer/consumer architecture enables a number of features:
+
+1. The worker load is naturally balanced. If a worker happens to come across
+a slow test, the other workers keep chugging away at faster tests.
+2. Tests defined in a single file can be executed by multiple workers,
+since paraspec operates on a test by test basis and not on a file by file basis.
+3. Standard output and error streams can be[*] captured and grouped on a
+test by test basis, avoiding interleaving output of different tests together.
+This output capture can be performed for output generated by C extensions
+as well as plain Ruby code.
+4. Test results are seamlessly integrated by the master, such that
+a parallel run produces a single progress bar with Fuubar across all workers.
+
+[*] This feature is not yet implemented.
+
+## Usage
+
+For a test suite with no external dependencies, using paraspec is
+trivially easy. Just run:
+
+    paraspec
+
+To specify concurrency manually:
+
+    paraspec -c 4
+
+To pass options to rspec, for example to filter examples to run:
+
+    paraspec -- -e 'My test'
+    paraspec -- spec/my_spec.rb
+
+For a test suite with external dependencies, paraspec sets the
+`TEST_ENV_NUMBER` environment variable, like
+[parallel_tests](https://github.com/grosser/parallel_tests) does.
+The test suite can then configure itself differently in each worker.
+
+By default the master process doesn't have `TEST_ENV_NUMBER` set.
+To have that set to `1` use `--master-is-1` option to paraspec:
+
+    paraspec --master-is-1
+
+## Advanced Usage
+
+### Formatters
+
+Paraspec works with any RSpec formatter, and supports multiple formatters
+just like RSpec does. If your test suite is big enough for parallel execution
+to make a difference, chances are the default progress and documentation
+formatters aren't too useful for dealing with its output.
+
+I recommend [Fuubar](https://github.com/thekompanee/fuubar) and
+[RSpec JUnit Formatter](https://github.com/sj26/rspec_junit_formatter)
+configured at the same time. Fuubar produces a very nice looking progress bar
+plus it prints failures and exceptions to the terminal as soon as they
+occur. JUnit output, passed through a JUnit XML to HTML converter like
+[junit2html](https://gitlab.com/inorton/junit2html), is much handier
+than going through terminal output when a run produces 100 or 1000
+failing tests.
+
+### Debugging
+
+Paraspec offers several debugging aids. The first one is the terminal option:
+
+    paraspec -T
+
+This option makes paraspec stay attached to the terminal it was
+launched in, making it possible to insert e.g. `byebug` calls in supervisor,
+master or worker code as well as anywhere in the test suite being executed
+and have byebug work. Setting this option also removes internal timeouts
+on interprocess waits and sets concurrency to 1, however concurrency
+can be reset with a subsequent `-c` option:
+
+    paraspec -T -c 2
+
+Paraspec can produce copious debugging output in several facilities.
+The debugging output is turned on with `-d`/`--debug` option:
+
+    paraspec -d state   # supervisor, master, worker state transitions
+    paraspec -d ipc     # IPC requests and responses
+    paraspec -d perf    # timing & performance information
+
+## Bugs & Patches
+
+Please report via issues and pull requests.
+
+## License
+
+MIT

data/bin/paraspec

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby
+
+$: << File.join(File.dirname(__FILE__), '..', 'lib')
+
+require 'paraspec'
+require 'optparse'
+
+options = {}
+OptionParser.new do |opts|
+  opts.banner = 'Usage: paraspec [options] [-- [rspec-options]...]'
+
+  opts.on('-c', '--concurrency=NUM', 'Number of concurrent workers to use') do |v|
+    if v.to_i == 0
+      raise "Invalid concurrency value: #{v}"
+    end
+    options[:concurrency] = v.to_i
+  end
+
+  opts.on('-d', '--debug=SUBSYSTEM', 'Output debugging information for SUBSYSTEM') do |v|
+    options[:"debug_#{v}"] = true
+    options[:debug] = true
+  end
+
+  opts.on('-T', '--terminal', 'Retain controlling terminal (debug only)') do |v|
+    options[:terminal] = v
+    options[:concurrency] = 1
+  end
+
+  opts.on('--master-is-1', 'Set TEST_ENV_NUMBER=1 in master process') do |v|
+    options[:master_is_1] = v
+  end
+end.parse!
+
+if options[:debug]
+  Paraspec.logger.level = Logger::DEBUG
+end
+
+=begin
+files = if ARGV.length > 0
+  ARGV
+else
+  ['spec']
+end
+RSpec.configuration.files_or_directories_to_run = files
+=end
+
+%w(ipc state perf).each do |subsystem|
+  if options.delete(:"debug_#{subsystem}")
+    Paraspec.logger.send("log_#{subsystem}=", true)
+  end
+end
+supervisor = Paraspec::Supervisor.new(options)
+supervisor.run

data/lib/paraspec.rb

@@ -0,0 +1,21 @@
+require 'paraspec/rspec_patches'
+require 'paraspec/version'
+require 'paraspec/logger'
+require 'paraspec/ipc'
+require 'paraspec/drb_helpers'
+require 'paraspec/process_helpers'
+require 'paraspec/supervisor'
+require 'paraspec/master'
+require 'paraspec/worker'
+require 'paraspec/master_runner'
+require 'paraspec/worker_runner'
+require 'paraspec/worker_formatter'
+require 'paraspec/rspec_facade'
+
+module Paraspec
+  autoload :HttpClient, 'paraspec/http_client'
+  autoload :HttpServer, 'paraspec/http_server'
+  autoload :MsgpackClient, 'paraspec/msgpack_client'
+  autoload :MsgpackServer, 'paraspec/msgpack_server'
+  autoload :MsgpackHelpers, 'paraspec/msgpack_helpers'
+end

data/lib/paraspec/drb_helpers.rb

@@ -0,0 +1,65 @@
+require 'timeout'
+
+module Paraspec
+  module DrbHelpers
+    WAIT_TIME = 500
+
+=begin
+    class TimeoutWrapper < BasicObject
+      def initialize(target, timeout)
+        @target, @timeout = target, timeout
+      end
+
+      def method_missing(m, *args)
+        ::Timeout.timeout(@timeout) do
+          @target.send(m, *args)
+        end
+      end
+
+      def respond_to?(m, *args)
+        super(m, *args) || @target.respond_to?(m, *args)
+      end
+    end
+
+    # Connects to a DRb service and waits for the connection to start working.
+    #
+    # Interestingly, even when the remote end is up and running
+    # talking to it may fail the first time (or few times?)?
+    # Supervisor is able to invoke methods on master and subsequently
+    # when a worker connects to the master the DRb calls from worker fail.
+    # No idea why this is.
+    # Work around this by pinging and retrying each DRb connection
+    # prior to using it for real work.
+    #
+    # It appears that any DRb operation can also hang while producing
+    # no exceptions or output of any sort.
+    private def drb_connect(uri, timeout: true)
+      start_time = Time.now
+      Paraspec.logger.debug("#{ident} Connecting to DRb")
+      remote = TimeoutWrapper.new(DRbObject.new_with_uri(uri), 2)
+      Paraspec.logger.debug("#{ident} Waiting for DRb")
+      begin
+        # Assumes remote has a ping method
+        remote.ping
+      rescue DRb::DRbConnError, TypeError
+        raise if timeout && Time.now - start_time > WAIT_TIME
+        sleep 0.5
+        Paraspec.logger.debug("#{ident} Retrying DRb ping")
+        retry
+      rescue Timeout::Error
+        raise if timeout && Time.now - start_time > WAIT_TIME
+        Paraspec.logger.debug("#{ident} Reconnecting to DRb")
+        remote = TimeoutWrapper.new(DRbObject.new_with_uri(uri), 2)
+        retry
+      end
+      remote
+    end
+=end
+
+    def master_client
+      # TODO pass terminal option
+      @master_client ||= MsgpackClient.new
+    end
+
+  end
+end

data/lib/paraspec/http_client.rb

@@ -0,0 +1,43 @@
+require 'faraday'
+require 'json'
+
+module Paraspec
+  class HttpClient
+    def initialize(options={})
+      @terminal = options[:terminal]
+
+      @client = Faraday.new(url: "http://localhost:#{Paraspec::MASTER_APP_PORT}") do |client|
+        client.adapter :net_http
+        if @terminal
+          client.options.timeout = 100000
+        else
+          client.options.timeout = DrbHelpers::WAIT_TIME
+        end
+      end
+    end
+
+    def request(action, payload=nil)
+      url = '/' + action
+      start_time = Time.now
+      begin
+        resp = @client.post(url) do |req|
+          if payload
+            req.headers['content-type'] = 'application/json'
+            req.body = payload.to_json
+          end
+        end
+        if resp.status != 200
+          raise "Request failed: #{url} (#{resp.status})"
+        end
+        JSON.parse(resp.body)
+      rescue Faraday::ConnectionFailed
+        if !@terminal && Time.now - start_time > DrbHelpers::WAIT_TIME
+          raise
+        else
+          sleep 0.1
+          retry
+        end
+      end
+    end
+  end
+end

data/lib/paraspec/http_server.rb

@@ -0,0 +1,24 @@
+require 'sinatra'
+
+module Paraspec
+  class HttpServer < Sinatra::Base
+    post '/:action' do
+      action = params[:action]
+      body = request.body.read
+      if body.empty?
+        args = []
+      else
+        payload = JSON.parse(body)
+        payload = IpcHash.new.merge(payload)
+        args = [payload]
+      end
+
+      master = self.class.settings.master
+      action = action.gsub('-', '_')
+      result = master.send(action, *args)
+
+      content_type 'application/json'
+      (result || {}).to_json
+    end
+  end
+end

data/lib/paraspec/ipc.rb

@@ -0,0 +1,11 @@
+require 'hashie'
+
+module Paraspec
+  #SUPERVISOR_DRB_URI = "druby://localhost:6030"
+  MASTER_DRB_URI = "druby://localhost:6031"
+  MASTER_APP_PORT = 6031
+
+  class IpcHash < Hash
+    include Hashie::Extensions::IndifferentAccess
+  end
+end

data/lib/paraspec/logger.rb

@@ -0,0 +1,44 @@
+require 'logger'
+
+module Paraspec
+  class LoggerWrapper
+    def initialize(logger)
+      @logger = logger
+    end
+
+    def method_missing(m, *args)
+      @logger.send(m, *args)
+    end
+
+    %w(ipc state perf).each do |subsystem|
+      define_method "log_#{subsystem}=" do |v|
+        @subsystems ||= {}
+        @subsystems[subsystem] = v
+      end
+
+      define_method "log_#{subsystem}?" do
+        @subsystems && @subsystems[subsystem] or false
+      end
+
+      define_method "debug_#{subsystem}" do |*args|
+        if send("log_#{subsystem}?")
+          msg = "#{ident || '[?]'} [#{subsystem}] #{args.shift}"
+          debug(msg, *args)
+        end
+      end
+    end
+
+    attr_accessor :ident
+  end
+
+  class << self
+    attr_reader :logger
+
+    def logger=(logger)
+      @logger = LoggerWrapper.new(logger)
+    end
+  end
+
+  self.logger = Logger.new(STDERR)
+  self.logger.level = Logger::WARN
+end

data/lib/paraspec/master.rb

@@ -0,0 +1,219 @@
+module Paraspec
+  # The master process has three responsibilities:
+  # 1. Load all tests and abort the run if there are errors outside of
+  #    examples.
+  # 2. Maintain the queue of tests to feed the workers. The master
+  #    process also synchronizes access to this queue.
+  # 3. Aggregate test reports from the workers and present them to
+  #    the outside world in a coherent fashion. The latter means
+  #    that numbers presented are for the entire suite, not for parts
+  #    of it as executed by any single worker, and that output from a
+  #    single test execution is not broken up by output from other test
+  #    executions.
+  class Master
+    def initialize(options={})
+      @supervisor_pipe = options[:supervisor_pipe]
+      #RSpec.configuration.formatter = 'progress'
+      if RSpec.world.example_groups.count > 0
+        raise 'Example groups loaded too early/spilled across processes'
+      end
+
+      rspec_options = RSpec::Core::ConfigurationOptions.new(ARGV)
+      @non_example_exception_count = 0
+      begin
+        # This can fail if for example a nonexistent formatter is referenced
+        rspec_options.configure(RSpec.configuration)
+      rescue Exception => e
+        puts "#{e.class}: #{e}"
+        puts e.backtrace.join("\n")
+        # TODO and report this situation as a configuration problem
+        # and not a test suite problem
+        @non_example_exception_count = 1
+      end
+
+=begin
+      if RSpec.configuration.files_to_run.empty?
+        RSpec.configuration.send(:remove_instance_variable, '@files_to_run')
+        RSpec.configuration.files_or_directories_to_run = RSpec.configuration.default_path
+        RSpec.configuration.files_to_run
+        p ['aa1',RSpec.configuration.files_to_run]
+        rspec_options.configure(RSpec.configuration)
+        RSpec.configuration.load_spec_files
+      end
+=end
+
+      # It seems that load_spec_files sometimes rescues exceptions outside of
+      # examples and sometimes does not, handle it both ways
+      if @non_example_exception_count == 0
+        begin
+          RSpec.configuration.load_spec_files
+        rescue Exception => e
+          puts "#{e.class}: #{e}"
+          puts e.backtrace.join("\n")
+          @non_example_exception_count = 1
+        end
+      end
+      if @non_example_exception_count == 0
+        @non_example_exception_count = RSpec.world.reporter.non_example_exception_count
+      end
+      @queue = []
+      if @non_example_exception_count == 0
+        @queue += RSpecFacade.all_example_groups
+        puts "#{@queue.length} example groups queued"
+      else
+        puts "#{@non_example_exception_count} errors outside of examples, aborting"
+      end
+    end
+
+    attr :non_example_exception_count
+
+    def run
+      Thread.new do
+        #HttpServer.set(:master, self).run!(port: 6031)
+        MsgpackServer.new(self).run
+      end
+      until @stop
+        sleep 1
+      end
+      Paraspec.logger.debug_state("Exiting")
+    end
+
+    def ping
+      true
+    end
+
+    def stop
+      Paraspec.logger.debug_state("Stopping")
+      @stop = true
+    end
+
+    def stop?
+      @stop
+    end
+
+    def suite_ok?
+      RSpec.configuration.reporter.send(:instance_variable_get,'@non_example_exception_count') == 0
+    end
+
+    def example_count
+      RSpecFacade.all_examples.count
+    end
+
+    def get_spec
+      while true
+        example_group = @queue.shift
+        return nil if example_group.nil?
+
+        # TODO I am still not 100% on what should be filtered and pruned where,
+        # but we shouldn't be returning a specification here unless
+        # there are tests in it that a worker will run
+        pruned_examples = RSpec.configuration.filter_manager.prune(example_group.examples)
+        next if pruned_examples.empty?
+
+        m = example_group.metadata
+        return {
+          file_path: m[:file_path],
+          scoped_id: m[:scoped_id],
+        }
+      end
+    end
+
+    def example_passed(payload)
+      spec = payload[:spec]
+      # ExecutionResult
+      result = payload['result']
+      do_example_passed(spec, result)
+    end
+
+    def notify_example_started(payload)
+      example = find_example(payload[:spec])
+      reporter.example_started(example)
+    end
+
+    def do_example_passed(spec, execution_result)
+    #return
+      example = find_example(spec)
+      # Can write to example here
+      example.metadata[:execution_result] = execution_result
+      status = execution_result.status
+      m = "example_#{status}"
+      reporter.send(m, example)
+      nil
+    end
+
+    def find_example(spec)
+      if spec.nil?
+        #byebug
+        raise ArgumentError, 'Nil spec'
+      end
+      example = (RSpecFacade.all_example_groups + RSpecFacade.all_examples).detect do |example|
+        example.metadata[:file_path] == spec[:file_path] &&
+        example.metadata[:scoped_id] == spec[:scoped_id]
+      end
+      unless example
+        puts "Not found: #{spec[:file_path]}[#{spec[:scoped_id]}]"
+      #byebug
+        raise "Not found: #{spec[:file_path]}[#{spec[:scoped_id]}]"
+      end
+      example
+    end
+
+    def reporter
+      @reporter ||= RSpec.configuration.reporter
+    end
+
+    def suite_started
+      @start_time = Time.now
+
+      notification = RSpec::Core::Notifications::StartNotification.new(
+        RSpecFacade.all_examples.count, 0
+      )
+      RSpec.configuration.formatters.each do |f|
+        if f.respond_to?(:start)
+          f.start(notification)
+        end
+      end
+
+      true
+    end
+
+    def dump_summary
+      reporter.stop
+
+      all_examples = RSpecFacade.all_examples
+      notification = RSpec::Core::Notifications::SummaryNotification.new(
+        @start_time ? Time.now-@start_time : 0,
+        all_examples,
+        all_examples.select { |e| e.execution_result.status == :failed },
+        all_examples.select { |e| e.execution_result.status == :pending },
+        0,
+        non_example_exception_count,
+      )
+      examples_notification = RSpec::Core::Notifications::ExamplesNotification.new(reporter)
+      RSpec.configuration.formatters.each do |f|
+        if f.respond_to?(:dump_summary)
+          f.dump_summary(notification)
+        end
+        if f.respond_to?(:dump_failures)
+          f.dump_failures(examples_notification)
+        end
+        if f.respond_to?(:dump_pending)
+          f.dump_pending(examples_notification)
+        end
+      end
+      nil
+    end
+
+    def status
+      if RSpecFacade.all_examples.any? { |example| example.execution_result.status == :failed }
+        1
+      else
+        0
+      end
+    end
+
+    def ident
+      "[m]"
+    end
+  end
+end