flatware 0.4.0 → 2.0.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 00a71ce8a23e22600391980c40aac3425a1efd27
-  data.tar.gz: '082d74e5882c2cb99bfc4af567618e70817bb08d'
+SHA256:
+  metadata.gz: 944469a1460443f84117dcdacbe79ec133c7605e83cc2b2882faaf573661a458
+  data.tar.gz: cc120e2ad04cf8baabf001921f16a03c67f15dd81884a6df83d097c06af420ae
 SHA512:
-  metadata.gz: c591c042ab1bae59ba7e25309ceae39ca389f60b1d6f340882db31e3b0806e9acd0a8dfaa512e5529a2f1f636c297c9ba5335e247d3de3b65b90e3929b2adbf3
-  data.tar.gz: 33043b7af9e2d929e7a99c8e53c127ac9bbc4bb366f78e74ae32e53432980d64bb2b61aa7eecf818b2611df25da68d12c0ff975a463125b35748c4227d22ab16
+  metadata.gz: 80d9ae698fd2f8c9c01c78e09d65fa36a1989b6bbdbee62cac5b74c5d69003734025cec6862f89415ca1e79ced1fbce97a17d37b31238f617b5d4fc5e160e74f
+  data.tar.gz: 35a22fe1de1df1a761ef16114729d10e51ddd797c3a7590293d49465f9ab9e86a77e22572d89abb4b603e56093da0d2ec8b1dccccd4c7cbc666339aa693d987d

data/README.md

@@ -7,39 +7,13 @@
 
 Flatware parallelizes your test suite to significantly reduce test time.
 
-## Requirements
-
-* ZeroMQ > 4.0
-
-## Installation
-
-### ZeroMQ
-
-#### Linux Ubuntu
-
-```sh
-sudo apt-get install -qq libzmq3-dev
-```
-
-(Never you mind the 3. This package contains ZMQ version 4.)
-
-#### Mac OSX
-
-Ruby FFI isn't getting along with the latest ZMQ formula. A tweaked verson is available in the Hashrocket tap.
-
-```sh
-brew tap hashrocket/formulas
-brew install hashrocket/formulas/zeromq
-brew install zeromq
-```
-
 ### Flatware
 
 Add the runners you need to your Gemfile:
 
 ```ruby
-gem 'flatware-rspec'    # one
-gem 'flatware-cucumber' # or both
+gem 'flatware-rspec', require: false    # one
+gem 'flatware-cucumber', require: false # or both
 ```
 
 then run
@@ -66,6 +40,18 @@ To run your entire suite with the default rspec options add the `flatware-rspec`
 $ flatware rspec
 ```
 
+The rspec runner can balance worker loads, making your suite even faster.
+
+It forms balaced groups of spec files according to their last run times, if you've set `example_status_persistence_file_path` [in your RSpec config](https://relishapp.com/rspec/rspec-core/v/3-8/docs/command-line/only-failures).
+
+For this to work the configuration option must be loaded before any specs are run. The `.rspec` file is one way to achive this:
+
+    --require spec_helper
+
+But beware, if you're using ActiveRecord in your suite you'll need to avoid doing things that cause it to establish a database connection in `spec_helper.rb`. If ActiveRecord connects before flatware forks off workers, each will die messily. All of this will just work if you're following [the recomended pattern of splitting your helpers into `spec_helper` and `rails_helper`](https://github.com/rspec/rspec-rails/blob/v3.8.2/lib/generators/rspec/install/templates/spec/rails_helper.rb). Another option is to use [the configurable hooks](
+#faster-startup-with-activerecord
+).
+
 ### Options
 
 If you'd like to limit the number of forked workers, you can pass the 'w' flag:
@@ -78,7 +64,7 @@ You can also pass most cucumber/rspec options to Flatware. For example, to run o
 features that are not tagged 'javascript', you can:
 
 ```sh
-$ flatware cucumber -t ~@javascript
+$ flatware cucumber -t 'not @javascript'
 ```
 
 Additionally, for either cucumber or rspec you can specify a directory:
@@ -116,9 +102,31 @@ Now you are ready to rock:
 $ flatware rspec && flatware cucumber
 ```
 
-## Planned Features
+### Faster Startup With ActiveRecord
 
-* Use heuristics to run your slowest tests first
+Flatware has a couple lifecycle callbacks that you can use to avoid booting your app
+over again on every core. One way to take advantage of this via a `spec/flatware_helper.rb` file like so:
+
+```ruby
+Flatware.configure do |conf|
+  conf.before_fork do
+    require 'rails_helper'
+
+    ActiveRecord::Base.connection.disconnect!
+  end
+
+  conf.after_fork do |test_env_number|
+    config = ActiveRecord::Base.connection_config
+
+    ActiveRecord::Base.establish_connection(
+      config.merge(
+        database: config.fetch(:database) + test_env_number.to_s
+      )
+    )
+  end
+end
+```
+Now when I run `bundle exec flatware rspec -r ./spec/flatware_helper` My app only boots once, rather than once per core.
 
 ## Design Goals
 
@@ -149,20 +157,10 @@ directory. CD there and `flatware` will be in your path so you can tinker away.
 
 ## How it works
 
-Flatware relies on a message passing system to enable concurrency.
-The main process declares a worker for each cpu in the computer. Each
-worker forks from the main process and is then assigned a portion of the
-test suite.  As the worker runs the test suite it sends progress
-messages to the main process.  These messages are collected and when
-the last worker is finished the main process provides a report on the
-collected progress messages.
+Flatware relies on a message passing system to enable concurrency. The main process forks a worker for each cpu in the computer. These workers are each given a chunk of the tests to run. The workers report back to the main process about their progress. The main process prints those progress messages. When the last worker is finished the main process prints the results.
 
 ## Resources
 
-To learn more about the messaging system that Flatware uses, take a look at the
-[excellent ZeroMQ guide][z].
-
-[z]: http://zguide.zeromq.org/page:all
 [a]: https://github.com/cucumber/aruba
 
 ## Contributing to Flatware

data/bin/flatware

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
-lib = File.expand_path('../../lib', __FILE__)
-$:.unshift lib unless $:.include? lib
+lib = File.expand_path('../lib', __dir__)
+$LOAD_PATH.unshift lib unless $LOAD_PATH.include? lib
 
 require 'flatware'
 Flatware::CLI.start

data/lib/flatware.rb

@@ -1,9 +1,39 @@
+# frozen_string_literal: true
+
+require 'logger'
+
 module Flatware
-  require 'flatware/processor_info'
+  require 'flatware/job'
   require 'flatware/cli'
-  require 'flatware/poller'
   require 'flatware/sink'
-  require 'flatware/socket'
   require 'flatware/worker'
   require 'flatware/broadcaster'
+
+  module_function
+
+  def logger
+    @logger ||= Logger.new($stderr, level: :fatal)
+  end
+
+  def logger=(logger)
+    @logger = logger
+  end
+
+  def log(*message)
+    case message.first
+    when Exception
+      logger.error message.first
+    else
+      logger.info(([$PROGRAM_NAME] + message).join(' '))
+    end
+    message
+  end
+
+  def verbose=(bool)
+    logger.level = bool ? :debug : :fatal
+  end
+
+  def verbose?
+    logger.level < Logger::SEV_LABEL.index('FATAL')
+  end
 end

data/lib/flatware/broadcaster.rb

@@ -1,5 +1,17 @@
+# frozen_string_literal: true
+
 module Flatware
+  # sends messages to all formatters
   class Broadcaster
+    FORMATTER_MESSAGES = %i[
+      jobs
+      started
+      progress
+      finished
+      summarize
+      summarize_remaining
+    ].freeze
+
     attr_reader :formatters
 
     def initialize(formatters)
@@ -7,9 +19,14 @@ module Flatware
     end
 
     def method_missing(name, *args)
-      formatters.each do |formatter|
-        formatter.send name, *args if formatter.respond_to? name
-      end
+      return super unless FORMATTER_MESSAGES.include? name
+
+      formatters.select { |formatter| formatter.respond_to? name }
+                .each { |formatter| formatter.send name, *args }
+    end
+
+    def respond_to_missing?(name, _include_all)
+      FORMATTER_MESSAGES.include? name
     end
   end
 end

data/lib/flatware/cli.rb

@@ -1,37 +1,52 @@
+# frozen_string_literal: true
+
 require 'thor'
-require 'flatware/pids'
+require 'flatware/pid'
+require 'etc'
+
 module Flatware
+  # shared flatware cli
   class CLI < Thor
-
     def self.processors
-      @processors ||= ProcessorInfo.count
+      @processors ||= Etc.nprocessors
     end
 
     def self.worker_option
-      method_option :workers, aliases: "-w", type: :numeric, default: processors, desc: "Number of concurent processes to run"
+      method_option(
+        :workers,
+        aliases: '-w',
+        type: :numeric,
+        default: processors,
+        desc: 'Number of concurent processes to run'
+      )
     end
 
-    class_option :log, aliases: "-l", type: :boolean, desc: "Print debug messages to $stderr"
+    class_option(
+      :log,
+      aliases: '-l',
+      type: :boolean,
+      desc: 'Print debug messages to $stderr'
+    )
 
     worker_option
-    desc "fan [COMMAND]", "executes the given job on all of the workers"
+    desc 'fan [COMMAND]', 'executes the given job on all of the workers'
     def fan(*command)
       Flatware.verbose = options[:log]
 
-      command = command.join(" ")
+      command = command.join(' ')
       puts "Running '#{command}' on #{workers} workers"
 
       workers.times do |i|
         fork do
-          exec({"TEST_ENV_NUMBER" => i.to_s}, command)
+          exec({ 'TEST_ENV_NUMBER' => i.to_s }, command)
         end
       end
       Process.waitall
     end
 
-    desc "clear", "kills all flatware processes"
+    desc 'clear', 'kills all flatware processes'
     def clear
-      (Flatware.pids - [$$]).each do |pid|
+      (Flatware.pids - [Process.pid]).each do |pid|
         Process.kill 6, pid
       end
     end
@@ -39,26 +54,35 @@ module Flatware
     private
 
     def start_sink(jobs:, workers:, formatter:)
-     $0 = 'flatware sink'
+      $0 = 'flatware sink'
       Process.setpgrp
-      passed = Sink.start_server jobs: jobs, formatter: Flatware::Broadcaster.new([formatter]), sink: options['sink-endpoint'], dispatch: options['dispatch-endpoint'], worker_count: workers
+      passed = Sink.start_server(
+        jobs: jobs,
+        formatter: Flatware::Broadcaster.new([formatter]),
+        sink: options['sink-endpoint'],
+        worker_count: workers
+      )
       exit passed ? 0 : 1
     end
 
-    def log(*args)
-      Flatware.log(*args)
-    end
-
     def workers
       options[:workers]
     end
   end
 end
 
-flatware_gems = Gem.loaded_specs.keys.grep(/^flatware-/)
+flatware_gems = %w[flatware-rspec flatware-cucumber]
+
+loaded_flatware_gem_count = flatware_gems.map do |flatware_gem|
+  require flatware_gem
+rescue LoadError
+  nil
+end.compact.size
 
-if flatware_gems.count > 0
-  flatware_gems.each(&method(:require))
-else
-  puts "The flatware gem is a dependency of flatware runners for rspec and cucumber.  Try adding flatware-rspec or flatware-cucumber to your Gemfile."
+if loaded_flatware_gem_count.zero?
+  warn(
+    format(<<~MESSAGE, gem_list: flatware_gems.join(' or ')))
+      The flatware gem is a dependency of flatware runners for rspec and cucumber.
+      Install %<gem_list>s for more usefull commands.
+    MESSAGE
 end

data/lib/flatware/configuration.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module Flatware
+  class Configuration
+    def initialize
+      reset!
+    end
+
+    def before_fork(&block)
+      if block_given?
+        @before_fork = block
+      else
+        @before_fork
+      end
+    end
+
+    def after_fork(&block)
+      if block_given?
+        @after_fork = block
+      else
+        @after_fork
+      end
+    end
+
+    def reset!
+      @before_fork = -> {}
+      @after_fork = ->(_) {}
+    end
+  end
+
+  module_function
+
+  def configuration
+    @configuration ||= Configuration.new
+  end
+
+  def configure(&_block)
+    yield configuration
+  end
+end

data/lib/flatware/job.rb

@@ -0,0 +1,22 @@
+module Flatware
+  Job = Struct.new :id, :args do
+    attr_accessor :worker
+    attr_writer :failed
+
+    def failed?
+      @failed == true
+    end
+
+    def failed!
+      @failed = true
+    end
+
+    def sentinel?
+      id == 'seppuku'
+    end
+
+    def self.sentinel
+      new 'seppuku'
+    end
+  end
+end

data/lib/flatware/pid.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require 'etc'
+
+module Flatware
+  module_function
+
+  # All the pids of all the processes called flatware on this machine
+  def pids
+    Pid.pids { |pid| pid.command =~ /flatware/ }
+  end
+
+  def pids_of_group(group_pid)
+    Pid.pids { |pid| pid.pgid == group_pid }
+  end
+
+  Pid = Struct.new(:pid, :pgid, :ppid, :command) do
+    def self.pids(&block)
+      ps.select(&block).map(&:pid)
+    end
+
+    def self.ps
+      args = ['-o', members.join(',')]
+      args += { 'Darwin' => %w[-c] }.fetch(Etc.uname.fetch(:sysname), [])
+
+      IO
+        .popen(['ps', *args])
+        .readlines
+        .map do |row|
+        fields = row.strip.split(/\s+/, 4)
+        new(*fields.take(3).map(&:to_i), fields.last)
+      end
+    end
+
+    def pids_of_group(group_pid)
+      ps
+        .select { |pid| pid.pgid == group_pid }
+        .map(&:pid)
+    end
+  end
+end

data/lib/flatware/serialized_exception.rb

@@ -2,15 +2,25 @@ module Flatware
   class SerializedException
     attr_reader :class, :message, :cause
     attr_accessor :backtrace
-    def initialize(klass, message, backtrace, cause='')
-      @class, @message, @backtrace, @cause = serialized(klass), message, backtrace, cause
+
+    def initialize(klass, message, backtrace, cause = '')
+      @class = serialized(klass)
+      @message = message
+      @backtrace = backtrace
+      @cause = cause
     end
 
     def self.from(exception)
-      new exception.class, exception.message, exception.backtrace, exception.cause
+      new(
+        exception.class,
+        exception.message,
+        exception.backtrace,
+        exception.cause
+      )
     end
 
     private
+
     def serialized(klass)
       SerializedClass.new(klass.to_s)
     end
@@ -19,6 +29,8 @@ module Flatware
   class SerializedClass
     attr_reader :name
     alias to_s name
-    def initialize(name); @name = name end
+    def initialize(name)
+      @name = name
+    end
   end
 end