specjour 0.2.5 → 0.3.0.rc1

data/History.markdown

@@ -1,22 +1,59 @@
 History
 =======
-0.2.5
------
-*2010-05-13*
 
-* [changed] The rails plugin now runs in a Rails.configuration.after_intialize
+## 0.3.0.rc1 / thor
+
+* [removed] Rake tasks have been removed, use the command-line instead.
+
+* [added] Thor is now used to parse command-line arguments. Try `specjour help`
+  for more details.
+
+* [added] Test discovery. Features will be autodiscovered by looking for a
+  `features` directory in your project. If you only want to run features, use
+  `specjour dispatch project_path/features`.
+
+* [changed] No longer need to run a manager and a dispatcher in separate
+  processes. When not distributing to other computers, simply run `specjour` in
+  your project directory to run the suite among the number of cores on your
+  machine.
+
+* [added] Project aliases. If you want to isolate a few computers in the
+  cluster, tell them to listen for a different project name and run the
+  dispatcher with that new name.
+
+        $ specjour listen --projects foo2
+        $ specjour dispatch --alias foo2
+
+* [added] Preparation. Running `specjour prepare` invokes the
+  `Specjour::Configuration.prepare` block on each worker. By default this
+  drops the worker's database and brings it back up.
+
+* [removed] --batch option which sent back results in batches. Now that each
+  spec is run one at a time, batching no longer makes sense.
+
+* [removed] Global listening. You now must provide the project names you want to
+  run specs for. Defaults to the project in your current working directory.
+
+        $ specjour listen --projects foo bar
+
+
+## 0.2.6 / master
+
+* [fixed] Rsync copies symbolic links. gh-6
+* [fixed] DbScrub explicitly requires its dependencies and no longer loads the
+  Rakefile. gh-10
+
+## 0.2.5 / 2010-05-13
+
+* [changed] The rails plugin now runs in a Rails.configuration.after\_initialize
   block
 
-0.2.4
------
-*2010-05-10*
+## 0.2.4 / 2010-05-10
 
 * [added] Correct exit status
 * [fixed] Will reconnect when connection is lost while requesting tests
 
-0.2.3
------
-*2010-04-25*
+## 0.2.3 / 2010-04-25
 
 * [fixed] Absolute paths in rsyncd.conf restrict portability. The rsync daemon
   completely fails when it can't find the path to serve which typically happens
@@ -26,23 +63,17 @@ History
 
 * [fixed] CPU core detection works on OSX Core i7 (thanks Hashrocket!)
 
-0.2.2
------
-*2010-04-22*
+## 0.2.2 / 2010-04-22
 
 * [added] Backtrace for cucumber failures
 
-0.2.1
------
-*2010-04-21*
+## 0.2.1 / 2010-04-21
 
 * [added] The rsync daemon configuration file now lives in
-  project_path/.specjour/rsyncd.conf. Edit your rsync exclusions there.
+  project\_path/.specjour/rsyncd.conf. Edit your rsync exclusions there.
 * [fixed] Don't report connection errors when CTRL-C is sent.
 
-0.2.0
------
-*2010-04-20*
+## 0.2.0 / 2010-04-20
 
 * [added] Cucumber support. `rake specjour:cucumber`
 * [added] CPU Core detection, use -w to override with less or more workers

data/README.markdown

@@ -16,25 +16,35 @@ _Distribute your spec suite amongst your LAN via Bonjour._
 ## Installation
     gem install specjour
 
-## Start a manager
-Running `specjour` on the command-line will start a manager which advertises that it's ready to run specs. By default, the manager will use your system cores to determine the number of workers to use. Two cores equals two workers. If you only want to dedicate 1 core to running specs, use `$ specjour --workers 1`.
+## Give it a try
+Running `specjour` starts a dispatcher, a manager, and multiple workers - all
+of the componenets necessary for distributing your test suite.
 
+    $ cd myproject
     $ specjour
 
-## Setup the dispatcher
-Require specjour's rake tasks in your project's `Rakefile`.
+## Start a manager
+Running `specjour listen` will start a manager which advertises that it's ready
+to run specs. By default, the manager runs tests for the project in the
+current directory and uses your system cores to determine the number of workers
+to start. If your system has two cores, two workers will run tests.
 
-    require 'specjour/tasks/specjour'
+    $ specjour listen
 
-## Distribute the specs
-Run the rake task to distribute the specs among the managers you started.
+## Distribute the tests
+Dispatch the tests among the managers you started. Specjour checks the 'spec' and
+'features' directories for tests.
 
-    $ rake specjour
+    $ specjour
 
-## Distribute the features
-Run the rake task to distribute the features among the managers you started.
+## Distribute a subset of tests
+The first parameter to the specjour command is a test directory. It defalts to
+the current directory and searches for 'spec' and 'features' paths therein.
 
-    $ rake specjour:cucumber
+    $ specjour spec # all rspec tests
+    $ specjour spec/models # only model tests
+    $ specjour features # only features
+    $ specjour ~/my_other_project/features
 
 ## Rails
 Each worker should run their specs in an isolated database. Modify the test database name in your `config/database.yml` to include the following environment variable (Influenced by [parallel\_tests](http://github.com/grosser/parallel_tests)):
@@ -42,32 +52,52 @@ Each worker should run their specs in an isolated database. Modify the test data
     test:
       database: blog_test<%=ENV['TEST_ENV_NUMBER']%>
 
-Add the specjour gem to your project:
+### ActiveRecord Hooks
+Specjour contains ActiveRecord hooks that clear database tables before running tests using `DELETE FROM <table_name>;`. Additionally, test databases will be created if they don't exist (i.e. `CREATE DATABASE blog_test8` for the 8th worker) and your schema will be loaded when the database is out of date.
 
-    config.gem 'specjour'
+## Custom Hooks
+Specjour allows you to hook in to the test process on a per-machine and
+per-worker level through the before_fork and after_fork configuration blocks.
+If the default ActiveRecord hook doesn't set up the database properly for your
+test suite, override it with a custom after_fork hook.
 
-Doing this enables a rails plugin wherein each worker will attempt to clear its database tables before running any specs via `DELETE FROM <table_name>;`. Additionally, test databases will be created if they don't exist (i.e. `CREATE DATABASE blog_test8` for the 8th worker) and your schema will be loaded when the database is out of date.
+    # config/initializers/specjour.rb
+    Rails.configuration.after_initialize do
 
-### Customizing database setup
-If the plugin doesn't set up the database properly for your test suite, bypass it entirely. Remove specjour as a project gem and create your own initializer to setup the database. Specjour sets the environment variable PREPARE\_DB when it runs your specs so you can look for that when setting up the database.
+      # Modify the way you use bundler
+      Specjour::Configuration.before_fork = lambda do
+        # TODO: not working as advertised
+        system('bundle install --without production')
+      end
 
-    # config/initializers/specjour.rb
+      # Modify your database setup
+      Specjour::Configuration.after_fork = lambda do
+        # custom database setup here
+      end
 
-    if ENV['PREPARE_DB']
-      load 'Rakefile'
+    end
+
+A preparation hook is run when `specjour prepare` is invoked. This hook allows
+you to run arbitrary code on all of the listening workers. By default, it drops
+and recreates the database on all workers.
+
+    Rails.configuration.after_initialize do
+
+      # Modify preparation
+      Specjour::Configuration.prepare = lambda do
+        # custom preparation code
+      end
 
-      # clear the db and load db/seeds.rb
-      Rake::Task['db:reset'].invoke
     end
 
 ## Only listen to supported projects
 By default, a manager will listen to all projects trying to distribute specs over the network. Sometimes you'll only want a manager to respond to one specific spec suite. You can accomplish this with the `--projects` flag.
 
-    $ specjour --projects bizconf # only run specs for the bizconf project
+    $ specjour listen --projects bizconf # only run specs for the bizconf project
 
 You could also listen to multiple projects:
 
-    $ specjour --projects bizconf,workbeast # only run specs for the bizconf and workbeast projects
+    $ specjour listen --projects bizconf,workbeast # only run specs for the bizconf and workbeast projects
 
 ## Customize what gets rsync'd
 The standard rsync configuration file may be too broad for your
@@ -76,9 +106,6 @@ directory, add an exclusion to your projects rsyncd.conf file.
 
     $ vi workbeast/.specjour/rsyncd.conf
 
-## Use one machine
-Distributed testing doesn't have to happen over multiple machines, just multiple processes. Specjour is an excellent candidiate for running 4 tests at once on one machine with 4 cores. Just run `$ specjour` in one window and `$ rake specjour` in another.
-
 ## Thanks
 
 * shayarnett - Cucumber support, pairing and other various patches

data/Rakefile

@@ -12,9 +12,10 @@ begin
     gem.authors = ["Sandro Turriate"]
     gem.add_dependency "dnssd", "1.3.1"
     gem.add_dependency "rspec"
+    gem.add_dependency "thor", "0.13.6"
     gem.add_development_dependency "rspec", "1.3.0"
-    gem.add_development_dependency "rr", "0.10.11"
-    gem.add_development_dependency "yard", "0.5.3"
+    gem.add_development_dependency "rr", ">=0.10.11"
+    gem.add_development_dependency "yard", ">=0.5.3"
     # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
   end
   Jeweler::GemcutterTasks.new
@@ -46,6 +47,3 @@ rescue LoadError
     abort "YARD is not available. In order to run yardoc, you must: sudo gem install yard"
   end
 end
-
-$:.unshift(File.dirname(__FILE__) + "/lib")
-require 'specjour/tasks/specjour'

data/VERSION

@@ -1 +1 @@
-0.2.5
+0.3.0.rc1

data/bin/specjour

@@ -1,51 +1,4 @@
 #!/usr/bin/env ruby
-require 'optparse'
 require 'specjour'
 
-options = {:batch_size => 1}
-
-optparse = OptionParser.new do |opts|
-  opts.banner = "Usage: specjour [options]"
-
-  opts.on('-w', '--workers WORKERS', Numeric, "Number of WORKERS to spin up, defaults to available cores") do |n|
-    options[:worker_size] = n
-  end
-
-  opts.on('-b', '--batch-size [SIZE]', Integer, "Number of specs to run before reporting back to the dispatcher, defaults to #{options[:batch_size]}") do |n|
-    options[:batch_size] = n
-  end
-
-  opts.on('-p', '--projects PROJECTS', Array, "Only run specs for these comma delimited project names, i.e. workbeast,taigan") do |project_names|
-    options[:registered_projects] = project_names
-  end
-
-  opts.on('--do-work OPTIONS', Array, 'INTERNAL USE ONLY') do |args|
-    options[:worker_args] = args[0], args[1], args[2]
-  end
-
-  opts.on('--log', TrueClass, 'print debug messages to stdout') do |val|
-    Specjour.new_logger Logger::DEBUG
-  end
-
-  opts.on_tail('-v', '--version', 'Show the version of specjour') do
-    abort Specjour::VERSION
-  end
-
-  opts.on_tail("-h", "--help", "Show this message") do
-    summary = opts.to_a
-    summary.first << "\n"
-    abort summary.reject {|s| s =~ /INTERNAL/}.join
-  end
-end
-
-optparse.parse!
-
-abort(%(ERROR: I don't understand the following flags: "#{ARGV.join(', ')}")) if ARGV.any?
-
-if options[:worker_args]
-  options[:worker_args] << options[:batch_size]
-  Specjour::Worker.new(*options[:worker_args]).run
-else
-  options[:worker_size] ||= Specjour::CPU.cores
-  Specjour::Manager.new(options).start
-end
+Specjour::CLI.start

data/lib/specjour/cli.rb

@@ -0,0 +1,97 @@
+module Specjour
+  require 'thor'
+  class CLI < Thor
+
+    def self.printable_tasks
+      super.reject{|t| t.last =~ /INTERNAL USE/ }
+    end
+
+    def self.worker_option
+      method_option :workers, :aliases => "-w", :type => :numeric, :desc => "Number of concurent processes to run. Defaults to your system's available cores."
+    end
+
+    def self.dispatcher_option
+      method_option :alias, :aliases => "-a", :desc => "Project name advertised to listeners"
+    end
+
+    def self.start(original_args=ARGV, config={})
+      real_tasks = all_tasks.keys | HELP_MAPPINGS
+      unless real_tasks.include? original_args.first
+        original_args.unshift default_task
+      end
+      super(original_args)
+    end
+
+    default_task :dispatch
+
+    class_option :log, :aliases => "-l", :type => :boolean, :desc => "Print debug messages to $stdout"
+
+    desc "listen", "Advertise availability to run specs\nDefaults to current directory"
+    worker_option
+    method_option :projects, :aliases => "-p", :type => :array, :desc => "Projects supported by this listener"
+    def listen
+      handle_logging
+      handle_workers
+      args[:registered_projects] = args.delete(:projects) || [File.basename(Dir.pwd)]
+      Specjour::Manager.new(args).start
+    end
+
+    desc "dispatch [PROJECT_PATH]", "Run specs in this project"
+    worker_option
+    dispatcher_option
+    def dispatch(path = Dir.pwd)
+      handle_logging
+      handle_workers
+      handle_dispatcher(path)
+      Specjour::Dispatcher.new(args).start
+    end
+
+    desc "prepare [PROJECT_PATH]", "Run the prepare block on all listening workers"
+    worker_option
+    dispatcher_option
+    def prepare(path = Dir.pwd)
+      handle_logging
+      handle_workers
+      handle_dispatcher(path)
+      args[:worker_task] = 'prepare'
+      Specjour::Dispatcher.new(args).start
+    end
+
+    desc "version", "Show the version of specjour"
+    def version
+      puts Specjour::VERSION
+    end
+
+    desc "work", "INTERNAL USE ONLY"
+    method_option :project_path, :required => true
+    method_option :printer_uri, :required => true
+    method_option :number, :type => :numeric, :required => true
+    method_option :preload_spec
+    method_option :preload_feature
+    method_option :task, :required => true
+    method_option :quiet, :type => :boolean
+    def work
+      handle_logging
+      Specjour::Worker.new(args).start
+    end
+
+    protected
+
+    def args
+      @args ||= options.dup
+    end
+
+    def handle_logging
+      Specjour.new_logger(Logger::DEBUG) if options['log']
+    end
+
+    def handle_workers
+      args[:worker_size] = options["workers"] || CPU.cores
+    end
+
+    def handle_dispatcher(path)
+      args[:project_path] = path
+      args[:project_alias] = args.delete(:alias)
+    end
+  end
+end

data/lib/specjour/configuration.rb

@@ -0,0 +1,73 @@
+module Specjour
+  module Configuration
+    extend self
+
+    attr_writer :before_fork, :after_fork, :prepare
+
+    # This block is run by each worker the manager forks.
+    # The Rails plugin uses this block to clear the databases defined in
+    # ActiveRecord.
+    # Set your own block if the default doesn't work for you.
+    def after_fork
+      @after_fork ||= default_after_fork
+    end
+
+    # This block is run after before forking. When ActiveRecord is
+    # defined, the default before_block disconnects from the database.
+    def before_fork
+      @before_fork ||= default_before_fork
+    end
+
+    # This block is run on all workers when invoking `specjour prepare`
+    # Defaults to dropping the worker's database and recreating it. This
+    # is especially useful when two teams are sharing workers and writing
+    # migrations at around the same time causing databases to get out of sync.
+    def prepare
+      @prepare ||= default_prepare
+    end
+
+    def reset
+      @before_fork = nil
+      @after_fork = nil
+      @prepare = nil
+    end
+
+    def bundle_install
+      if system('which bundle')
+        system('bundle check') || system('bundle install')
+      end
+    end
+
+    def default_before_fork
+      lambda do
+        ActiveRecord::Base.remove_connection if defined?(ActiveRecord::Base)
+        bundle_install
+      end
+    end
+
+    def default_after_fork
+      lambda do
+        DbScrub.scrub if rails_with_ar?
+      end
+    end
+
+    def default_prepare
+      lambda do
+        if rails_with_ar?
+          DbScrub.drop
+          DbScrub.scrub
+        end
+      end
+    end
+
+    protected
+
+    def rails_with_ar?
+      defined?(Rails) && defined?(ActiveRecord::Base)
+    end
+
+    def system(cmd)
+      Kernel.system("#{cmd} > /dev/null")
+    end
+  end
+end

data/lib/specjour/connection.rb

@@ -36,7 +36,7 @@ module Specjour
     def timeout(&block)
       Timeout.timeout(2, &block)
     rescue Timeout::Error
-      raise Error, "Connection to dispatcher timed out"
+      raise Error, "Connection to dispatcher timed out", []
     end
 
     def next_test

data/lib/specjour/cucumber/distributed_formatter.rb

@@ -1,9 +1,6 @@
 module Specjour::Cucumber
+  ::Term::ANSIColor.coloring = true
   class DistributedFormatter < ::Cucumber::Formatter::Progress
-    class << self
-      attr_accessor :batch_size
-    end
-    @batch_size = 1
 
     def initialize(step_mother, io, options)
       @step_mother = step_mother
@@ -63,7 +60,7 @@ module Specjour::Cucumber
       prepare_steps(:failed)
       prepare_steps(:undefined)
 
-      @io.send_message(:worker_summary=, to_hash)
+      @io.send_message(:cucumber_summary=, to_hash)
     end
 
     OUTCOMES = [:failed, :skipped, :undefined, :pending, :passed]

data/lib/specjour/cucumber/preloader.rb

@@ -0,0 +1,13 @@
+class Specjour::Cucumber::Preloader
+  def self.load(feature_file)
+    # preload all features
+    cli = ::Cucumber::Cli::Main.new []
+    step_mother = cli.class.step_mother
+
+    step_mother.log = cli.configuration.log
+    step_mother.load_code_files(cli.configuration.support_to_load)
+    step_mother.after_configuration(cli.configuration)
+    features = step_mother.load_plain_text_features(cli.configuration.feature_files)
+    step_mother.load_code_files(cli.configuration.step_defs_to_load)
+  end
+end

data/lib/specjour/cucumber.rb

@@ -4,10 +4,11 @@ module Specjour
       require 'cucumber'
       require 'cucumber/formatter/progress'
 
-      require 'specjour/cucumber/dispatcher'
       require 'specjour/cucumber/distributed_formatter'
       require 'specjour/cucumber/final_report'
-      require 'specjour/cucumber/printer'
+      require 'specjour/cucumber/preloader'
+
+      ::Cucumber::Cli::Options.class_eval { def print_profile_information; end }
     rescue LoadError
     end
   end

data/lib/specjour/db_scrub.rb

@@ -1,8 +1,15 @@
 module Specjour
   module DbScrub
-    load 'Rakefile'
+    require 'rake'
+    load 'tasks/misc.rake'
+    load 'tasks/databases.rake'
+
     extend self
 
+    def drop
+      Rake::Task['db:drop']
+    end
+
     def scrub
       connect_to_database
       if pending_migrations?