drake 0.8.4.1.1.0 → 0.8.4.1.2.0

data/CHANGES.drake

@@ -1,6 +1,10 @@
 
 = Drake Changelog
 
+== Version 0.8.4.1.2.0
+
+* github master now represents mainline-merge-candidate branch.
+
 == Version 0.8.4.1.0.16-18
 
 * Merge with latest comp_tree.

data/README

@@ -1,6 +1,6 @@
 = DRAKE -- Distributed Rake
 
-A branch of Rake supporting parallel task execution.
+A branch of Rake supporting automatic parallelizing of tasks.
 
 == Synopsis
 
@@ -18,90 +18,7 @@ or equivalently,
 
 == Notes
 
-=== Compatibility
-
-Drake is 100% compatible with Rake.  The code path for
-<tt>--threads=1</tt> is effectively identical to that of Rake's.
-Drake passes all of Rake's unit tests, with any number of threads from
-1 to 1000 (the most tested).
-
-=== Dependencies
-
-In a given Rakefile, it is possible (even likely) that the
-dependency tree has not been properly defined.  Consider
-
-   task :a => [:x, :y, :z]
-
-With single-threaded Rake, _x_,_y_,_z_ will be invoked <em>in that
-order</em> before _a_ is invoked (assuming there are no other rules
-involving these tasks).  However with <code>drake -jN</code> (for +N+
-> 1), one should not expect any particular order of execution.  Since
-there is no dependency specified between _x_,_y_,_z_ above, Drake is
-free to run them in any order.
-
-If you wish _x_,_y_,_z_ to be invoked sequentially, then write
-
-   task :a => seq[:x, :y, :z]
-
-This is shorthand for
-
-   task :a => :z
-   task :z => :y
-   task :y => :x
-
-Upon invoking _a_, the above rules say: "Can't do _a_ until _z_ is
-complete; can't do _z_ until _y_ is complete; can't do _y_ until _x_
-is complete; therefore do _x_."  In this fashion the sequence
-_x_,_y_,_z_ is enforced.
-
-The problem of insufficient dependencies plagues Makefiles as well,
-and is sometimes called "not j-safe".
-
-=== MultiTask
-
-When more than one thread is given, +multitask+ behaves just like
-+task+.  Those tasks which may properly be run in parallel will be run
-in parallel; those which cannot, will not.  It is not the user's job
-to decide.  In other words, for <tt>-jN</tt> (+N+ > 1), +multitask+ is
-an alias of +task+.
-
-For <tt>-j1</tt> (default), +multitask+ behaves as the original.
-
-=== Task#invoke inside Task#invoke
-
-Parallelizing code means surrendering control over the
-micro-management of its execution.  Manually invoking tasks inside
-other tasks is rather contrary to this notion, throwing a monkey
-wrench into the system.  An exception will be raised when this is
-attempted in multi-threaded mode.
-
-=== Migrating to -j
-
-First of all, do you want to bother with <tt>-j</tt>?  If you are
-satisfied with your build time, then there is really no reason to use
-it.
-
-If on the other hand your build takes twenty minutes to complete, you
-may be interested in investing some time getting the full dependency
-tree correct in order to take advantage of multiple CPUs or cores.
-
-Though Drake cannot fathom what <em>you</em> mean by a correct
-dependency, there is a tool available which may help you get closer to
-saying what you mean:
-
-  % drake --rand[=SEED]
-
-This will randomize the order of sibling prerequisites for each task.
-When given the optional SEED string, it will call
-<tt>srand(SEED.hash)</tt> to produce the same permutation each time.
-The randomize option also disables +multitask+, making it a regular
-+task+.  (In multi-threaded mode, +multitask+ is already a regular
-+task+.)
-
-Though this option may produce an error due to unspecified
-dependencies, with SEED at least it will be an error which is exactly
-the same on each run.  In addition you'll have the major debugging
-advantage of using a single thread.
+See parallel.rdoc.
 
 == Links
 

data/Rakefile

@@ -63,23 +63,22 @@ task :tf => :test_functional
 task :tu => :test_units
 task :tc => :test_contribs
 task :test => :test_units
+task :test_all => [:test_serial, :test_parallel]
 
-all_test_files = FileList[
+test_files = FileList[
   'test/test*.rb',
   'test/contrib/test*.rb',
   'test/fun*.rb'
 ]
 
-task :test_all => [:test_single_threaded, :test_parallel]
-
-Rake::TestTask.new(:test_parallel) do |t|
-  t.test_files = FileList['test/parallel.rb'] + all_test_files
+Rake::TestTask.new(:test_serial) do |t|
+  t.test_files = ['test/serial_setup.rb'] + test_files
   t.warning = true
   t.verbose = false
 end
 
-Rake::TestTask.new(:test_single_threaded) do |t|
-  t.test_files = FileList['test/single_threaded.rb'] + all_test_files
+Rake::TestTask.new(:test_parallel) do |t|
+  t.test_files = ['test/parallel_setup.rb'] + test_files
   t.warning = true
   t.verbose = false
 end
@@ -102,12 +101,6 @@ Rake::TestTask.new(:test_contribs) do |t|
   t.verbose = false
 end
 
-Rake::TestTask.new(:test_current) do |t|
-  t.test_files = FileList['test/parallel.rb', 'test/test_tasks.rb']
-  t.warning = true
-  t.verbose = false
-end
-
 begin
   require 'rcov/rcovtask'
 
@@ -167,7 +160,6 @@ PKG_FILES = FileList[
   '[A-Z]*',
   'bin/drake', 
   'lib/**/*.rb', 
-  'lib/rake/comp_tree/**/*.rb', 
   'test/**/*.rb',
   'test/**/*.rf',
   'test/**/*.mf',
@@ -187,7 +179,7 @@ else
 
     s.name = 'drake'
     s.version = $package_version
-    s.summary = "A fork of Rake supporting parallel task execution."
+    s.summary = "A branch of Rake supporting automatic parallelizing of tasks."
     s.description = <<-EOF
       Rake is a Make-like program implemented in Ruby. Tasks
       and dependencies are specified in standard Ruby syntax. 
@@ -195,6 +187,7 @@ else
 
     #### Dependencies and requirements.
 
+    s.add_dependency('comp_tree', '>= 0.7.1')
     #s.add_dependency('log4r', '> 1.0.4')
     #s.requirements << ""
 

data/doc/command_line_usage.rdoc

@@ -40,6 +40,9 @@ Options are:
 [<tt>--libdir</tt> _directory_  (-I)]
     Add _directory_ to the list of directories searched for require.
 
+[<tt>--threads</tt> _number_ (-j)]
+    Run up to N independent tasks simultaneously in separate threads.
+
 [<tt>--nosearch</tt>  (-N)]
     Do not search for a Rakefile in parent directories.
 
@@ -60,6 +63,12 @@ Options are:
 [<tt>--rakelibdir</tt> _rakelibdir_ (-R)]
     Auto-import any .rake files in RAKELIBDIR. (default is 'rakelib')
 
+[<tt>--randomize</tt>[=_seed_]]
+    Randomize the order of sibling prerequisites for each task.  When
+    _seed_ is given, <tt>srand(seed.hash)</tt> will be called so that
+    the same permutations will be produced for subsequent runs with
+    the same _seed_.
+
 [<tt>--require</tt> _name_ (-r)]
     Require _name_ before executing the Rakefile.
 

data/doc/parallel.rdoc

@@ -0,0 +1,112 @@
+
+= Automatically Running Tasks in Parallel
+
+Consider these task definitions:
+
+  task :default => [:a, :b]
+  task :a => [:x, :y]
+  task :b
+
+To create the dependency graph, let nodes represent tasks and child
+nodes represent prerequisites,
+
+                default
+                  / \
+                 /   \
+                a     b
+               / \
+              /   \
+             x     y
+
+Notice the dependency graph already contains the information necessary
+for parallelizing tasks.  In particular, tasks _x_, _y_, and _b_ may
+run in parallel.  When _x_ and _y_ are finished, _a_ and _b_ may run
+in parallel.
+
+So why, then, does +multitask+ exist?  Rake already has the
+information it needs -- why must we provide it a second time?  The
+short answer is probably that +multitask+ was easy to implement, while
+dynamically finding parallelizable tasks and safely executing them in
+separate threads requires a little more code.
+
+Rake now supports the -j option which does exactly this.  To run up to
+three tasks in parallel,
+
+  % rake -j3
+
+or equivalently,
+
+  % rake --threads 3
+
+== A Note on Dependencies
+
+Without +multitask+ or -j (or --randomize, explained below), Rake
+invokes tasks in a predetermined order.  Rakefile authors inevitably
+come to rely on this order.  Even though their dependency graph may be
+wide and hierarchical, the only graph which has ever been tested is
+linear: one which results from visiting each node in sequence.
+
+Consider
+
+   task :a => [:x, :y, :z]
+
+When Rake runs in single-threaded mode (the default), _x_,_y_,_z_ will
+be invoked <em>in that order</em> before _a_ is invoked, assuming no
+other rules exist involving these tasks.  However with -j _N_ for
+_N_ > 1, one should not expect any particular order of execution.
+Since there is no dependency specified between _x_,_y_,_z_ above, Rake
+is free to run them in any order.
+
+The problem of underspecified dependencies plagues Makefiles as well
+(GNU make has supported -j for some time).
+
+== Migrating to -j
+
+Suppose -j produces errors from insufficient dependencies in your
+Rakefile.  Do you want to bother fixing them?  If you are satisfied
+with your build time, then there is really no reason to use -j.
+
+If on the other hand your build takes twenty minutes to complete, you
+may be interested in getting the full dependency graph correct in
+order to take advantage of multiple CPUs or cores.
+
+Though Rake cannot fathom what <em>you</em> mean by a correct
+dependency, there is a tool available which may help you get closer to
+saying what you mean:
+
+  % rake --randomize[=SEED]
+
+This will randomize the order of sibling prerequisites for each task.
+When SEED is given, <tt>srand(SEED.hash)</tt> will be called so that
+the same permutations will be produced for subsequent runs with the
+same SEED.
+
+Though this option may cause an error due to underspecified
+dependencies, with SEED at least it will be an error which is exactly
+the same on each run.  In addition you'll have the major debugging
+advantage of using a single thread.
+
+== Task#invoke inside Task#invoke
+
+Parallelizing tasks means surrendering control over the
+micro-management of their execution.  Manually invoking tasks inside
+other tasks is rather contrary to this notion, throwing a monkey
+wrench into the system.  An exception will be raised when this is
+attempted in -j mode.
+
+== What is the Difference Between -j and Multitask?
+
+What if you replaced +task+ with +multitask+ everywhere in your
+Rakefile?  Isn't that the same as -j?
+
+It is effectively the same when the number of tasks is small.  However
+an all-multitask Rakefile becomes problematic as the number of tasks
+and dependencies increase.  The number of lines in the dependency
+graph would be equal to the number of threads running simultaneously.
+Multitask blindly fires off N threads for the N prerequisites of a
+task.
+
+A simplistic +multitask+ setup for compiling 100 C files might spawn
+100 threads running 100 compiler processes all at the same time.  To
+reduce the load you could make several reasonably-sized multitasks
+then tie them together with a regular +task+.  Or you could just use -j.

data/doc/rakefile.rdoc

@@ -150,6 +150,12 @@ to do extra synchronization for Rake's benefit.  However, if there are
 user data structures shared between the parallel prerequisites, the
 user must do whatever is necessary to prevent race conditions.
 
+=== Automatically Running Tasks in Parallel
+
+Rake now supports the command-line option -j for automatically
+detecting non-dependent tasks and safely executing them in parallel.
+See parallel.rdoc.
+
 == Tasks with Arguments
 
 Prior to version 0.8.0, rake was only able to handle command line

data/install.rb

@@ -85,4 +85,4 @@ end
 
 # and the executable
 
-installBIN("bin/drake", "drake")
+installBIN("bin/rake", "rake")

data/lib/rake.rb

@@ -29,7 +29,7 @@
 # as a library via a require statement, but it can be distributed
 # independently as an application.
 
-RAKEVERSION = '0.8.4.1.1.0'
+RAKEVERSION = '0.8.4.1.2.0'
 
 require 'rbconfig'
 require 'fileutils'
@@ -37,7 +37,6 @@ require 'singleton'
 require 'monitor'
 require 'optparse'
 require 'ostruct'
-require 'rake/parallel'
 
 require 'rake/win32'
 
@@ -408,13 +407,13 @@ module Rake
   # InvocationChain tracks the chain of task invocations to detect
   # circular dependencies.
   class InvocationChain
+    attr_reader :value # :nodoc:
+
     def initialize(value, tail)
       @value = value
       @tail = tail
     end
 
-    attr_reader :value # :nodoc:
-
     def member?(obj)
       @value == obj || @tail.member?(obj)
     end
@@ -580,74 +579,46 @@ module Rake
       self
     end
 
-    def base_invoke(*args) #:nodoc:
+    def invoke_serial(*args) # :nodoc:
       task_args = TaskArguments.new(arg_names, args)
       invoke_with_call_chain(task_args, InvocationChain::EMPTY)
     end
 
     # Invoke the task if it is needed.  Prerequites are invoked first.
     def invoke(*args)
-      if application.num_threads == 1
-        base_invoke(*args)
+      if application.options.threads == 1
+        invoke_serial(*args)
       else
-        if application.parallel.lock.locked?
-          raise "Calling Task#invoke within a task is not allowed."
-        end
-        application.parallel.lock.synchronize {
-          application.parallel.tasks.clear
-          application.parallel.needed.clear
-          base_invoke(*args)
-          application.invoke_parallel(self.name)
-        }
+        invoke_parallel(*args)
       end
     end
-
+    
     # Same as invoke, but explicitly pass a call chain to detect
     # circular dependencies.
     def invoke_with_call_chain(task_args, invocation_chain) # :nodoc:
       new_chain = InvocationChain.append(self, invocation_chain)
-      @lock.synchronize do
-        if application.options.trace and application.num_threads == 1
-          puts "** Invoke #{name} #{format_trace_flags}"
-        end
-        return if @already_invoked
-        @already_invoked = true
-
-        if application.options.randomize
-          @prerequisites = @prerequisites.sort_by { rand }
-        end
-
-        if application.num_threads == 1
+      if application.options.threads == 1
+        @lock.synchronize do
+          return unless prepare_invoke
           invoke_prerequisites(task_args, new_chain)
           execute(task_args) if needed?
-        else
-          #
-          # Parallel mode -- gather tasks for batch execution.
-          #
-          # Either the task knows it's needed or we've marked it as
-          # needed.
-          #
-          # Why do we manually mark tasks as needed?  Since this is a
-          # dry run, files are not created or modified.  Therefore the
-          # 'needed?' result does not propagate through the recursion.
-          #
-          prereqs = invoke_prerequisites_parallel(task_args, new_chain)
-          if needed? or application.parallel.needed[self]
-            application.parallel.tasks[name] = [task_args, prereqs]
-            unless invocation_chain == InvocationChain::EMPTY
-              application.parallel.needed[invocation_chain.value] = true
-            end
-          end
         end
+      else
+        return unless prepare_invoke
+        invoke_with_call_chain_collector(task_args, new_chain, invocation_chain)
       end
     end
     protected :invoke_with_call_chain
 
-    def invoke_prerequisite(prereq_name, task_args, invocation_chain) #:nodoc:
-      prereq = application[prereq_name, @scope]
-      prereq_args = task_args.new_scope(prereq.arg_names)
-      prereq.invoke_with_call_chain(prereq_args, invocation_chain)
-      prereq
+    def prepare_invoke # :nodoc:
+      if application.options.randomize
+        @prerequisites = @prerequisites.sort_by { rand }
+      end
+      if application.options.trace
+        puts "** Invoke #{name} #{format_trace_flags}"
+      end
+      return if @already_invoked
+      @already_invoked = true
     end
 
     # Invoke all the prerequisites of a task.
@@ -657,14 +628,13 @@ module Rake
       }
     end
 
-    # Parallel dry-run accumulator.
-    # This also serves to circumvent MultiTask#invoke_prerequisites.
-    def invoke_prerequisites_parallel(task_args, invocation_chain) #:nodoc:
-      @prerequisites.map { |n|
-        invoke_prerequisite(n, task_args, invocation_chain)
-      }
+    def invoke_prerequisite(prereq_name, task_args, invocation_chain) #:nodoc:
+      prereq = application[prereq_name, @scope]
+      prereq_args = task_args.new_scope(prereq.arg_names)
+      prereq.invoke_with_call_chain(prereq_args, invocation_chain)
+      prereq
     end
-      
+
     # Format the trace flags for display.
     def format_trace_flags
       flags = []
@@ -1009,21 +979,6 @@ def import(*fns)
   end
 end
 
-#
-# +seq+ : Force tasks to be executed sequentially.
-#
-def seq
-  Rake::SEQ_LAMBDA
-end
-module Rake
-  SEQ_LAMBDA = lambda { |*task_names|
-    (1...task_names.size).each { |n|
-      task task_names[n] => task_names[n - 1]
-    }
-    task_names.last
-  }
-end
-
 # ###########################################################################
 # This a FileUtils extension that defines several additional commands to be
 # added to the FileUtils utility functions.
@@ -1754,22 +1709,12 @@ module Rake
     attr_accessor :last_description
     alias :last_comment :last_description    # Backwards compatibility
 
-    attr_accessor :num_threads
-    attr_reader :parallel
-
     def initialize
       super
       @tasks = Hash.new
       @rules = Array.new
       @scope = Array.new
       @last_description = nil
-
-      @num_threads = 1
-
-      @parallel = Struct.new(:tasks, :needed, :lock).new
-      @parallel.tasks = Hash.new
-      @parallel.needed = Hash.new
-      @parallel.lock = Mutex.new
     end
 
     def create_rule(*args, &block)
@@ -2024,6 +1969,26 @@ module Rake
 
   end # TaskManager
 
+  #
+  # Lazily pull in the parallelizing code
+  #
+  class Options < OpenStruct  # :nodoc:
+    attr_reader :threads
+
+    def initialize
+      super
+      @threads = 1
+    end
+
+    def threads=(n)
+      if n > 1 and require('rake/parallel')
+        Task.module_eval { include Parallel::TaskMixin }
+        Application.module_eval { include Parallel::ApplicationMixin }
+      end
+      @threads = n
+    end
+  end
+
   ######################################################################
   # Rake main application object.  When invoking +rake+ from the
   # command line, a Rake::Application object is created and run.
@@ -2118,7 +2083,7 @@ module Rake
 
     # Application options from the command line
     def options
-      @options ||= OpenStruct.new
+      @options ||= Options.new
     end
 
     # private ----------------------------------------------------------------
@@ -2306,17 +2271,8 @@ module Rake
           "Execute some Ruby code, then continue with normal task processing.",
           lambda { |value| eval(value) }            
         ],
-        ['--threads', '-j N', "Specifies the number of threads to run simultaneously.",
-          lambda { |value| self.num_threads = value.to_i }
-        ],
-        ['--randomize[=SEED]', "Randomize task prerequisite orders",
-          lambda { |value|
-            MultiTask.class_eval { remove_method(:invoke_prerequisites) }
-            options.randomize = true
-            if value
-              srand(value.hash)
-            end
-          }
+        ['--threads', '-j N', "Run up to N independent tasks simultaneously in separate threads.",
+          lambda { |value| options.threads = value.to_i }
         ],
         ['--libdir', '-I LIBDIR', "Include LIBDIR in the search path for required modules.",
           lambda { |value| $:.push(value) }
@@ -2338,6 +2294,13 @@ module Rake
           "Auto-import any .rake files in RAKELIBDIR. (default is 'rakelib')",
           lambda { |value| options.rakelib = value.split(':') }
         ],
+        ['--randomize[=SEED]', "Randomize the order of sibling prerequisites.",
+          lambda { |value|
+            options.randomize = true
+            MultiTask.class_eval { remove_method(:invoke_prerequisites) }
+            srand(value.hash) if value
+          }
+        ],
         ['--require', '-r MODULE', "Require MODULE before executing rakefile.",
           lambda { |value|
             begin