tap 0.8.0 → 0.9.0

data/Basic Overview

@@ -0,0 +1,151 @@
+= The Tap Application (Tap::App)
+
+Instances of Tap::App coordinate the execution of tasks.  Normally a script will only need and use a single instance (often Tap::App.instance), but there is no reason why multiple instances could not be used.  Apps basically consist of reference to a root directory, and task queue.  
+
+=== Root Directory (Tap::Root)
+
+The root directory establishes the base of a standard directory structure that facilitates configuration, dependency loading, and file handling.  Apps 'live' in the directory structure, but are not confined to it.  Code associated with the root directory allows you to alias directories, basically meaning that the conceptual standard directories can be put anywhere in the file system.  
+
+The basic structure looks like this:
+
+  /path/to/root_dir
+  |- config
+  |   `- sample
+  |       `- task.yml             # a default config file for Sample::Task
+  |- lib
+  |   `- sample
+  |       `- task.rb              # a class file like above
+  |- tap.yml                      # a configuration file for Tap::App
+
+All these files/directories are optional (and there are others that could be added), but for illustration say you instantiated an App to the directory above:
+
+  require 'tap'
+  
+  app = Tap::App.new :root => '/path/to/root_dir'
+  app.root                                      # => '/path/to/root_dir'
+  app['config']                                 # => '/path/to/root_dir/config'
+  app.filepath('config', 'sample/task.yml')     # => '/path/to/root_dir/config/sample/task.yml'
+  
+Then if you wanted to alias the config directory somewhere else:
+
+  # set a new directory relative to root
+  app['config'] = 'alt'
+  app['config']                                 # => '/path/to/root_dir/alt'
+  app.filepath('config', 'sample/task.yml')     # => '/path/to/root_dir/alt/sample/task.yml'
+  
+  # set an absolute path to anywhere
+  app['config', true] = '/path/somewhere/else'
+  app['config']                                 # => '/path/somewhere/else'
+  app.filepath('config', 'sample/task.yml')     # => '/path/somewhere/else/sample/task.yml'
+  
+While very simple, this ability to reference files using directory aliases is useful, powerful, and forms the basis of the Tap execution environment.  All directory aliases and environment configurations can be defined in the 'tap.yml' file.
+
+=== Task Queue
+
+Apps coordinate the execution of tasks through a queue.  Although referred to here as a 'task queue', the queue is actually just a stack of methods to run and the inputs to the methods.  Any methods from any object can be enqued; tasks do not need to be involved.  
+
+During a run the enqued methods are run sequentially.  When multithreading, multiple methods can be run at once, but normally methods are run one at a time on the same thread.  
+
+= Tasks (Tap::Task)
+
+Tasks can be defined within a script using a syntax similar to Rake:
+
+  t = Tap::Task.new do |task|
+    # .. do something ...
+  end
+  t.execute
+  
+Unlike a Rake task, Tap tasks can be made to take inputs:
+
+  t = Tap::Task.new do |task, one, two|
+    # .. do something with the inputs ...
+  end
+  t.execute(1,2)
+  
+Tasks are designed for subclassing.  A task class is structured like this (with annotation):
+  
+  module Sample
+    # == Description
+    # This documentation is available in RDoc, as well as from the command line.
+    # === Usage
+    #  % tap run sample/task
+    #
+    # By default, usage info comes up as well...
+    #
+    class Task < Tap::Task
+
+      # Configurations are defined like this.  Tap makes
+      # the config documentation available in Rdoc and
+      # in the command line.
+      #
+      # The default 'config' creates accessors, as can 
+      # be seen in the process method.
+      
+      # Full documentation for the configuration
+      # can be written here.
+      config :key, "input"                            # command line documentation
+      
+      # Configurations can be transformed or 
+      # validated by an optional block.
+      config :transformation, "string" do |value|     # this config is modified to "STRING"
+        value.upcase
+      end
+      config :standard_validation, {}, &c.check(Hash) # non-hash values raise an error
+    
+      # Process receives each input from the command line
+      # and is analogous to the block in the samples above.
+      # Any number of inputs are allowed, including an 
+      # arbitrary number of inputs (using *) and no inputs.
+      def process(input)
+      
+        # log is available to record information
+        result = "#{input} was processed with #{key}"
+        log self.name, result
+
+        result
+      end
+    end
+  end
+
+Once instantiated, tasks can be enqued to an App or joined into a workflow.  
+      
+All task instances may have a name, which serves as a relative filepath to a configuration file and other associated files.  When the application instantiates a task by the name 'sample/task' it sets the instance configs from 'config/sample/task.yml'.  Nil can be used to specify no config file. 
+
+Names need not be unique.  Multiple task instances, even from different classes, may be named the same and thus get their configurations from the same file.  A map of task names to task classes can be set up in an app, allowing easy lookup and task instantiation through the app.task method:
+
+  app = Tap:App.instance
+
+  # set up a mapping between a name and a task class
+  app.map["alt"] = Sample::Task 
+
+  t1 = app.task("sample/task")     # Sample::Task configured from 'config/sample/task.yml'
+  t2 = app.task("alt")             # Sample::Task configured from 'config/alt.yml'
+  
+  # the idea is the same if you manually make a task... 
+
+  t3 = Tap::Task.new("alt")        # Tap::Task configured from 'config/alt.yml'
+  
+  t1.class     # => Sample::Task
+  t2.class     # => Sample::Task
+  t3.class     # => Tap::Task
+
+= Workflows (Tap::Workflow)
+
+Once instantiated, tasks can be joined into workflows.  The workflow model used by Tap is very simple; workflows are implemented using the App queue, whereby each tasks enques the next task (or tasks) in the workflow when the task completes.  
+
+Each task can be assigned an on_complete block to do so.  Arbitrary workflow logic can be used to join tasks in this way, but by default Tap supports sequencing, forking, and merging.  If a task has no on_complete block, App collects the unhandled results in app.aggregator so they can be handled somewhere else.
+
+Workflows are not Tasks, but work exactly like Tasks.  Workflows can be configured, enqued, used in or joined to other workflows, and subclassed.
+
+= Auditing (Tap::Support::Audit)
+
+Tap tracks inputs as they are modified by various tasks.  At the end of a run, the individual results can be tracked back to it's original value, through the tasks that modified it.  This auditing can be very useful when workflows diverge (as they often do), adding or subtracting steps depending on the input.
+
+Auditing is largely invisible except in on_complete blocks.  On complete blocks receive the audited results so that this information can be used, as needed, to make decisions.  
+
+  t = Task.new
+  t.on_complete do |_result|      # _result is an Audit instance
+    _result._current              # the current value
+    _result._original             # the original value
+  end
+

data/Command Reference

@@ -0,0 +1,99 @@
+= Tap from the Command Line
+
+The tap command is the gateway for the execution of sub-commands.  To get help for tap, type:
+
+  % tap --help
+  
+The tap command sets up the execution environment from 'tap.yml', if it exists, and passes control to the specified sub-command.  Sub-command help can be obtained using:
+
+  % tap [sub-command] --help
+
+== run
+
+Run configures, enqueues, and executes tasks.  Run has a rich syntax allowing the specification of any number of tasks with configurations and inputs, but simplifies under most circumstances.  Several examples illustrate the key points:
+
+  % tap run sample/task 
+  % tap run -- sample/task --key=value input_one -- another/task input_two
+  
+The second statement specifies two tasks with inputs, and specifies a configuration for sample/task.  As can be seen, run separates tasks using a double-dash, the standard option break.  Options for run itself can be specified before the first option break.
+
+  % tap run --debug -- sample/task --key=value
+  
+Here run receives the '--debug' option and sample/task receives the '--key=value' option'.  <em>NOTE</em> it's always a good idea to include the first option break to formally signify when configuration of run stops.  Otherwise you may be confused when the following commands both produce the help for run:
+
+  % tap run --help
+  % tap run sample/task --help
+
+Inputs work the same way.  For example:
+
+  % tap run -- sample/task --key=value one -- another/task two three
+  
+Specifies the following:
+
+  t1 = app.task('sample/task', :key => 'value')
+  t1.enq('one')
+  
+  t2 = app.task('another/task')
+  t2.enq('two', 'three') 
+  
+Any number of tasks, configurations, and inputs may be specified in this way.  As a special note, rake tasks can be specified as well.  If the app cannot find a tap task by the specified name, rake is loaded (using the same loading  rules as the rake command) and run tries to find a corresponding rake task. ENV options can be specified in the rake statement.
+
+  % tap run -- test KEY=value
+
+=== YAML inputs
+
+Non-string inputs can be provided through run.  If an input begins with "---\n" then it is loaded as YAML into an object before being passed to a task. The syntax can be a lot to type, but is very handy to have around.  The following enques 'sample/task' with a hash input, {'number' => 1}.  
+
+On *nix, just hit enter to get the next line:
+
+  % tap run -- sample/task '---
+  > number: 1'
+
+On Windows, you need to pull some tricks to get newlines into your argument. Cleverly use a caret to ignore the next line feed:
+
+  % tap run -- sample/task '---^
+  More?
+  More? number: 1'
+  
+Notice that pressing enter _every other line_ is what actually puts the "\n" into the  parameter.  Keep using carets to enter more lines. The syntax on Windows isn't exactly pretty, but it works... with a caveat.  The latest execution script generated by rubygems (tap.bat) re-processes inputs to tap before executing the command for real.  I haven't found a workaround short of invoking tap from ruby itself:
+
+  % ruby C:/ruby/bin/tap run -- sample/task '---^
+  More?
+  More? number: 1'
+
+== generate/destroy
+
+Generate and destory launch generator scripts, the same as in Rails.  By default Tap provides generators for:
+
+root:: the basic Tap directory structure
+task:: a Tap:Task and test for the task
+file_task:: a Tap::FileTask and test for the task
+workflow:: a Tap::Workflow and test for the workflow
+config:: a yaml config file for the specified task 
+command:: a new subcommand
+generator:: a new generator
+
+Some examples:
+
+  % tap generate root .
+  % tap generate task sample_task
+  % tap generate config sample_task
+  
+  % tap destroy config sample_task
+  % tap destroy task sample_task
+  % tap destroy root .
+
+== console
+
+Console opens an irb session with Tap loaded and configured using 'tap.yml'.  Console defines a variable 'app' referencing Tap::App.instance, for easy access.
+
+  % tap console
+  irb(main):001:0> app.log(:hello)
+    I[17:18:53]              hello
+  => true
+  irb(main):002:0>
+ 
+For actively testing your code, remember that many of your files will be loaded using Dependencies (from {ActiveSupport}[http://rubyforge.org/projects/activesupport/]).  You can modify your code, clear the  dependencies, and observe the changes within a single session.  The command is:
+
+  irb(main):003:0> app.reload
+  # => [... an array of unloaded constants ...]

data/History

@@ -24,5 +24,29 @@ Documentation is still patchy, but improving.
 * documentation, documentation, documentation
 * many other things as well...
 
+== 0.9.0 / 2008-04-09 revision 246
 
+Major update to Tap. Many changes in this release ARE NOT BACKWARD COMPATIBLE.  
+
+* Reworked Task and App such that methods,  are
+  enqued and executed.  Made the Executable module 
+  to allow any Method to be enqued and executed.
+* App now aggregates results for methods that have no 
+  on_complete block; enabled access through App#results 
+  and App#_results.
+* Removed iteration from tasks
+* Task no longer templates app.config_templates.  Templating code
+  removed from this distribution.
+* Work to improve audit usability and visualization
+* Removed methods forwarding from Audit to Audit#_current,
+  as well as ambiguous Audit methods
+* Root [] now returns expanded paths unchanged. Example:
+  app['relative/path']  # => File.join(app.root, 'relative/path')
+  app['/expanded/path'] # => '/expanded/path'
+* Generalized rake support; now supports version 0.8.1
+* Fixes in generators
+* Improvements in running of multithread tasks
+* Removed condition blocks from tasks
+* many other things also...
 
+  

data/MIT-LICENSE

@@ -1,5 +1,5 @@
 Copyright (c) 2006-2007, Regents of the University of Colorado.
-Developer:: Simon Chiang, Biomolecular Structure Program
+Developer:: Simon Chiang, Biomolecular Structure Program, Hansen Lab
 Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this

data/README

@@ -2,41 +2,44 @@
 
 A framework for creating configurable, distributable, and easy-to-use tasks and workflow applications.
 
-See the {website}[http://tap.rubyforge.org] for more information and tutorials.
+See the {website}[http://tap.rubyforge.org] for more information.
 
 == Description
 
-Tap provides a framework facilitaing the definition, configuration, and joining of tasks into workflows.
-Tap provides a lot of framework niceties like generators and libraries geared towards testing tasks, 
-generating documentation, as well as task execution and distribution.  Tasks are immediately available 
-from the comand line through though the 'tap' command.
+Tap provides a framework to define, configure, and join tasks into workflows. Tap provides a lot of niceties like generators and libraries geared towards testing tasks, generating documentation, as well as task execution and distribution.  Tasks are immediately available from the command line through the 'tap' command.
 
-Running a task named 'sample/task' is simply:
-
-  % tap run sample/task
+Make a task:
 
-Packaging tasks into an executables can be accomplished using {rubyscript2exe}[http://rubyforge.org/projects/rubyscript2exe/].  
-First you generate a packaging script, then use rubyscript2exe to make the executable.
+  % tap generate task sample/task
 
-  % tap generate package sample "run sample/task"
-  % rubyscript2exe pkg/sample.rb
+Run the task setting the 'key' configuration:
   
-The result is a fully contained, documented executable that runs on other computers
-of the same platform, without Tap or even Ruby[http://www.ruby-lang.org/en/] being installed.
+  % tap run -- sample/task --key=value input
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[10:53:11]        sample/task input was processed with value
+  
+Interact with the task using irb:
+
+  % tap console
+  irb(main):001:0> t = app.task('sample/task', :key => 'value')
+  irb(main):002:0> t.enq('input')
+  irb(main):003:0> app.run
+    I[10:55:17]        sample/task input was processed with value
 
 Additional Notes:
 
 - Tap can incorporate and run Rake[http://rake.rubyforge.org/] tasks as well as Tap tasks. 
 - Tap task libraries are designed for distribution as gems.  
-- Tap is tested on both MRI (the standard Ruby interpreter) and JRuby[http://jruby.codehaus.org/].  
+- Tap is tested on both MRI (the standard Ruby interpreter, versions 1.8.6 and 1.9.0) and JRuby[http://jruby.codehaus.org/].  
 
 === Bugs/Known Issues
 
-- The threading model for executing multithreaded tasks does not currently make shared resources
-  like Tap::App completely thread-safe and should be considered *experimental*.  If you're concerned, 
-  don't multithread your tasks.
 - Some inconsequential tests on JRuby fail due to bugs in JRuby itself.
-- TDoc documentation is still under development.  You may find small typographic bugs.
+- Several patches are required so that Tap runs properly on MRI 1.9.0.  These   
+  will likely resolve themselves as 1.9.0 becomes stable.
+- Some code is still in progress; in general the well-documented code is stable.
 
 == Info 
 
@@ -51,49 +54,18 @@ Tap is available as a gem on RubyForge[http://rubyforge.org/projects/tap].  Use:
 
   % gem install tap
 
-== Usage
-
-<b>See the tutorial for information on using Tap programatically.</b>
-
-Begin by creating a tap root directory structure:
-
-  % tap generate root /path/to/root
-  % cd /path/to/root
-  
-Make a task:
-
-  % tap generate task sample/task
+See the tutorial or check out the {website}[http://tap.rubyforge.org] to get started using Tap programatically.
 
-Test the task (in fact invokes the rake test task):
- 
-  % tap run test 
-  
-Get help for the task:
+== Credits
 
-  % tap run -- sample/task --help
-  
-Run the task, setting the 'key' configuration and passing some inputs:
-  
-  % tap run -- sample/task --label=processing one two
-  
-Package the task into an executable (rubyscript2exe must be installed
-and may not work on platforms where rubyscript2exe is squirrelly, ie
-OS X)
+A great deal of inspiration came from other excellent open-source projects including Rake[http://rake.rubyforge.org] and {Ruby on Rails}[http://www.rubyonrails.org].  
 
-  % tap generate package sample "run sample/task"
-  % rubyscript2exe pkg/sample.rb
+Thanks to my advisor {Dr. Kirk Hansen}[http://hsc-proteomics.uchsc.edu/hansenlab/] for his patience in letting me work on this project, and for putting together an awesome lab.
 
-== Credits
+Most of all I would like to <b>thank my family</b> for their never-ending support.
 
-A great deal of inspiration came from other excellent open-source projects including
-Rake[http://rake.rubyforge.org] and {Ruby on Rails}[http://www.rubyonrails.org].  
-I also want to give credit to RubyGems[http://www.rubygems.org] and 
-RubyForge[http://rubyforge.org] because they are crucial parts of what make Ruby a 
-great language to work with.
+=== Code
 
-Thanks to my advisor {Dr. Kirk Hansen}[http://hsc-proteomics.uchsc.edu/hansenlab/] 
-for his patience in letting me work on this project, and for putting together an awesome
-lab.
+Tap uses ActiveSupport for dependencies loading and a variety of other extensions.  Tap uses Rails generators and redistributes the code in the gem, to avoid an additional dependency.
 
-Most of all I would like to thank my family for their never-ending support.
 

data/Rakefile

@@ -4,7 +4,8 @@ require 'rake/rdoctask'
 require 'rake/gempackagetask'
 
 $:.unshift "./lib"
-require 'tap/version.rb'
+require 'tap/constants'
+require 'tap/patches/rake/testtask.rb'
 
 desc 'Default: Run tests.'
 task :default => :test
@@ -19,12 +20,12 @@ spec = Gem::Specification.new do |s|
 	s.version = Tap::VERSION
 	s.author = "Simon Chiang"
 	s.email = "simon.chiang@uchsc.edu"
-	s.homepage = "http://rubyforge.org/projects/tap/"
+	s.homepage = Tap::WEBSITE
 	s.platform = Gem::Platform::RUBY
 	s.summary = "A framework for configurable, distributable, and easy-to-use tasks and workflow applications."
-	s.files = File.read("Manifest.txt").split("\n").select {|f| f !~ /^\s*#/ && File.file?(f) }
+	s.files = File.read("Manifest.txt").split("\n").select {|f| f !~ /^\s*#/ && !File.directory?(f) }
 	s.require_path = "lib"
-	s.autorequire = "tap"
+	s.rubyforge_project = "tap"
 	s.test_file = "test/tap_test_suite.rb"
 	s.bindir = "bin"
 	s.executables = ["tap"]
@@ -32,7 +33,7 @@ spec = Gem::Specification.new do |s|
   
 	s.has_rdoc = true
   s.rdoc_options << '--title' << 'Tap - Task Application' << '--main' << 'README' 
-	s.extra_rdoc_files = ["README", "MIT-LICENSE", "History", "Tutorial"]
+	s.extra_rdoc_files = ["README", "MIT-LICENSE", "History", "Tutorial", "Basic Overview", "Command Reference"]
   s.add_dependency("activesupport", ">=2.0.1")
 end
 
@@ -49,7 +50,14 @@ end
 
 desc 'Run tests.'
 Rake::TestTask.new(:test) do |t|
-  t.pattern = File.join('test', ENV['subset'] || '', ENV['pattern'] || '**/*_test.rb')
+  t.test_files = if ENV['check']
+    Dir.glob( File.join('test',  "**/*#{ENV['check']}*_check.rb") )
+  else
+    Dir.glob( File.join('test', ENV['pattern'] || '**/*_test.rb') ).delete_if do |filename|
+      filename =~ /test\/check/
+    end
+  end
+  
   t.verbose = true
   t.warning = true
 end
@@ -66,45 +74,30 @@ Rake::RDocTask.new(:rdoc) do |rdoc|
   rdoc.title    = 'tap'
   rdoc.template = 'tap/support/tdoc/tdoc_html_template' 
   rdoc.options << '--line-numbers' << '--inline-source' << '--fmt' << 'tdoc'
-  rdoc.rdoc_files.include('README', 'MIT-LICENSE', "History", "Tutorial")
+  rdoc.rdoc_files.include('README', 'MIT-LICENSE', "History", "Tutorial", "Basic Overview", "Command Reference")
   rdoc.rdoc_files.include('lib/tap/**/*.rb')
 end
 
 desc 'Generate website.'
-task :website => [:rdoc] do |t| 
-  require 'tap/root'
-  require 'redcloth'
-  require 'erb'
+task :website => [:rdoc] do |t|
+  require 'tap'
   
-  # setup some variables
-  page_template = File.read('website/page.erb')
-  output_dir = "pkg/website-#{Tap::VERSION}"
-  pages = FileList.new('website/pages/**/*.txt') do |list|
-    list.exclude /_sub\.txt$/
-  end
+  # temporary
+  $: << File.dirname(__FILE__) + "/tap/simple_site/lib"
+  $: << File.dirname(__FILE__) + "/tap/ddb/lib"
+  Dependencies.load_paths << File.dirname(__FILE__) + "/tap/simple_site/lib"
+  Dependencies.load_paths << File.dirname(__FILE__) + "/tap/ddb/lib"
+
+  require 'simple_site'
   
-  # remove existing dir, move in or create pages
-  rm_r output_dir if File.exists?(output_dir)
-  mkdir_p output_dir
-  cp "website/index.html", File.join(output_dir, "index.html")
-  cp_r "website/images", File.join(output_dir, "images")
-  cp_r "website/stylesheets", File.join(output_dir, "stylesheets")
-  cp_r "rdoc", File.join(output_dir, "rdoc")
+  app = Tap::App.instance
   
-  pages.each do |source|
-    base = Tap::Root.relative_filepath("./website/pages", source).chomp(".txt")
-    sub_source = source.chomp(".txt") + "_sub.txt"
-    
-    @title = base.capitalize
-    @content_main = RedCloth.new(File.read(source)).to_html
-    @content_sub = File.exists?(sub_source) ? RedCloth.new(File.read(sub_source)).to_html : ""
-    
-    target = File.join(output_dir, base + ".html") 
-    File.open(target, "w") do |file|
-      file << ERB.new(page_template).result
-    end
-  end
+  app['pkg'] = "pkg/website-#{Tap::VERSION}"
+  app['content'] = 'website/content'
+  app['views'] = 'website/views'
+  app.run 'simple_site/compile', '.'
   
+  cp_r "rdoc", app.filepath('pkg', "rdoc")
 end
 
 #