tap 0.7.9 → 0.8.0

data/History

@@ -0,0 +1,28 @@
+== 0.7.9 / 2007-09-14
+
+* Initial testing release with partial documentation
+
+== 0.8.0 / 2007-12-08 revision 138
+
+Major update to Tap. Many changes in this release ARE NOT BACKWARD COMPATIBLE.  
+Documentation is still patchy, but improving.
+
+* Expanded/reworked application configuration, allowing specification of gems
+  to be loaded as task libraries.
+* Default app config file is now tap.yml (from app.yml)
+* Updated and improved generators
+* Included rails_generators in distribution, removing rails dependency
+* Reworked many classes to clean up interface
+* Reworked testing methods
+* Improved configuration for tasks
+* Optimized loading
+* Reworked threading model
+* Addition of TDoc documentation
+* Reworked tap command and subcommands
+* Added packaging into executables
+* Updated to ActiveSupport 2.0.1
+* documentation, documentation, documentation
+* many other things as well...
+
+
+

data/MIT-LICENSE

@@ -1,6 +1,6 @@
 Copyright (c) 2006-2007, Regents of the University of Colorado.
 Developer:: Simon Chiang, Biomolecular Structure Program
-Support:: UCHSC School of Medicine Deans Academic Enrichment Fund
+Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this
 software and associated documentation files (the "Software"), to deal in the Software

data/README

@@ -1,22 +1,48 @@
-= Tap
+= Tap (Task Application)
 
-Task Application: A framework for configurable, file based, web ready workflow applications.
+A framework for creating configurable, distributable, and easy-to-use tasks and workflow applications.
+
+See the {website}[http://tap.rubyforge.org] for more information and tutorials.
 
 == Description
 
-Tap fundamentally is a basic workflow engine wherein you can define tasks and rules to pass
-outputs from one task into another.  Tap uses a standardized folder layout to assist in 
-configuring tasks and managing files.  
+Tap provides a framework facilitaing the definition, configuration, and joining of tasks into workflows.
+Tap provides a lot of framework niceties like generators and libraries geared towards testing tasks, 
+generating documentation, as well as task execution and distribution.  Tasks are immediately available 
+from the comand line through though the 'tap' command.
+
+Running a task named 'sample/task' is simply:
+
+  % tap run sample/task
+
+Packaging tasks into an executables can be accomplished using {rubyscript2exe}[http://rubyforge.org/projects/rubyscript2exe/].  
+First you generate a packaging script, then use rubyscript2exe to make the executable.
+
+  % tap generate package sample "run sample/task"
+  % rubyscript2exe pkg/sample.rb
+  
+The result is a fully contained, documented executable that runs on other computers
+of the same platform, without Tap or even Ruby[http://www.ruby-lang.org/en/] being installed.
+
+Additional Notes:
+
+- Tap can incorporate and run Rake[http://rake.rubyforge.org/] tasks as well as Tap tasks. 
+- Tap task libraries are designed for distribution as gems.  
+- Tap is tested on both MRI (the standard Ruby interpreter) and JRuby[http://jruby.codehaus.org/].  
 
-Tap can be run from the console or from an integrated web server built on the camping[link to camping]
-framework.  The Tap Server makes it easy to build tools and share them within an organization,
-or with a wider audience as you please.
+=== Bugs/Known Issues
+
+- The threading model for executing multithreaded tasks does not currently make shared resources
+  like Tap::App completely thread-safe and should be considered *experimental*.  If you're concerned, 
+  don't multithread your tasks.
+- Some inconsequential tests on JRuby fail due to bugs in JRuby itself.
+- TDoc documentation is still under development.  You may find small typographic bugs.
 
 == Info 
 
 Copyright (c) 2006-2007, Regents of the University of Colorado.
-Developer:: Simon Chiang, Biomolecular Structure Program
-Support:: UCHSC School of Medicine Deans Academic Enrichment Fund
+Developer:: {Simon Chiang}[http://bahuvrihi.wordpress.com], {Biomolecular Structure Program}[http://biomol.uchsc.edu/], {Hansen Lab}[http://hsc-proteomics.uchsc.edu/hansenlab/] 
+Support:: CU Denver School of Medicine Deans Academic Enrichment Fund
 Licence:: MIT-Style
 
 == Installation
@@ -24,48 +50,50 @@ Licence:: MIT-Style
 Tap is available as a gem on RubyForge[http://rubyforge.org/projects/tap].  Use:
 
   % gem install tap
-  
-In addition, Tap may be installed through a {Firefox Add-On}[http://prosperity.uchsc.edu/tap/addon].  
-The Tap add-on provides shortcuts for starting and running Tap in a console or as a server.  
 
 == Usage
 
-Begin by creating a Tap folder:
+<b>See the tutorial for information on using Tap programatically.</b>
 
-  % tap --generate root path/to/folder
-  
-This command generates the standard file structure for Tap, including folders for 
-configurations, data, task libraries, documentation, and testing.  Navigate into your
-folder and generate a task:
+Begin by creating a tap root directory structure:
 
-  % tap --generate task say_hello
+  % tap generate root /path/to/root
+  % cd /path/to/root
   
-The task generator makes a stub configuration file, task, and test.  Open up each file to 
-see the basics of task creation.  Everthing is set up for the test run, so give it a go:
-
-  % tap say_hello "from me"
-  # => I[hh::mm::ss]       message hello from me!
+Make a task:
 
-You can start an IRB session to interact with your tap tasks...
+  % tap generate task sample/task
 
-  % tap --console
-  > Tap::App.run(:say_hello, "from me")
-  # => I[hh::mm::ss]       message hello from me!
+Test the task (in fact invokes the rake test task):
+ 
+  % tap run test 
   
-Or start the server...
+Get help for the task:
 
-  % tap --server
+  % tap run -- sample/task --help
+  
+Run the task, setting the 'key' configuration and passing some inputs:
   
-Now you can go to {http://localhost:3000/tap}[http://localhost:3000/tap] to see your
-Tap Server in action.
-
-== Auditing
-== Notes
-
-Tap takes a simplified approach to workflows where the idea is to setup 
-tasks, their configs, conditions, and on_complete behavior before running,
-and then to let the application act out this predefined logic.  Changing 
-task setup during execution can naturally cause unexpected behavior.  
-Various safeguards should let you get away with this inadvisable practice,
-but on the other hand you may just thread lock the whole caboodle.
+  % tap run -- sample/task --label=processing one two
+  
+Package the task into an executable (rubyscript2exe must be installed
+and may not work on platforms where rubyscript2exe is squirrelly, ie
+OS X)
+
+  % tap generate package sample "run sample/task"
+  % rubyscript2exe pkg/sample.rb
+
+== Credits
+
+A great deal of inspiration came from other excellent open-source projects including
+Rake[http://rake.rubyforge.org] and {Ruby on Rails}[http://www.rubyonrails.org].  
+I also want to give credit to RubyGems[http://www.rubygems.org] and 
+RubyForge[http://rubyforge.org] because they are crucial parts of what make Ruby a 
+great language to work with.
+
+Thanks to my advisor {Dr. Kirk Hansen}[http://hsc-proteomics.uchsc.edu/hansenlab/] 
+for his patience in letting me work on this project, and for putting together an awesome
+lab.
+
+Most of all I would like to thank my family for their never-ending support.
 

data/Rakefile

@@ -3,31 +3,15 @@ require 'rake/testtask'
 require 'rake/rdoctask'
 require 'rake/gempackagetask'
 
+$:.unshift "./lib"
+require 'tap/version.rb'
+
 desc 'Default: Run tests.'
 task :default => :test
 
-desc 'Run tests.'
-Rake::TestTask.new(:test) do |t|
-  t.libs << 'lib'
-  t.pattern = File.join('test', ENV['subset'] || '', ENV['pattern'] || '**/*_test.rb')
-  t.verbose = true
-  t.warning = true
-end
-
-desc 'Generate documentation.'
-Rake::RDocTask.new(:rdoc) do |rdoc|
-  rdoc.rdoc_dir = 'rdoc'
-  rdoc.title    = 'tap'
-  rdoc.options << '--line-numbers' << '--inline-source'
-  rdoc.rdoc_files.include('README', 'MIT-LICENSE')
-  rdoc.rdoc_files.include('lib/tap/**/*.rb')
-end
-
 #
 # Gem specification
 #
-require './lib/tap/version.rb'
-
 Gem::manage_gems
 
 spec = Gem::Specification.new do |s|
@@ -37,8 +21,8 @@ spec = Gem::Specification.new do |s|
 	s.email = "simon.chiang@uchsc.edu"
 	s.homepage = "http://rubyforge.org/projects/tap/"
 	s.platform = Gem::Platform::RUBY
-	s.summary = "Framework for configurable, file-based applications."
-	s.files = Dir.glob("{bin,test,lib}/**/*") + ["MIT-LICENSE", "README", "Rakefile"]
+	s.summary = "A framework for configurable, distributable, and easy-to-use tasks and workflow applications."
+	s.files = File.read("Manifest.txt").split("\n").select {|f| f !~ /^\s*#/ && File.file?(f) }
 	s.require_path = "lib"
 	s.autorequire = "tap"
 	s.test_file = "test/tap_test_suite.rb"
@@ -48,67 +32,100 @@ spec = Gem::Specification.new do |s|
   
 	s.has_rdoc = true
   s.rdoc_options << '--title' << 'Tap - Task Application' << '--main' << 'README' 
-	s.extra_rdoc_files = ["README", "MIT-LICENSE"]
-  s.add_dependency("rails", ">= 1.2.3")  # only for generators?
-  s.add_dependency("activesupport", ">=1.4.2")
+	s.extra_rdoc_files = ["README", "MIT-LICENSE", "History", "Tutorial"]
+  s.add_dependency("activesupport", ">=2.0.1")
 end
 
 Rake::GemPackageTask.new(spec) do |pkg|
 	pkg.need_tar = true
 end
 
-#
-# Hoe tasks
-#
-
-desc 'Install the package as a gem'
-task :install_gem => [:package] do
-#task :install_gem => [:clean, :package] do
-  sh "#{'sudo ' unless WINDOZE}gem install pkg/*.gem"
+desc 'Prints the gem manifest.'
+task :print_manifest do
+  spec.files.each do |file|
+    puts file
+  end
 end
 
-desc "Publish RDoc to RubyForge"
-task :publish_docs => [:clean, :docs] do
-  config = YAML.load(File.read(File.expand_path("~/.rubyforge/user-config.yml")))
-  host = "#{config["username"]}@rubyforge.org"
+desc 'Run tests.'
+Rake::TestTask.new(:test) do |t|
+  t.pattern = File.join('test', ENV['subset'] || '', ENV['pattern'] || '**/*_test.rb')
+  t.verbose = true
+  t.warning = true
+end
 
-  remote_dir = "/var/www/gforge-projects/#{rubyforge_name}/#{remote_rdoc_dir}"
-  local_dir = 'doc'
+#
+# Documentation tasks
+#
 
-  sh %{rsync #{rsync_args} #{local_dir}/ #{host}:#{remote_dir}}
+desc 'Generate documentation.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  require 'tap/support/tdoc'
+  
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'tap'
+  rdoc.template = 'tap/support/tdoc/tdoc_html_template' 
+  rdoc.options << '--line-numbers' << '--inline-source' << '--fmt' << 'tdoc'
+  rdoc.rdoc_files.include('README', 'MIT-LICENSE', "History", "Tutorial")
+  rdoc.rdoc_files.include('lib/tap/**/*.rb')
 end
 
-desc 'Package and upload the release to rubyforge.'
-task :release => [:package] do |t|
-  require 'rubyforge'
+desc 'Generate website.'
+task :website => [:rdoc] do |t| 
+  require 'tap/root'
+  require 'redcloth'
+  require 'erb'
   
-#task :release => [:clean, :package] do |t|
-  v = ENV["VERSION"] or abort "Must supply VERSION=x.y.z"
-  abort "Versions don't match #{v} vs #{spec.version}" if v != spec.version.to_s
-  pkg = "pkg/#{spec.name}-#{spec.version}"
-
-  if $DEBUG then
-    puts "release_id = rf.add_release #{rubyforge_name.inspect}, #{name.inspect}, #{version.inspect}, \"#{pkg}.tgz\""
-    puts "rf.add_file #{rubyforge_name.inspect}, #{name.inspect}, release_id, \"#{pkg}.gem\""
+  # setup some variables
+  page_template = File.read('website/page.erb')
+  output_dir = "pkg/website-#{Tap::VERSION}"
+  pages = FileList.new('website/pages/**/*.txt') do |list|
+    list.exclude /_sub\.txt$/
   end
+  
+  # remove existing dir, move in or create pages
+  rm_r output_dir if File.exists?(output_dir)
+  mkdir_p output_dir
+  cp "website/index.html", File.join(output_dir, "index.html")
+  cp_r "website/images", File.join(output_dir, "images")
+  cp_r "website/stylesheets", File.join(output_dir, "stylesheets")
+  cp_r "rdoc", File.join(output_dir, "rdoc")
+  
+  pages.each do |source|
+    base = Tap::Root.relative_filepath("./website/pages", source).chomp(".txt")
+    sub_source = source.chomp(".txt") + "_sub.txt"
+    
+    @title = base.capitalize
+    @content_main = RedCloth.new(File.read(source)).to_html
+    @content_sub = File.exists?(sub_source) ? RedCloth.new(File.read(sub_source)).to_html : ""
+    
+    target = File.join(output_dir, base + ".html") 
+    File.open(target, "w") do |file|
+      file << ERB.new(page_template).result
+    end
+  end
+  
+end
 
-  rf = RubyForge.new
-  puts "Logging in"
-  rf.login
+#
+# Hoe tasks
+#
 
-  c = rf.userconfig
-  c["release_notes"] = description if description
-  c["release_changes"] = changes if changes
-  c["preformatted"] = true
+desc 'Install the package as a gem'
+task :install_gem => [:package] do
+  sh "#{'sudo ' unless WINDOZE}gem install pkg/*.gem"
+end
 
-  #files = [(@need_tar ? "#{pkg}.tgz" : nil),
-  #         (@need_zip ? "#{pkg}.zip" : nil),
-  #         "#{pkg}.gem"].compact
-  
-  files = "#{pkg}.gem"
+desc "Publish Website to RubyForge"
+task :publish_website => [:website] do
+  config = YAML.load(File.read(File.expand_path("~/.rubyforge/user-config.yml")))
+  host = "#{config["username"]}@rubyforge.org"
   
-  puts "Releasing #{name} v. #{version}"
-  rf.add_release rubyforge_name, name, version, *files
+  rsync_args = "-v -c -r"
+  remote_dir = "/var/www/gforge-projects/tap"
+  local_dir = "pkg/website-#{Tap::VERSION}"
+ 
+  sh %{rsync #{rsync_args} #{local_dir}/ #{host}:#{remote_dir}}
 end
 
 desc 'Show information about the gem.'

data/Tutorial

@@ -0,0 +1,235 @@
+= Tutorial
+
+<b>Under Construction!</b>
+
+== Overview
+
+=== Tasks
+
+# insert task image
+
+Tap tasks are designed to be modular: they can be configured, they take inputs, and they 
+produce results.  On a programatic level this system is convenient for joining tasks into
+workflows using condition and decision blocks.  On an application level, the task structure
+makes it easy to incorporate tasks into configurable, user-ready scripts.
+
+A task is structured like this:
+  
+  module Sample
+    # == Documentation
+    # This documentation is available in RDoc, as well as from the
+    # command line, if you were to run it through tap, like this:
+    #   % tap run -- sample/task --help
+    #
+    # === Usage
+    # By default, usage info comes up as well...
+    #
+    class Task < Tap::Task
+
+      # Configurations are defined like this.  In this format, 
+      # this comment is used in generating customized RDoc
+      # documenting the configurations and any accessors created
+      # for them.  The default 'config' creates a config reader
+      # and writer.
+      config :config_name, "default value"      #-- a command line comment
+    
+      # Process receives each input from the command line
+      # and can do arbitrary work to make an output.
+      def process(input)
+        # ... log a message and return the input plus one ...
+        log "#{self.name} recieved #{input}"
+        input + 1
+      end
+    end
+  end
+
+Once instantiated, tasks can be joined into a workflow by the Tap application, which is
+made to work with a standard directory structure.  The basics looks like this:
+
+  tap
+  |- Rakefile
+  |- ReadMe.txt
+  |- config
+  |   `- sample
+  |       `- task.yml             # a default config file for Sample::Task
+  |- lib
+  |   `- sample
+  |       `- task.rb              # a class file like above
+  |- tap.yml                      # a configuration file for Tap::App
+  `- test
+      |- sample
+      |   `- task_test.rb         # a test file for Sample::Task
+      |- tap_test_helper.rb
+      `- tap_test_suite.rb
+
+All task instances have a name, which by default serves as the relative filepath from an 
+application directory to assoicated files.  So for instance when the application instantiates 
+a task by the name 'sample/task' it sets the task configs from 'config/sample/task.yml'.
+Any number of configurations can exist for a single task class.  In addition, Tap knows 
+how to load unknown task classes by name or via mapping...
+
+  app = Tap:App.instance
+  app.root = '/path/to/tap'
+  
+  # set up a mapping between a name and a task class
+  app.map("two" => Sample::Task)  
+
+  # the task method looks up and instantiates the proper
+  # task, with the input name and configuration (if the
+  # config file exists)
+  
+  t1 = app.task("sample/task")               # configured from 'config/sample/task.yml'
+  t2 = app.task("task_two")                    # configured from 'config/task_two.yml'
+  
+  # the idea is the same if you manually make a task... 
+  # t3 is configured from 'config/task_three.yml'
+  
+  t3 = Tap::Task.new("task_three") do |task, input| 
+    log "#{task.name} recieved #{input}"
+    input + 1
+  end 
+  
+  t1.class     # => Sample::Task
+  t2.class     # => Sample::Task
+  t3.class     # => Tap::Task
+
+=== Workflows
+
+# insert workflow image
+
+Once tasks are instantiated, they can be joined into workflows, and run with inputs.
+Tasks are joined using a condition block and a decision (on_complete) block that runs 
+when the task finishes.  Arbitrary workflow logic can be used to join tasks in this
+way, but by default Tap supports sequencing, forking, and merging. 
+
+--
+ ...and multithread execution of tasks.
+ Add this when multithreading works properly.
+++
+
+  # using the same tasks as above...
+  
+  app.sequence(t1, t2, t3)
+  app.run(t1, 0)
+
+  # the log output is like this:
+  #
+  # I[10:38:06]       sample/task recieved 0
+  # I[10:38:06]       task_two recieved 1
+  # I[10:38:06]       task_three recieved 2
+  # I[10:38:06]       sample/task recieved 3
+
+Workflows can be constructed as tasks themselves, and joined with other tasks and 
+workflows in the same fashion. 
+
+== Running Tap from the Command Line
+
+The tap command is a gateway for the execution of other subcommands.  To
+get general help for tap, type:
+
+  % tap --help
+  
+The tap command does only a few things.  It's main responsibility is configuring
+Tap::App.instance with 'tap.yml' from the present working directory, if it exists,
+running any before/after scripts specified therein, and passing control to the
+specifed subcommand.  Subcommand help can be obtained using:
+
+  % tap [subcommand] --help
+
+=== run
+
+The run subcommand configures, enqueues, and executes tasks.  Run has a rich
+syntax allowing the full specification of any number of tasks which simplifies
+under most circumstances.  Several examples illustrate the key points:
+
+  % tap run sample/task
+  % tap run -- sample/task
+  
+Both of these commands are equivalent but the second statement is a more
+formally correct syntax.  The option break '--' signifies the start of a task (or
+equivalently a break in processing statements) and is essential to pass options 
+to the task:
+
+  % tap run --debug -- sample/task --key=value
+  
+Here run recieves the '--debug' option and sample/task recieves the 
+'--key=value' option'.  Inputs work the same way.  By default task options 
+are used to set configurations, and inputs enqued to the task.  For example:
+
+  % tap run -- sample/task --key=value one -- another/task two three
+  
+Specifies the following:
+
+  t1 = app.task('sample/task', :key => 'value')
+  app.queue.enq(t1, 'one')
+  
+  t2 = app.task('another/task')
+  app.queue.enq(t2, 'two', 'three') 
+  
+Any number of tasks, configurations, and inputs may be specified in this
+way.  As a special note, rake tasks can be specifed as well.  If the app cannot
+find a tap task by the specified name, rake is loaded (using the same loading 
+rules as the rake command) and run tries to find a corresponding rake task.
+ENV options can be specified in the rake statement.
+
+  % tap run -- sample/task --key=value one -- rake_task KEY=value
+  
+Non-string inputs can be provided as well.  If the input begins with "---\n"
+then it will be loaded as YAML into an object before being passed to a task.
+The syntax can be a lot to type, but is very handy to have around especially
+when tap is lauched automatically or from some other program.  The following  
+enques 'sample/task' with a hash input, {'number' => 1}.
+
+  % tap -- sample/task "---
+  > number: 1"
+
+On Windows, you need to pull some tricks to get newlines into your argument.
+Cleverly use a caret to ignore the next line feed:
+
+  % tap -- sample/task '---^
+  More?
+  More? key: value'
+  
+Notice that hitting enter every other line is what actually puts the "\n" into the 
+parameter.  Keep using carets to enter more lines. The syntax on Windows isn't
+exactly pretty, but it works.  
+
+=== generate/destroy
+
+Generate and destory are subcommands launching generators.  They use the
+same back-end code as Rails generators, and are used identically.  By default
+Tap provides generators for:
+
+- root (creating the basic Tap directory structure)
+- task
+- file_task
+- workflow
+- config
+- script
+- package
+
+--
+Consult the help for each of these generators:
+
+  % tap [generate/destroy] [generator] --help
+  
+++
+
+=== console
+
+Console opens an irb session with Tap loaded.  Here you can directly interact
+with your code.  Console defines a variable 'app' referencing Tap::App.instance,
+for easy access.
+
+  % tap console
+  irb(main):001:0>  app.log(:hello)
+    I[17:18:53]              hello
+  => true
+  irb(main):002:0>
+ 
+For actively testing your code, remember that many of your files will be loaded 
+using Dependencies (from ActiveSupport).  You can modify your code, clear the 
+dependencies, and observe the changes within a single session.  The irb 
+command is:
+
+  irb(main):001:0>Dependencies.clear