tap 0.8.0 → 0.9.0

data/Tutorial

@@ -1,235 +1,287 @@
-= Tutorial
+= Quick Start Tutorial
 
-<b>Under Construction!</b>
+This tutorial demonstrates the basic use of tap from the command line, how to:
 
-== Overview
+* generate and run a task
+* generate and run a command
 
-=== Tasks
+=== Task Basics
 
-# insert task image
+Begin by creating a tap root directory structure:
 
-Tap tasks are designed to be modular: they can be configured, they take inputs, and they 
-produce results.  On a programatic level this system is convenient for joining tasks into
-workflows using condition and decision blocks.  On an application level, the task structure
-makes it easy to incorporate tasks into configurable, user-ready scripts.
+  % tap generate root /path/to/root_dir
+  % cd /path/to/root_dir
+  
+Make a task:
+
+  % tap generate task sample/task
 
-A task is structured like this:
+Test the task (in fact using the rake test task):
+ 
+  % tap run test 
+  
+Get help for the task:
+
+  % tap run -- sample/task --help
+  Sample::Task
+
+  Replace with a description.  The default task simply 
+  demonstrates the use of a config and logging.  
+  Usage:
+  Replace with usage instructions
+
+  Options:
+    --key                               a sample config ('value')
+    --help                              Print this help.
+    --debug                             Trace execution and debug
+    --use                               Loads inputs from file.
+    --iterate                           Iterates over inputs.
+
+Run the task in the most basic way:
+
+  % tap run sample/task input_one
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[21:32:28]        sample/task input_one was processed with value
+    
+Run the task, setting the 'key' configuration and iterating over several inputs:
   
+  % tap run -- sample/task --key='the new value' --iterate one two three
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[21:33:31]        sample/task one was processed with the new value
+    I[21:33:31]        sample/task two was processed with the new value
+    I[21:33:31]        sample/task three was processed with the new value
+    
+Run multiple tasks, or in this case the same task twice, followed by a dump task.  The tasks are separated into 'rounds' of execution, where the rounds are indicated by '--' and '--+' respectively.   The sample tasks in the first round run to completion before tap runs the dump task.  Normally the dump task saves results to a file you specify, but with no specified file (as below) the dump is to the console:
+
+  % tap run -- sample/task one -- sample/task two --+ tap/dump 
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[21:50:40]        sample/task one was processed with value
+    I[21:50:40]        sample/task two was processed with value
+  # audit:
+  # o-[] "two"
+  # o-[sample/task] "two was processed with value"
+  # 
+  # o-[] "one"
+  # o-[sample/task] "one was processed with value"
+  # 
+  # date: 2008-04-08 21:50:40
+  --- 
+  sample/task (8603460): 
+  - one was processed with value
+  sample/task (8602320): 
+  - two was processed with value
+
+Don't be thrown by the shuffled order in the dump -- the dump is valid YAML and represents the results hashed by task.
+
+Now navigate to and open the 'lib/sample/task.rb' file.  Inside you can see the class definition, with comments indicating how to define additional configurations, add command line documentation, log activity, and define the process method.  Let's change it up a bit:
+
+  [lib/sample/task.rb]
   module Sample
-    # == Documentation
-    # This documentation is available in RDoc, as well as from the
-    # command line, if you were to run it through tap, like this:
-    #   % tap run -- sample/task --help
-    #
+    # == Description
+    # A simple math task that multiplies an input by a 
+    # factor. Remember, strings can be multipled too!
     # === Usage
-    # By default, usage info comes up as well...
+    #   % tap run -- sample/task INPUT
     #
     class Task < Tap::Task
-
-      # Configurations are defined like this.  In this format, 
-      # this comment is used in generating customized RDoc
-      # documenting the configurations and any accessors created
-      # for them.  The default 'config' creates a config reader
-      # and writer.
-      config :config_name, "default value"      #-- a command line comment
+  
+      # The sum of the inputs is multiplied by this
+      # factor.  String factors are converted to 
+      # an integer.
+      config(:factor, 2) do |value|    # a factor
+        value.to_i
+      end
     
-      # Process receives each input from the command line
-      # and can do arbitrary work to make an output.
       def process(input)
-        # ... log a message and return the input plus one ...
-        log "#{self.name} recieved #{input}"
-        input + 1
+        result = input * factor
+        log self.name, result
+    
+        result
       end
+    
     end
   end
 
-Once instantiated, tasks can be joined into a workflow by the Tap application, which is
-made to work with a standard directory structure.  The basics looks like this:
+The new configurations and documentation are immediately available:
 
-  tap
-  |- Rakefile
-  |- ReadMe.txt
-  |- config
-  |   `- sample
-  |       `- task.yml             # a default config file for Sample::Task
-  |- lib
-  |   `- sample
-  |       `- task.rb              # a class file like above
-  |- tap.yml                      # a configuration file for Tap::App
-  `- test
-      |- sample
-      |   `- task_test.rb         # a test file for Sample::Task
-      |- tap_test_helper.rb
-      `- tap_test_suite.rb
+  % tap run -- sample/task --help
+  Sample::Task
 
-All task instances have a name, which by default serves as the relative filepath from an 
-application directory to assoicated files.  So for instance when the application instantiates 
-a task by the name 'sample/task' it sets the task configs from 'config/sample/task.yml'.
-Any number of configurations can exist for a single task class.  In addition, Tap knows 
-how to load unknown task classes by name or via mapping...
+  A simple math task that multiplies an input by a 
+  factor. Remember, strings can be multipled too!
+  Usage:
+    % tap run -- sample/task INPUT
 
-  app = Tap:App.instance
-  app.root = '/path/to/tap'
-  
-  # set up a mapping between a name and a task class
-  app.map("two" => Sample::Task)  
+  Options:
+    --factor                            a factor (2)
+    --help                              Print this help.
+    --debug                             Trace execution and debug
+    --use                               Loads inputs from file.
+    --iterate                           Iterates over inputs.
 
-  # the task method looks up and instantiates the proper
-  # task, with the input name and configuration (if the
-  # config file exists)
-  
-  t1 = app.task("sample/task")               # configured from 'config/sample/task.yml'
-  t2 = app.task("task_two")                    # configured from 'config/task_two.yml'
-  
-  # the idea is the same if you manually make a task... 
-  # t3 is configured from 'config/task_three.yml'
-  
-  t3 = Tap::Task.new("task_three") do |task, input| 
-    log "#{task.name} recieved #{input}"
-    input + 1
-  end 
-  
-  t1.class     # => Sample::Task
-  t2.class     # => Sample::Task
-  t3.class     # => Tap::Task
+And the task is ready to go:
 
-=== Workflows
+  % tap run -- sample/task "yo"
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[22:22:13]        sample/task yo yo
+    
+Make a configuration file for the task:
 
-# insert workflow image
+  % tap generate config sample/task
+        
+You get documentation in the configuration file as well.  We can set configurations here and have them be used by the task:
 
-Once tasks are instantiated, they can be joined into workflows, and run with inputs.
-Tasks are joined using a condition block and a decision (on_complete) block that runs 
-when the task finishes.  Arbitrary workflow logic can be used to join tasks in this
-way, but by default Tap supports sequencing, forking, and merging. 
+  [config/sample/task.yml]
+  # Sample::Task configuration
 
---
- ...and multithread execution of tasks.
- Add this when multithreading works properly.
-++
+  # The sum of the inputs is multiplied by this
+  # factor.  String factors are converted to
+  # an integer.
+  # a factor (2)
+  factor: 3
 
-  # using the same tasks as above...
-  
-  app.sequence(t1, t2, t3)
-  app.run(t1, 0)
+As can be seen here:
 
-  # the log output is like this:
-  #
-  # I[10:38:06]       sample/task recieved 0
-  # I[10:38:06]       task_two recieved 1
-  # I[10:38:06]       task_three recieved 2
-  # I[10:38:06]       sample/task recieved 3
+  % tap run -- sample/task yo
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[22:26:41]        sample/task yo yo yo
+
+If you need to batch run the same task with the same inputs, but multiple sets of configurations, simply define an array of configurations in the config file:
+  
+  [config/sample/task.yml]
+  # Sample::Task configuration
+  - factor: 1
+  - factor: 2
+  - factor: 3
 
-Workflows can be constructed as tasks themselves, and joined with other tasks and 
-workflows in the same fashion. 
+  % tap run -- sample/task yo
+  ctl-i prints information
+  ctl-c interupts execution
+  beginning run...
+    I[22:37:46]        sample/task yo
+    I[22:37:46]        sample/task yo yo
+    I[22:37:46]        sample/task yo yo yo
 
-== Running Tap from the Command Line
+=== Command Basics
 
-The tap command is a gateway for the execution of other subcommands.  To
-get general help for tap, type:
+Now say you want to do something fancier than the basic execution provided by the run command.  First, generate a new command:
 
-  % tap --help
+  % tap generate command echo
   
-The tap command does only a few things.  It's main responsibility is configuring
-Tap::App.instance with 'tap.yml' from the present working directory, if it exists,
-running any before/after scripts specified therein, and passing control to the
-specifed subcommand.  Subcommand help can be obtained using:
+The new command is now available from tap:
 
-  % tap [subcommand] --help
+  % tap echo --help
+  tap echo {options} ARGS...
 
-=== run
+  The default command simply prints the input arguments
+  and application information, then exits.
 
-The run subcommand configures, enqueues, and executes tasks.  Run has a rich
-syntax allowing the full specification of any number of tasks which simplifies
-under most circumstances.  Several examples illustrate the key points:
+  Options:
+    --help                    (-h)      Print this help.
+    --debug                             Specifies debug mode.
+    
+  % tap echo one two three
+  Received: one, two, three
+  state: 0 (READY) queue: 0 thread_queue: 0 threads: 0 results: 0
 
-  % tap run sample/task
-  % tap run -- sample/task
-  
-Both of these commands are equivalent but the second statement is a more
-formally correct syntax.  The option break '--' signifies the start of a task (or
-equivalently a break in processing statements) and is essential to pass options 
-to the task:
+The command script itself is very general and can be run through ruby directly.  The tap command basically just sets up the environment using tap.yml and then loads the command script, hence you can do whatever you like within the command script.  Lets open it up and change a few things:
 
-  % tap run --debug -- sample/task --key=value
+  [cmd/echo.rb]
+  # = Usage
+  # tap echo {options} ARGS...
+  #
+  # = Description
+  # Formats the inputs, then echoes them to the 
+  # 'echo/output.txt' file and to the console.
+  #
   
-Here run recieves the '--debug' option and sample/task recieves the 
-'--key=value' option'.  Inputs work the same way.  By default task options 
-are used to set configurations, and inputs enqued to the task.  For example:
-
-  % tap run -- sample/task --key=value one -- another/task two three
+  require 'tap'
+  require 'tap/script'
   
-Specifies the following:
-
-  t1 = app.task('sample/task', :key => 'value')
-  app.queue.enq(t1, 'one')
+  app = Tap::App.instance   
   
-  t2 = app.task('another/task')
-  app.queue.enq(t2, 'two', 'three') 
+  #
+  # handle options
+  #
   
-Any number of tasks, configurations, and inputs may be specified in this
-way.  As a special note, rake tasks can be specifed as well.  If the app cannot
-find a tap task by the specified name, rake is loaded (using the same loading 
-rules as the rake command) and run tries to find a corresponding rake task.
-ENV options can be specified in the rake statement.
-
-  % tap run -- sample/task --key=value one -- rake_task KEY=value
+  opts = [
+    ['--help', '-h', GetoptLong::NO_ARGUMENT, "Print this help."],
+    ['--debug', nil, GetoptLong::NO_ARGUMENT, "Specifies debug mode."]]
+    
+  Tap::Script.handle_options(*opts) do |opt, value| 
+    case opt
+    when '--help'
+      puts Tap::Script.usage(__FILE__, "Usage", "Description", "Information", :keep_headers => false)
+      puts
+      puts "Options:"
+      puts Tap::Script.usage_options(opts)
+      exit
+      
+    when '--debug'
+      app.options.debug = true
   
-Non-string inputs can be provided as well.  If the input begins with "---\n"
-then it will be loaded as YAML into an object before being passed to a task.
-The syntax can be a lot to type, but is very handy to have around especially
-when tap is lauched automatically or from some other program.  The following  
-enques 'sample/task' with a hash input, {'number' => 1}.
-
-  % tap -- sample/task "---
-  > number: 1"
-
-On Windows, you need to pull some tricks to get newlines into your argument.
-Cleverly use a caret to ignore the next line feed:
-
-  % tap -- sample/task '---^
-  More?
-  More? key: value'
+    end
+  end
+  
+  #
+  # initialize some tasks
+  #
+  
+  # transform the inputs
+  t1 = Tap::Task.new do |task, *inputs| 
+    inputs.collect {|input| "echo #{input}" }.join("\n")
+  end
+  
+  # write to a file
+  t2 = Tap::FileTask.new do |task, input|
+    target = "echo/output.txt"
+    
+    # backup existing files, and ensure the directory exists
+    task.prepare(target)  
+    
+    File.open(target, "wb") {|f| f << input }
+    target
+  end
+  
+  # now make a simple workflow
+  app.sequence(t1, t2)
+  
+  #
+  # enque, run, and print out the results
+  #
+  
+  t1.enq(*ARGV)
+  app.run
+  
+  app.results(t2).each do |target|
+    puts "[#{target}]"
+    puts File.read(target)
+  end
+  
+Now run the command a few times:
+
+  % tap echo one two three
+  [echo/output.txt]
+  echo one
+  echo two
+  echo three
+
+  % tap echo tip top tap
+  [echo/output.txt]
+  echo tip
+  echo top
+  echo tap
+  
+You should find that the 'echo/output.txt' file exists with the 'tip top tap' result.  Moreover a backup directory will exist with the 'one two three' result, due to the FileTask#prepare method.  
   
-Notice that hitting enter every other line is what actually puts the "\n" into the 
-parameter.  Keep using carets to enter more lines. The syntax on Windows isn't
-exactly pretty, but it works.  
-
-=== generate/destroy
-
-Generate and destory are subcommands launching generators.  They use the
-same back-end code as Rails generators, and are used identically.  By default
-Tap provides generators for:
-
-- root (creating the basic Tap directory structure)
-- task
-- file_task
-- workflow
-- config
-- script
-- package
-
---
-Consult the help for each of these generators:
-
-  % tap [generate/destroy] [generator] --help
   
-++
-
-=== console
-
-Console opens an irb session with Tap loaded.  Here you can directly interact
-with your code.  Console defines a variable 'app' referencing Tap::App.instance,
-for easy access.
-
-  % tap console
-  irb(main):001:0>  app.log(:hello)
-    I[17:18:53]              hello
-  => true
-  irb(main):002:0>
- 
-For actively testing your code, remember that many of your files will be loaded 
-using Dependencies (from ActiveSupport).  You can modify your code, clear the 
-dependencies, and observe the changes within a single session.  The irb 
-command is:
-
-  irb(main):001:0>Dependencies.clear