tap 0.19.0 → 1.3.0

data/doc/Examples/Command Line

@@ -1,62 +1,30 @@
-= Command Line Examples
+= Command Line
 
-== Basic Input/Output
+Tap can work in the context of a command line like any other executable. By
+default the load/dump tasks read and write to stdin/stdout. These patterns
+represent common ways to hook up a Tap workflow.
 
 === Read data from $stdin
   # [goodnight.txt]
   # goodnight moon
   
-  % tap run -- load --: dump < goodnight.txt
+  % tap load -: dump < goodnight.txt
   goodnight moon
 
 === Pipe data from $stdin
-  % echo goodnight moon | tap run -- load --: dump
+  % echo goodnight moon | tap load -: dump
   goodnight moon
 
-=== Load data from argument
-  % tap run -- load 'goodnight moon' --: dump
+=== Manually specify data with an argument
+  % tap load 'goodnight moon' -: dump
   goodnight moon
 
 === Dump data to $stdout
-  % tap run -- load 'goodnight moon' --: dump > goodnight.txt
+  % tap load 'goodnight moon' -: dump > goodnight.txt
   % more goodnight.txt
   goodnight moon
 
 === Pipe data via $stdout
-  % tap run -- load 'goodnight moon' --: dump | more
+  % tap load 'goodnight moon' -: dump | more
   goodnight moon
 
-== Setup
-
-=== Use an ENV variable to setup the Tap::Env
-
-  % TAP_GEMS= tap run -T
-  % TAP_GEMS=:all tap run -T
-  % TAP_GEMS=:latest tap run -T
-  % TAP_GEMS="[rap, tap-tasks]" tap run -T
-  
-== Signals
-
-The run command will bring up a signal prompt with the --prompt flag, or by
-sending and interrupt signal.  To illustrate the latter, setup an infinite
-loop and hit ctl-c to enter the prompt.  To exit, signal stop as shown here:
-
-  % tap run -- dump --[app][0]q --/0/enq 'hello world'
-  hello world
-  hello world
-  hello world
-  ... (this will continue to print until you press ctl-c) ...
-  
-  starting prompt (enter for help):
-  --/
-  => 
-    run       # run the app
-    stop      # stop the app
-    terminate # terminate the app
-    info      # prints app status
-    enque    
-    build    
-    use
-  --//info
-  => state: 1 (RUN) queue: 0
-  --//stop

data/doc/Examples/Tapfile

@@ -0,0 +1,124 @@
+= Tapfile
+
+Tapfiles can be used to define tasks and workflows. Typically using the
+Tap::Declarations module which provides a syntax reminiscent of
+{Rake}[http://rake.rubyforge.org/].
+
+  [tapfile]
+  # A task declaration looks like this. The declaration creates a subclass of
+  # Tap::Task, literally this:
+  #
+  #   class Goodnight < Tap::Task
+  #     config :msg, 'goodnight'
+  #     def process(thing)
+  #       "#{msg} #{thing}!"
+  #     end
+  #   end
+  #
+  # The 'config' passed to the block is actually the task instance but it's
+  # useful to think of it as a source of configs.  Note these comments are
+  # meaningful, if present they become the task help.
+
+  desc "a goodnight task"
+  task :goodnight, :msg => 'goodnight' do |config, thing|
+    "#{config.msg} #{thing}!"
+  end
+
+  # Use namespace to set tasks within a module.
+  namespace :example do
+    desc "concat files"
+    task :cat do |config, *paths|
+      paths.collect {|path| File.read(path) }.join
+    end
+
+    desc "sort a string by line"
+    task :sort, :reverse => false do |config, str|
+      lines = str.split("\n").sort
+      config.reverse ? lines.reverse : lines
+    end
+  end
+
+  # Use workflow to create a subclass of Tap::Workflow.  Access any tasks you
+  # define in the workflow using node and return the tasks you want to use as
+  # the entry/exit points for the workflow.
+  
+  desc "sort the lines of a file"
+  work :sort_file, %q{
+    -   cat
+    -:a sort
+    -:i dump
+  },{
+    :reverse => false
+  } do |config|
+    n0 = node(0)    # cat
+    n1 = node(1)    # sort
+    n1.reverse = config.reverse 
+    [n0, n1]
+  end
+  
+  # Tasks defined in a singleton block will only execute once (given a set
+  # of inputs) and can be used to make dependency-based workflows.
+  
+  singleton do
+    task(:a)             { puts 'a' }
+    task(:b => :a)       { puts 'b' }
+    task(:c => [:b, :a]) { puts 'c' }
+  end
+  
+Declaration tasks are available for use in workflows:
+
+  % tap goodnight moon -: dump
+  goodnight moon!
+  % tap goodnight world --msg hello -: dump
+  hello world!
+
+And documented:
+
+  % tap goodnight --help
+  Tapfile::Goodnight -- a goodnight task
+  --------------------------------------------------------------------------------
+    A task declaration looks like this. The declaration creates a subclass of
+    Tap::Task, literally this:
+    
+      class Goodnight < Tap::Task
+        config :msg, 'goodnight'
+        def process(thing)
+          "#{msg} #{thing}!"
+        end
+      end
+    
+    The 'config' passed to the block is actually the task instance but it's
+    useful to think of it as a source of configs.  Note these comments are
+    meaningful, if present they become the task help.
+  --------------------------------------------------------------------------------
+  usage: tap tapfile/goodnight arg
+  
+  configurations:
+          --msg MSG
+  
+  options:
+          --help                       Print this help
+          --config FILE                Specifies a config file
+
+Workflows work just like tasks:
+
+  % tap sort_file tapfile --reverse true
+  work :sort_file, :reverse => false do |config|
+  task :goodnight, :msg => 'goodnight' do |config, thing|
+  ...
+
+  % order c a b
+  a b c
+
+Singleton tasks can be used to create dependency-based workflows that function
+much like rake tasks in practice:
+
+  % tap c
+  a
+  b
+  c
+  % tap b -- a
+  a
+  b
+
+See Tap::Declarations for syntax details.

data/doc/Ruby to Ruby

@@ -0,0 +1,87 @@
+= Ruby (1.8.*) to Ruby (1.9.*)
+
+Tap runs on at least these interpreters without problem:
+
+* ruby-1.8.7
+* ruby-1.9.1
+* jruby-1.4.0
+
+However there are suble and important changes in ruby-1.9.* that affect how
+you construct workflows and what outputs you see on the command line. The
+most noticeable changes are related to arrays and strings. 
+
+== Array#to_s
+
+In 1.9 Array#to_s is more like Array#inspect in 1.8. There's no real harm in
+the to_s change, but it does change the output of dumps and in many cases
+makes dumps more clear:
+
+  # 1.8
+  % tap load string -: dump
+  string
+  
+  % tap load/yaml [array] -: dump
+  array
+
+  # 1.9
+  % tap load string -: dump
+  string
+  
+  % tap load/yaml [array] -: dump
+  ["array"]
+
+== String.to_a
+
+In 1.9 this method goes away, whereas in 1.8 to_a will split a string along
+newlines into an array. Seems inconsequential but it has big ramifications for
+objects that splat a string into a helper method. One example is Tap::Task
+itself, which splats the call input to process.
+
+  class Tap::Task
+    def call(inputs)
+      process(*inputs)
+    end
+    
+    def process(*inputs)
+      inputs
+    end
+  end
+
+Declaration tasks do the same thing, such that arguments to the block are
+populated by a splat. Say you made a couple tasks:
+
+  [tapfile]
+  require 'tap/declarations'
+  extend Tap::Declarations
+  
+  task(:one) {|config, a, b, c| "#{a}\n#{b}\n#{c}" }
+  task(:two) {|config, input| puts input }
+
+On 1.9 you can directly join one to two without problem:
+
+  # 1.9
+  % tap one 1 2 3 -: two
+  1
+  2
+  3
+
+Now see what happens on 1.8:
+
+  # 1.8
+  % tap one 1 2 3 -: two
+  1
+
+The string created by one has newlines so it gets splatted into an array
+before being passed into two. Task two only takes one argument and so all that
+it receives is the first argument of the new array (ie '1'). Totally lousy,
+but easy to fix. Just be sure to arrayify outputs if the target splats it's
+input.
+
+  # 1.8
+  % tap one 1 2 3 -:a two
+  1
+  2
+  3
+
+Note that arrayification is less necessary 1.9, it works fine either way, but
+cross-ruby workflows should consider this a best practice.

data/doc/Workflow Syntax

@@ -0,0 +1,185 @@
+= Workflow Syntax
+
+Tap uses a custom syntax for specifying workflows. The syntax consists of
+argument vectors separated by various breaks (ex '-', '--', '-:'). The
+argument vectors define tasks, joins, and other workflow objects, while the
+breaks signify various things to do with the objects.
+
+*Note* this documentation is written from the perspective of ruby-1.9.*, which
+changes the output of Array#to_s (among other things). See the {Ruby to
+Ruby}[link:files/doc/Ruby%20to%20Ruby.html] document for more details.
+
+== Basics
+
+Argument vectors start with a constant identifier and subsequent arguments
+indicate inputs and/or options.  Inputs are passed to the task as an array
+while options are parsed out as configurations.
+
+  % tap dump 'goodnight moon'
+  ["goodnight moon"]
+  
+  % tap dump --help
+  Tap::Tasks::Dump -- dump data
+  ...
+
+Multiple objects can be specified using delimiters where a single-dash (-)
+delimits, and a double-dash (--) enques. The first object is implicitly
+enqued, but that behavior can be changed by manually setting a delimiter.
+
+  % tap dump a - dump b -- dump c
+  ignoring args: ["b"]
+  ["a"]
+  ["c"]
+
+  % tap - dump a -- dump b - dump c
+  ignoring args: ["a"]
+  ignoring args: ["c"]
+  ["b"]
+
+Each object in a workflow is stored in the application by index, allowing them
+to be referenced from contexts that specifically need them, such as during
+joins or in signaling (note the context chooses when this happens; normally
+indexes like 0 are just another character).
+
+  % tap - dump -/0/enq 'enqued by a signal'
+  ["enqued by a signal"]
+
+Constants are identified by matching strings right-to-left as if the constant
+were underscored into a path. Alternatively, the full constant name can be
+provided. These all resolve to Tap::Tasks::Load.
+
+  % tap load
+  % tap tap/tasks/load
+  % tap /tap/tasks/load
+  % tap Tap::Tasks::Load
+
+In addition a left:right identifier can be specified to match both sides of
+the constant path:
+
+  % tap tap:load
+  % tap /tap/tasks:tasks/load
+  % tap /tap/tasks/load:
+  % tap :/tap/tasks/load
+
+The workflow syntax reserves dash and all dash-nonword pairs as breaks. Use
+the escape begin (-.) and escape end (.-) sequences to provide a breaks as
+inputs.
+
+  % tap dump begin -. -- - -- --- -: -/ --/ .- end
+  ["begin", "-", "--", "---", "-:", "-/", "--/", "end"]
+
+Not all possible breaks are used, but reserving them all makes for one simple
+rule.  The various breaks and common use cases are described below.
+
+== Sequence (-:)
+
+Objects can be linked into sequences using a -: break. Here load processes the
+input into a string, which gets passed to dump and printed (hence the output
+looks like a string, not an array).
+
+  % tap load 'goodnight moon' -: dump
+  goodnight moon
+
+The sequence break is a convenient shorthand for manually building two objects
+and a join.
+
+  % tap load 'goodnight moon' - dump - join 0 1
+  goodnight moon
+
+The longer syntax supports multi-way joins like fork and merge.
+
+  % tap load 'goodnight moon' - dump - dump - join 0 1,2
+  goodnight moon
+  goodnight moon
+  
+  % tap load goodnight -- load moon - dump - join 0,1 2
+  goodnight
+  moon
+
+It also allows for more complex join types. For example this is a gate join
+where the outputs are collected into a size-limited array before being passed
+along.
+
+  % tap load a -- load b -- load c - dump - gate 0,1,2 3 --limit 2
+  ["a", "b"]
+  ["c"]
+
+=== Variations
+
+Joins can iterate or arrayify outputs before passing them to join targets.
+These options are useful for changing the input/output signatures of objects
+across a join. For example (borrowing the yaml loading task from tap-tasks):
+
+  % tap load/yaml "[1, 2, 3]" - dump - join 0 1
+  [1, 2, 3]
+  
+  % tap load/yaml "[1, 2, 3]" - dump - join 0 1 --iterate
+  1
+  2
+  3
+
+Sequences can specify flags and a join class like '-:flags.class'. For
+example, these are equivalent workflows:
+
+  % tap load/yaml "[1, 2, 3]" - dump - dump - join 0 1 --iterate - gate 1 2
+  1
+  2
+  3
+  [1, 2, 3]
+  
+  % tap load/yaml "[1, 2, 3]" -:i dump -:.gate dump
+  1
+  2
+  3
+  [1, 2, 3]
+
+== Signals (-/, --/)
+
+Internally tap parses workflows into signals that build and control workflow
+objects. These signals can be specified manually on the command line.
+
+  % tap -/set 0 load -/set 1 dump -/bld join 0 1 -/enq 0 'goodnight moon'
+  goodnight moon
+
+Invoked from a prompt.
+
+  % tap prompt
+  /set 0 load
+  /set 1 dump
+  /bld join 0 1
+  /enq 0 'goodnight moon'
+  /run
+  goodnight moon
+
+Or specified in a taprc file, one or more of which can be loaded with a
+triple-dash. Complex workflows are normally written in this way when they
+become too cumbersome to write in the single-line syntax.
+
+  [taprc]
+  set 0 load
+  set 1 dump
+  bld join 0 1
+  enq 0 'goodnight moon'
+
+  % tap --- taprc
+  goodnight moon
+
+  % tap --- taprc taprc taprc
+  goodnight moon
+  goodnight moon
+  goodnight moon
+
+Signals are the basis for controlling apps and objects. As mentioned before,
+objects in the application space can be signaled directly by name.
+
+  % tap - dump -/0/exe 'goodnight moon'
+  ["goodnight moon"]
+
+Note that the dash-slash break (-/) enques signals, and as such they will run
+in order, as declared in the workflow. To execute a signal immediately, use a
+double-dash-slash (--/).
+
+  % tap -- dump a -/0/exe b --/0/exe c
+  ["c"]
+  ["a"]
+  ["b"]