tap 0.18.0 → 0.19.0

data/doc/API

@@ -1,83 +1,223 @@
 = Application Programming Interface
 
-(Note these are all currently provisional.)
-
-== Tap::App
-
-Applications require the following API for nodes, joins, and middleware. Tap
-provides modules or base classes that implement these APIs and may be used as
-the foundation for subclasses.
-
-==== Node
-
-  call(*inputs)    any return is allowed
-  joins()          returns an array of joins
-
-The signature for call can be modified as necessary.
+Applications require the following methods for nodes, joins, and middleware.
+Tap provides modules and base classes that implement these APIs.
+
+=== {Node}[link:classes/Tap/App/Node.html] ({Task}[link:classes/Tap/Task.html])
+
+  call(*inputs)    # any return is allowed
+  joins()          # returns an array of joins, or nil (optional)
+
+The signature for call defines the arguments that must be enqued to the node
+or passed to the node via a join. All signature constructs are allowed
+including multiple arguments, default arguments and splats (blocks are
+technically allowed but will never receive a value).
+
+The optional joins method specifies an array of joins to be called by a
+running application when the node completes. Each join is called in order. An
+empty array specifies the default application joins should be called; nil
+specifies that no joins should be called. No joins will be called when joins
+is left undefined.
+
+=== {Join}[link:classes/Tap/Join.html]
+
+  call(result)     # any return is allowed
+
+The call method receives the result of input nodes. The result of call is not
+used; call must internally performing the join actions.
+
+=== {Middleware}[link:classes/Tap/Middleware.html]
+
+  Middleware.new(stack, *args)   # returns an instance of middleware
+  call(node, inputs=[])          # any return is allowed
+  stack()                        # returns the original stack
+
+Middleware wraps the execution of nodes. Nodes and inputs are passed to the
+middleware during execution; the middleware is responsible for processing the
+node or passing it into the stack using the same call API. By default the base
+stack invokes call on the node with the inputs. Joins are performed after the
+middleware returns.
+
+== Application Interface
+
+In addition to the APIs for individual workflow objects, Tap defines an
+application interface allowing objects to be created and modified using
+signals sent to an application (think HTTP to a web app). Signals are simple
+hash constructs and typically form the basis for user interfaces. The
+application interface is distinct from the object APIs, although it is typical
+to implement the application interface on top of an object API.
+
+Tap::App::Api implements the application interface in a general way and is the
+baseclass for Tap::Task, Tap::Join, and Tap::Middleware. The basic idea is to
+allow instantiation of a class from a hash and serialization of an instance
+back into a hash. The hash is referred to as a specification and must be
+serializable as {JSON}[http://json.org/], basically meaning the hash must
+consist of simple object types: numbers, strings, hashes, and arrays.
+
+The application interface consists of two methods, build and to_spec:
+
+  class Stub
+    class << self
+      # Build takes a specification hash and returns an instance of self.
+      # The spec must be serializable as JSON.
+      def build(spec={}, app=Tap::App.instance)
+      end
+    end
+    
+    # Takes no inputs and returns a specification hash that, when built,
+    # returns an object like self.
+    #
+    #   obj.class.build(obj.to_spec)      # => returns an object like obj
+    #
+    # Users can determine for themselves what constitutes 'likeness'.
+    def to_spec
+    end
+  end
 
-==== Join
+The application API reserves several additional methods that do not need to be
+implemented but add functionality for specific, common use cases. If they are
+present they must adhere to these specifications.
+
+  # Optional methods #
+  class Stub
+    class << self
+      # Parse takes an argument vector (an array, usually from the command
+      # line) and returns an instance of self and any remaining arguments
+      # in an array like [instance, args].  The remaining arguments may be
+      # nil.  Parse cannot modify argv.
+      def parse(argv=ARGV, app=Tap::App.instance)
+      end
+      
+      # Same as parse, but able to modify argv.
+      def parse!(argv=ARGV, app=Tap::App.instance)
+      end
+    end
+    
+    # Takes a signal name and returns an object that responds to call; the
+    # call method invokes the signal actions.
+    def signal(sig)
+    end
+    
+    # Returns a nested array of workflow objects associated with self (ex 
+    # input/output nodes for a join).  The array should be structured like
+    # [refs, brefs], where refs are references to objects that must be built
+    # BEFORE self and brefs are back-references to objects that must be built
+    # AFTER self.
+    #
+    # For example, nodes must be built before joins.  As such, the associations
+    # method for a node returns a brefs for each of its joins. Similarly, joins
+    # must be built after nodes and hence the associations method for a join
+    # returns refs to their input and output nodes:
+    #
+    #   node.associations       # => [nil, join]
+    #   join.associations       # => [inputs + outputs, nil]
+    #
+    # Nil is a valid return for associations, indicating no associations.
+    def associations
+    end
+  end
 
-  call(result)     any return is allowed
+The parse methods are used for building objects from interfaces that provide
+an array of inputs (ex the command line) rather a hash; without them objects
+are effectively excluded from use within these interfaces.
 
-==== Middleware
+Signals can be used to interact with specific objects from a user interface
+much as signals can interact with an app. Objects without a signal method
+cannot receive signals.
 
-  Middleware.new(stack)   returns an instance of middleware
-  call(node, inputs=[])   any return is allowed
-  stack()                 returns the original stack
+The associations method is used to order complex builds and is described in
+more detail below.
 
-The middleware API is essentially the same as for {Rack}[http://rack.rubyforge.org/].
+=== Spec References
 
-== Tap::Schema
+Specifications often require references to other resources, as when a join
+refers to input and output nodes. These references are normally specified as
+variables that, unlike the resource itself, are easily serializable as JSON
+and may be used in multiple places. Apps are constructed to do this easily via
+the +obj+ and +var+ methods.
 
-Schema describe workflows as data. To build a workflow from a schema, workflow
-resources like nodes, joins, and middleware need to instantiate themselves
-using the schema data. The <tt>parse!</tt> and <tt>instantiate</tt> methods
-must be provided to do so.
+As an example, consider the Sample class that references some other
+application object:
 
-  Resource.parse!(argv=ARGV, app=App.instance)
-  Resource.instantiate(argh, app=App.instance)
+  class Sample
+    def initialize(object)
+      @object = object
+    end
+
+    def to_spec
+      {'key' => app.var(@object)}        # store a variable into the spec
+    end
+
+    def associations
+      [[@object], nil]                   # establish a build order
+    end
+    
+    class << self
+      def build(spec={}, app=Tap::App.instance)
+        object = app.obj(spec['key'])    # retrieve an object referenced by the spec
+        new(object)
+      end
+    end
+  end
 
-As implied in by the inputs, <tt>parse!</tt> instantiates from an array, while
-<tt>instantiate</tt> instantiates from a hash with symbol keys. If
-<tt>parse!</tt> receives a string, it must be able to convert it to an array
-(ex using Shellwords).
+Using this technique the spec will have a serializable variable representing
+the object and the app will be able to properly schematize and rebuild the
+instance and all its references. Apps use the associations array to determine
+the correct build order for the references. In the example the @object
+reference must be built before the Sample instance and correspondingly the
+associations method returns @object in the 'ref' array.
 
-How the class actually performs the instantiation is up to the class but
-typically parse creates a hash and calls instantiate.
+Note that only references to objects implementing the application interface
+may be stored this way; references to objects that do not implement the
+application interface must be serialized and deserialized by the build/to_spec
+methods internally.
 
-== Tap::Env
+=== Resource Identifiers
 
-Envs identify resources by resource identifiers (ie constant attributes
-recognized by Lazydoc). This identifies 'Sample' as an 'example' resource.
+Tap discovers application resources using resource identifiers (ie constant
+attributes, see {Lazydoc}[http://tap.rubyforge.org/lazydoc]). Resources
+identified in this way can be automatically loaded by the Tap::Env. If no
+identifiers are specified for a resource, the user must manually load the
+resource files.
 
-  [file.rb]
+As an example, this identifies the Sample constant as an 'example' resource.
 
+  [lib/file.rb]
+  
   # Sample::example summary
-  # description
   class Sample
   end
 
-The constant name will be inferred from the path for the file containing the
-resource identifier if no constant name is specified.
+A resource can be identified by zero or more identifiers. Typically all
+identifiers will be put in the same file as the class, but this does not have
+to be the case; applications automatically require all files that identify a
+resource. The order in which the files are required is indeterminate and it is
+up to the user to ensure consistency. For example:
 
-  [sample.rb]
+  [a.rb]
+  # Sample::a
+  class Sample
+  end
   
-  # ::example summary
-  # description
+  [b.rb]
+  require 'a'
+
+  # Sample::b
   class Sample
   end
 
-Resources may be accessed using Tap::Env#manifest. Resources intended to be
-discovered by Env must map the identifier documentation to the class 'desc'
-method. The easiest way to do so is with a
-{lazy_attr}[link:lazydoc/classes/Lazydoc/Attributes.html].
+Here the require statement ensures a.rb is always required before b.rb. Note
+that consistency is automatic when all identifiers are in the same file (and
+thus only one file is required).
+
+The constant name will be inferred from file path if no constant name is
+specified. This is the most compact form for identifying a resource:
 
+  [lib/sample.rb]
+  
   # ::example summary
-  # description
   class Sample
-    lazy_attr :desc, 'example'
   end
-  
-  Sample::desc.summary         # => "summary"
-  Sample::desc.to_s            # => "description"
-  
+
+In this case no constant name is specified, so 'Sample' is inferred from
+'sample.rb'. Constant names are determined from the path using camelization.

data/doc/Examples/Command Line

@@ -33,4 +33,30 @@
   % TAP_GEMS= tap run -T
   % TAP_GEMS=:all tap run -T
   % TAP_GEMS=:latest tap run -T
-  % TAP_GEMS="[rap, tap-tasks]" tap run -T
+  % TAP_GEMS="[rap, tap-tasks]" tap run -T
+  
+== Signals
+
+The run command will bring up a signal prompt with the --prompt flag, or by
+sending and interrupt signal.  To illustrate the latter, setup an infinite
+loop and hit ctl-c to enter the prompt.  To exit, signal stop as shown here:
+
+  % tap run -- dump --[app][0]q --/0/enq 'hello world'
+  hello world
+  hello world
+  hello world
+  ... (this will continue to print until you press ctl-c) ...
+  
+  starting prompt (enter for help):
+  --/
+  => 
+    run       # run the app
+    stop      # stop the app
+    terminate # terminate the app
+    info      # prints app status
+    enque    
+    build    
+    use
+  --//info
+  => state: 1 (RUN) queue: 0
+  --//stop

data/lib/tap.rb

@@ -2,4 +2,5 @@ lib = File.expand_path(File.dirname(__FILE__))
 $:.unshift(lib) unless $:.include?(lib)
 
 require 'tap/version'
-require 'tap/exe'
+require 'tap/app'
+require 'tap/task'

data/lib/tap/app.rb

@@ -1,98 +1,17 @@
 require 'logger'
-require 'configurable'
+require 'tap/app/api'
 require 'tap/app/node'
 require 'tap/app/state'
 require 'tap/app/stack'
 require 'tap/app/queue'
+require 'tap/env'
+require 'tap/parser'
 
 module Tap
   
-  # App coordinates the setup and execution of workflows.
-  #
-  # === Workflows
-  #
-  # Workflows are composed of nodes and joins such as instances of Tap::Task
-  # and Tap::Join.  The actual workflow exists between nodes; each node can
-  # specify a join to receive it's output and enque or execute other nodes.
-  # When a node does not have a join, apps allow the specification of a
-  # default join to, for instance, aggregate results.
-  #
-  # Any object satisfying the correct API[link:files/doc/API.html] can be used
-  # as a node or join.  Apps have helpers to make nodes out of blocks.
-  #
-  #   app = Tap::App.new
-  #   n = app.node {|*inputs| inputs }
-  #   app.enq(n, 'a', 'b', 'c')
-  #   app.enq(n, 1)
-  #   app.enq(n, 2)
-  #   app.enq(n, 3)
-  #
-  #   results = []
-  #   app.on_complete {|result| results << result }
-  #
-  #   app.run
-  #   results                        # => [['a', 'b', 'c'], [1], [2], [3]]
-  #
-  # To construct a workflow, set joins for individual nodes.  Here is a simple
-  # sequence:
-  #
-  #   n0 = app.node { "a" }
-  #   n1 = app.node {|input| "#{input}.b" }
-  #   n2 = app.node {|input| "#{input}.c"}
-  #
-  #   n0.on_complete {|result| app.execute(n1, result) }
-  #   n1.on_complete {|result| app.execute(n2, result) }
-  #   app.enq(n0)
-  #
-  #   results.clear
-  #   app.run
-  #   results                        # => ["a.b.c"]
-  #
-  # Tasks have helpers to simplify the manual constructon of workflows, but
-  # even with these methods large workflows are cumbersome to build.  More
-  # typically, a Tap::Schema is used in such cases.
-  #
-  # === Middleware
-  #
-  # Apps allow middleware to wrap the execution of each node.  This can be
-  # particularly useful to track the progress of a workflow.  Middleware is
-  # initialized with the application stack and uses the call method to
-  # wrap the execution of the stack.
-  #
-  # Using middleware, an auditor looks like this:
-  #
-  #   class AuditMiddleware
-  #     attr_reader :stack, :audit
-  #
-  #     def initialize(stack)
-  #       @stack = stack
-  #       @audit = []
-  #     end
-  #
-  #     def call(node, inputs=[])
-  #       audit << node
-  #       stack.call(node, inputs)
-  #     end
-  #   end
-  #
-  #   auditor = app.use AuditMiddleware
-  #
-  #   app.enq(n0)
-  #   app.enq(n2, "x")
-  #   app.enq(n1, "y")
-  #
-  #   results.clear
-  #   app.run
-  #   results                        # => ["a.b.c", "x.c", "y.b.c"]
-  #   auditor.audit                  
-  #   # => [
-  #   # n0, n1, n2, 
-  #   # n2,
-  #   # n1, n2
-  #   # ]
-  # 
-  # Middleware can be nested with multiple calls to use.
+  # :startdoc::app
   #
+  # App coordinates the setup and execution of workflows.
   class App
     class << self
       # Sets the current app instance
@@ -100,18 +19,54 @@ module Tap
       
       # Returns the current instance of App.  If no instance has been set,
       # then instance initializes a new App with the default configuration.
-      #
-      # Instance is used to initialize tasks when no app is specified.  Aside
-      # from that, there is nothing magical about instance.
+      # Instance is used to initialize tasks when no app is specified and
+      # exists for convenience only.
       def instance(auto_initialize=true)
         @instance ||= (auto_initialize ? new : nil)
       end
+      
+      # Sets up and returns App.instance with an Env setup to the specified
+      # directory.  This method is used to initialize the app and env as seen
+      # by the tap executable.
+      def setup(dir=Dir.pwd)
+        env = Env.setup(dir)
+        @instance = new(:env => env)
+      end
+      
+      def build(spec={}, app=nil)
+        config = spec['config'] || {}
+        signals = spec['signals'] || []
+        
+        if spec['self']
+          app.reconfigure(config)
+        else
+          app = new(config)
+        end
+        
+        signals.each do |args|
+          app.call(args)
+        end
+        
+        app.gc
+        app
+      end
     end
     
     include Configurable
     include MonitorMixin
+    include Signals
+    include Node
+    
+    # The reserved call keys
+    CALL_KEYS = %w{obj sig args}
+    
+    # The reserved init keys
+    INIT_KEYS = %w{var class spec}
     
-    # The default App logger writes to $stderr at level INFO.
+    # Reserved call and init keys as a single array
+    RESERVED_KEYS = CALL_KEYS + INIT_KEYS
+    
+    # The default App logger (writes to $stderr at level INFO)
     DEFAULT_LOGGER = Logger.new($stderr)
     DEFAULT_LOGGER.level = Logger::INFO
     DEFAULT_LOGGER.formatter = lambda do |severity, time, progname, msg|
@@ -127,54 +82,143 @@ module Tap
     # The application queue
     attr_reader :queue
     
-    # A cache of application-specific data.
-    attr_reader :cache
-    
-    # The default joins for nodes that have no joins set
-    attr_accessor :default_joins
+    # A cache of application objects
+    attr_reader :objects
     
     # The application logger
-    attr_reader :logger
+    attr_accessor :logger
+    
+    config :debug, false, :short => :d, &c.flag      # Flag debugging
+    config :force, false, :short => :f, &c.flag      # Force execution at checkpoints
+    config :quiet, false, :short => :q, &c.flag      # Suppress logging
+    config :verbose, false, :short => :v, &c.flag    # Enables extra logging (overrides quiet)
+    config :auto_enque, true, &c.switch              # Auto-enque parsed args
+    config :bang, true, &c.switch                    # Use parse! when possible
     
-    config :debug, false, &c.flag                 # Flag debugging
-    config :force, false, &c.flag                 # Force execution at checkpoints
-    config :quiet, false, &c.flag                 # Suppress logging
-    config :verbose, false, &c.flag               # Enables extra logging (overrides quiet)
+    nest :env, Env,                                  # The application environment
+      :type => :hidden,
+      :writer => false, 
+      :init => false
     
-    # Creates a new App with the given configuration.  
+    signal_hash :set,                                # set or unset objects
+      :signature => ['var', 'class'],
+      :remainder => 'spec',
+      :bind => :build
+      
+    signal :get, :signature => ['var']               # get objects
+    
+    signal_class :list do                            # list available objects
+      def call(args) # :nodoc:
+        lines = obj.objects.collect {|(key, obj)|  "#{key}: #{obj.class}" }
+        lines.empty? ? "No objects yet..." : lines.sort.join("\n")
+      end
+    end
+    
+    signal :enque                                   # enques an object
+    
+    signal_class :parse do                          # parse a workflow
+      def call(args) # :nodoc:
+        argv = convert_to_array(args, ['args'])
+        obj.send(obj.bang ? :parse! : :parse, argv, &block)
+      end
+    end
+    
+    signal_class :use do                             # enables middleware
+      def call(args) # :nodoc:
+        spec = convert_to_hash(args, ['class'], 'spec')
+        obj.stack = obj.build(spec, &block)
+      end
+    end
+    
+    signal :run                                      # run the app
+    signal :stop                                     # stop the app
+    signal :terminate                                # terminate the app
+    signal :info                                     # prints app status
+    
+    signal_class :exit do                            # exit immediately
+      def process(args) # :nodoc:
+        exit(1)
+      end
+    end
+    
+    signal :help, :class => Help, :bind => nil       # signals help
+    
+    # Creates a new App with the given configuration.  Options can be used to
+    # specify objects that are normally initialized for every new app:
+    #
+    #   :stack      the application stack; an App::Stack
+    #   :queue      the application queue; an App::Queue
+    #   :objects    application objects; a hash of (var, object) pairs
+    #   :logger     the application logger
+    #
+    # A block may also be provided; it will be set as a default join.
     def initialize(config={}, options={}, &block)
       super() # monitor
       
       @state = State::READY
       @stack = options[:stack] || Stack.new(self)
       @queue = options[:queue] || Queue.new
-      @cache = options[:cache] || {}
-      @default_joins = []
+      @objects = options[:objects] || {}
+      @logger = options[:logger] || DEFAULT_LOGGER
+      @joins = []
       on_complete(&block)
       
+      self.env = config.delete(:env) || config.delete('env')
       initialize_config(config)
-      self.logger = options[:logger] || DEFAULT_LOGGER
     end
     
-    # True if debug or the global variable $DEBUG is true.
-    def debug?
-      debug || $DEBUG
+    # Sets the application environment and validates that env provides an AGET
+    # ([]) and invert method.  AGET is used to lookup constants during init;
+    # it receives the 'class' parameter and should return a corresponding
+    # class.  Invert should return an object that reverses the AGET lookup.
+    # Tap::Env and a regular Hash both satisfy this api.
+    #
+    # Env can be set to nil and is set to nil by default, but initialization
+    # is constrained without it.
+    def env=(env)
+      Validation.validate_api(env, [:[], :invert]) unless env.nil?
+      @env = env
     end
     
-    # Sets the current logger. The logger level is set to Logger::DEBUG if
-    # debug? is true.
-    def logger=(logger)
-      unless logger.nil?
-        logger.level = Logger::DEBUG if debug?
+    # Sets the application stack.
+    def stack=(stack)
+      synchronize do
+        @stack = stack
       end
-      
-      @logger = logger
     end
     
-    # Logs the action and message at the input level (default INFO).  
+    # True if the debug config or the global variable $DEBUG is true.
+    def debug?
+      debug || $DEBUG
+    end
+    
+    # Logs the action and message at the input level (default INFO).  The
+    # message may be generated by a block; in that case leave the message
+    # unspecified as nil.
+    #
     # Logging is suppressed if quiet is true.
-    def log(action, msg="", level=Logger::INFO)
-      logger.add(level, msg, action.to_s) if !quiet || verbose
+    #
+    # ==== Performance Considerations
+    #
+    # Using a block to generate a message is quicker if logging is off, but
+    # slower when logging is on.  However, when messages use a lot of
+    # interpolation the log time is dominated by the interpolation; at some
+    # point the penalty for using a block is outweighed by the benefit of
+    # being able to skip the interpolation.
+    #
+    # For example:
+    #
+    #   log(:action, "this is fast")
+    #   log(:action) { "and there's not much benefit to the block" }
+    #
+    #   log(:action, "but a message with #{a}, #{b}, #{c}, and #{d}")
+    #   log(:action) { "may be #{best} in a block because you can #{turn} #{it} #{off}" }
+    #
+    def log(action, msg=nil, level=Logger::INFO)
+      if !quiet || verbose
+        msg ||= yield
+        logger.add(level, msg, action.to_s)
+      end
     end
     
     # Returns a new node that executes block on call.
@@ -195,81 +239,323 @@ module Tap
       node
     end
     
-    # Adds the specified middleware to the stack.
+    # Adds the specified middleware to the stack.  The argv will be used as
+    # extra arguments to initialize the middleware.
     def use(middleware, *argv)
       synchronize do
         @stack = middleware.new(@stack, *argv)
       end
     end
     
+    # Sets the object to the specified variable and returns obj.  Provide nil
+    # as obj to un-set a variable (in which case the existing object is
+    # returned).
+    #
+    # Nil is reserved as a variable name and cannot be used by set.
+    def set(var, obj)
+      raise "no var specified" if var.nil?
+      
+      if obj
+        objects[var] = obj
+      else
+        objects.delete(var)
+      end
+    end
+    
+    # Returns the object set to var, or self if var is nil.
+    def get(var)
+      var.nil? ? self : objects[var]
+    end
+    
+    # Same as get, but raises an error if no object is set to the variable.
+    def obj(var)
+      get(var) or raise "no object set to: #{var.inspect}"
+    end
+    
+    # Returns the variable for the object.  If the object is not assigned to a
+    # variable and auto_assign is true, then the object is set to an unused
+    # variable and the new variable is returned.
+    #
+    # The new variable will be an integer and will be removed upon gc.
+    def var(obj, auto_assign=true)
+      objects.each_pair do |var, object|
+        return var if obj == object
+      end
+      
+      return nil unless auto_assign
+      
+      var = objects.length
+      loop do 
+        if objects.has_key?(var)
+          var += 1
+        else
+          set(var, obj)
+          return var
+        end
+      end
+    end
+    
+    # Removes objects keyed by integers.  If all is specified, gc will clear
+    # all objects.
+    def gc(all=false)
+      if all
+        objects.clear
+      else
+        objects.delete_if {|var, obj| var.kind_of?(Integer) }
+      end
+      
+      self
+    end
+    
+    # Sends a signal to an application object.  The input should be a hash
+    # defining these fields:
+    #
+    #   obj      # a variable identifying an object, or nil for self
+    #   sig      # the signal name
+    #   args     # arguments to the signal (typically a Hash)
+    #
+    # Call does the following:
+    #
+    #   object = app.get(obj)        # lookup an application object by obj
+    #   signal = object.signal(sig)  # lookup a signal by sig
+    #   signal.call(args)            # call the signal with args
+    #
+    # Call returns the result of the signal call.
+    #
+    def call(args, &block)
+      obj = args['obj']
+      sig = args['sig']
+      args = args['args'] || args
+      
+      route(obj, sig, &block).call(args)
+    end
+    
+    def route(obj, sig, &block)
+      unless object = get(obj)
+        raise "unknown object: #{obj.inspect}"
+      end
+      
+      unless object.respond_to?(:signal)
+        raise "cannot signal: #{object.inspect}"
+      end
+      
+      object.signal(sig, &block)
+    end
+
+    def resolve(const_str)
+      constant = env ? env[const_str] : Env::Constant.constantize(const_str)
+      constant or raise "unresolvable constant: #{const_str.inspect}"
+    end
+    
+    def build(spec)
+      var = spec['var']
+      clas = spec['class']
+      spec = spec['spec'] || spec
+      obj = nil
+      
+      if clas.nil?
+        unless spec.empty?
+          raise "no class specified"
+        end
+      else
+        clas = resolve(clas)
+        
+        case spec
+        when Array
+          parse = bang ? :parse! : :parse
+          obj, args = clas.send(parse, spec, self)
+      
+          if block_given?
+            yield(obj, args)
+          else
+            warn_ignored_args(args)
+          end
+        
+        when Hash
+          obj = clas.build(spec, self)
+        else
+          raise "invalid spec: #{spec.inspect}"
+        end
+      end
+      
+      unless var.nil?
+        if var.respond_to?(:each)
+          var.each {|v| set(v, obj) }
+        else
+          set(var, obj)
+        end
+      end
+      
+      obj
+    end
+    
+    def parse(argv, &block) # :yields: spec
+      parse!(argv.dup, &block)
+    end
+    
+    def parse!(argv, &block) # :yields: spec
+      parser = Parser.new
+      argv = parser.parse!(argv)
+      
+      # The queue API does not provide a delete method, so picking out the
+      # deque jobs requires the whole queue be cleared, then re-enqued.
+      # Safety (and speed) is improved with synchronization.
+      queue.synchronize do
+        deque = []
+        blocks = {}
+        
+        if auto_enque
+          blocks[:node] = lambda do |obj, args|
+            queue.enq(obj, args)
+            args = nil
+          end
+          
+          blocks[:join] = lambda do |obj, args|
+            unless obj.respond_to?(:outputs)
+              # warning
+            end
+            deque.concat obj.outputs
+          end
+        end
+        
+        parser.specs.each do |spec|
+          if block_given?
+            next unless yield(spec)
+          end
+          
+          type, obj, sig, *args = spec
+          
+          sig_block = case sig
+          when 'set'
+            blocks[type]
+          when 'parse'
+            block
+          else
+            nil
+          end
+          
+          call('obj' => obj, 'sig' => sig, 'args' => args, &sig_block)  
+        end
+        
+        deque.uniq!
+        queue.clear.each do |(obj, args)|
+          if deque.delete(obj)
+            warn_ignored_args(args)
+          else
+            queue.enq(obj, args)
+          end
+        end
+      end
+
+      argv
+    end
+    
+    # Enques the application object specified by var with args.  Raises
+    # an error if no such application object exists.
+    def enque(var, *args)
+      unless node = get(var)
+        raise "unknown object: #{var.inspect}"
+      end
+      
+      queue.enq(node, args)
+      node
+    end
+    
     # Returns an array of middlware in use by self.
     def middleware
       middleware = []
       
+      # collect middleware by walking up the stack
       synchronize do
         current = stack
-        until current.kind_of?(Stack)
+        visited = [current]
+        
+        while current.respond_to?(:stack)
           middleware << current
           current = current.stack
+          
+          circular_stack = visited.include?(current)
+          visited << current
+          
+          if circular_stack
+            visited.collect! {|middleware| middleware.class.to_s }.join(', ')
+            raise "circular stack detected:\n[#{visited}]"
+          end
         end
       end
       
       middleware
     end
     
-    # Clears the cache, the queue, and resets the stack so that no middleware
-    # is used.  Reset raises an error unless state == State::READY.
+    # Clears objects, the queue, and resets the stack so that no middleware
+    # is used.  Reset raises an error unless state is READY.
     def reset
       synchronize do
         unless state == State::READY
           raise "cannot reset unless READY"
         end
         
-        @stack = Stack.new(self)
-        cache.clear
+        # walk up middleware to find the base of the stack
+        while @stack.respond_to?(:stack)
+          @stack = @stack.stack
+        end
+        
+        objects.clear
         queue.clear
       end
     end
     
-    # Dispatches node to the application stack with the inputs.
+    # Execute is a wrapper for dispatch allowing inputs to be listed out
+    # rather than provided as an array.
     def execute(node, *inputs)
       dispatch(node, inputs)
     end
     
-    # Dispatch sends the node into the application stack with the inputs.
     # Dispatch does the following in order:
     #
     # - call stack with the node and inputs
-    # - call the node joins, if set, or the default_joins with the results
+    # - call the node joins (node.joins)
+    #
+    # The joins for self will be called if the node joins are an empty array.
+    # No joins will be called if the node joins are nil, or if the node does
+    # not provide a joins method.
     #
-    # Dispatch returns the node result.
+    # Dispatch returns the stack result.
     def dispatch(node, inputs=[])
       result = stack.call(node, inputs)
       
-      joins = node.joins.empty? ? default_joins : node.joins
-      joins.each do |join|
-        join.call(result)
+      if node.respond_to?(:joins)
+        if joins = node.joins
+
+          if joins.empty?
+            joins = self.joins
+          end
+        
+          joins.each do |join|
+            join.call(result)
+          end
+        end
       end
+      
       result
     end
     
     # Sequentially dispatches each enqued (node, inputs) pair to the
-    # application stack.  A run continues until the queue is empty.  Returns
-    # self.
-    #
-    # ==== Run State
+    # application stack.  A run continues until the queue is empty.  
     #
     # Run checks the state of self before dispatching a node.  If the state
-    # changes from State::RUN, the following behaviors result:
-    # 
-    # State::STOP:: No more nodes will be dispatched; the current node will
-    #               continute to completion.
-    # State::TERMINATE:: No more nodes will be dispatched and the currently
-    #                    running node will be discontinued as described in
-    #                    terminate.
-    #
-    # Calls to run when the state is not State::READY do nothing and
-    # return immediately.
+    # changes from RUN, the following behaviors result:
+    #   
+    #   STOP        No more nodes will be dispatched; the current node
+    #               will continute to completion.
+    #   TERMINATE   No more nodes will be dispatched and the currently
+    #               running node will be discontinued as described in
+    #               terminate.
+    #
+    # Calls to run when the state is not READY do nothing and return
+    # immediately.
+    #
+    # Returns self.
     def run
       synchronize do
         return self unless state == State::READY
@@ -292,49 +578,51 @@ module Tap
     end
     
     # Signals a running app to stop dispatching nodes to the application stack
-    # by setting state = State::STOP.  The node currently in the stack will
-    # will continue to completion.
+    # by setting state to STOP.  The node currently in the stack will continue
+    # to completion.
     #
-    # Does nothing unless state is State::RUN.
+    # Does nothing unless state is RUN.
     def stop
       synchronize { @state = State::STOP if state == State::RUN }
       self
     end
 
-    # Signals a running application to terminate execution by setting 
-    # state = State::TERMINATE.  In this state, calls to check_terminate
-    # will raise a TerminateError.  Run considers TerminateErrors a normal
-    # exit and rescues them quietly.
+    # Signals a running application to terminate execution by setting state to
+    # TERMINATE.  In this state, calls to check_terminate will raise a
+    # TerminateError.  Run considers TerminateErrors a normal exit and rescues
+    # them quietly.
     #
     # Nodes can set breakpoints that call check_terminate to invoke
     # node-specific termination.  If a node never calls check_terminate, then
-    # it will continue to completion and terminate is functionally the same
-    # as stop.
+    # it will continue to completion.
     #
-    # Does nothing if state == State::READY.
+    # Does nothing if state is READY.
     def terminate
       synchronize { @state = State::TERMINATE unless state == State::READY }
       self
     end
     
-    # Raises a TerminateError if state == State::TERMINATE.  Nodes should call
+    # Raises a TerminateError if state is TERMINATE.  Nodes should call
     # check_terminate to provide breakpoints in long-running processes.
+    #  
+    # A block may be provided to check_terminate to execute code before
+    # raising the TerminateError.
     def check_terminate
-      if state == App::State::TERMINATE
-        yield if block_given?
-        raise App::TerminateError.new
+      if state == State::TERMINATE
+        yield() if block_given?
+        raise TerminateError.new
       end
     end
     
     # Returns an information string for the App.  
     #
-    #   App.instance.info   # => 'state: 0 (READY) queue: 0'
+    #   App.new.info   # => 'state: 0 (READY) queue: 0'
     #
     def info
       "state: #{state} (#{State.state_str(state)}) queue: #{queue.size}"
     end
     
-    # Dumps self to the target as YAML.
+    # Dumps self to the target as YAML. (note dump is still experimental)
     #
     # ==== Notes
     #
@@ -372,14 +660,173 @@ module Tap
       target
     end
     
-    # Sets the block to receive the result of nodes with no joins
-    # (ie the block is set as a default_join).
-    def on_complete(&block) # :yields: _result
-      self.default_joins << block if block
-      self
+    # Converts the self to a schema that can be used to build a new app with
+    # equivalent application objects, queue, and middleware.  Schema are a
+    # collection of signal hashes such that this will rebuild the state of a
+    # on b:
+    #
+    #   a, b = App.new, App.new
+    #   a.to_schema.each {|spec| b.call(spec) }
+    #
+    # Application objects that do not satisfy the application object API are
+    # quietly ignored; enable debugging to be warned of their existance.
+    #
+    def serialize(bare=true)
+      # setup variables
+      specs = {}
+      order = []
+      
+      # collect enque signals to setup queue
+      signals = queue.to_a.collect do |(node, args)|
+        {'sig' => 'enque', 'args' => [var(node)] + args}
+      end
+      
+      # collect and trace application objects
+      objects.keys.sort_by do |var|
+        var.to_s
+      end.each do |var|
+        obj = objects[var]
+        order.concat trace(obj, specs)
+      end
+      
+      middleware.each do |obj|
+        order.concat trace(obj, specs)
+      end
+      
+      if bare
+        order.delete(self)
+        specs.delete(self)
+      else
+        order.unshift(self)
+        trace(self, specs)
+      end
+      order.uniq!
+      
+      # assemble specs
+      variables = {}
+      objects.each_pair do |var, obj|
+        (variables[obj] ||= []) << var
+      end
+      
+      invert_env = env ? env.invert : nil
+      specs.keys.each do |obj|
+        spec = {'sig' => 'set'}
+        
+        # assign variables
+        if vars = variables[obj]
+          if vars.length == 1
+            spec['var'] = vars[0]
+          else
+            spec['var'] = vars
+          end
+        end
+
+        # assign the class
+        klass = obj.class
+        klass = invert_env[klass] if invert_env
+        spec['class'] = klass.to_s
+
+        # merge obj_spec if possible
+        obj_spec = specs[obj]
+        if (obj_spec.keys & RESERVED_KEYS).empty?
+          spec.merge!(obj_spec)
+        else
+          spec['spec'] = obj_spec
+        end
+        
+        specs[obj] = spec
+      end
+      
+      middleware.each do |obj|
+        spec = specs[obj]
+        spec['sig'] = 'use'
+      end
+      
+      order.collect! {|obj| specs[obj] }.concat(signals)
     end
     
-    protected
+    def to_spec
+      signals = serialize(false)
+      spec = signals.shift
+      
+      spec.delete('self')
+      spec.delete('sig')
+      
+      var = spec.delete('var')
+      klass = spec.delete('class')
+      spec = spec.delete('spec') || spec
+      
+      signals.unshift(
+        'sig' => 'set',
+        'var' => var, 
+        'class' => klass, 
+        'self' => true
+      ) if var
+      spec['signals'] = signals
+      
+      spec
+    end
+    
+    def inspect
+      "#<#{self.class}:#{object_id} #{info}>"
+    end
+
+    private
+
+    # warns of ignored args
+    def warn_ignored_args(args) # :nodoc:
+      if args && debug? && !args.empty?
+        warn "ignoring args: #{args.inspect}"
+      end
+    end
+    
+    # Traces each object backwards and forwards for node, joins, etc. and adds
+    # each to specs as needed.  The trace determines and returns the order in
+    # which these specs must be initialized to make sense.  Circular traces
+    # are detected.
+    #
+    # Note that order should not be provided for the first call; order must be
+    # trace-specific.  For example (a -> b means 'a' references or requires
+    # 'b' so read backwards for order):
+    #
+    #   # Circular trace [a,c,b,a]
+    #   a -> b -> c -> a
+    #
+    #   # Not a problem [[b,a], [b,c]] => [b,a,c]
+    #   a -> b
+    #   c -> b
+    #   
+    def trace(obj, specs, order=[]) # :nodoc:
+      if specs.has_key?(obj)
+        return order
+      end
+      
+      # check the object can be serialized
+      unless obj.respond_to?(:to_spec)
+        warn "cannot serialize: #{obj}"
+        return order
+      end
+      specs[obj] = obj == self ? self_to_spec : obj.to_spec
+      
+      # trace references; refs must exist before obj and
+      # obj must exist before brefs (back-references)
+      if obj.respond_to?(:associations)
+        refs, brefs = obj.associations
+      
+        refs.each {|ref| trace(ref, specs, order) } if refs
+        order << obj
+        brefs.each {|bref| trace(bref, specs, order) } if brefs
+      else
+        order << obj
+      end
+      
+      order
+    end
+    
+    def self_to_spec # :nodoc:
+      config = self.config.to_hash {|hash, key, value| hash[key.to_s] = value }
+      {'config' => config, 'self' => true}
+    end
     
     # TerminateErrors are raised to kill executing nodes when terminate is 
     # called on an running App.  They are handled by the run rescue code.