wukong 3.0.0.pre3 → 3.0.0

data/Gemfile

@@ -5,6 +5,7 @@ gemspec
 group :development do
   gem 'rake',        '>= 0.9'
   gem 'rspec',       '>= 2.8'
+  gem 'spork',       '0.9.2'
   gem 'guard',       '>= 1.0'
   gem 'guard-rspec', '>= 0.6'
   gem 'simplecov',   '>= 0.5'

data/README.md

@@ -19,6 +19,8 @@ Here is a list of various other projects which you may also want to
 peruse when trying to understand the full Wukong experience:
 
 * <a href="http://github.com/infochimps-labs/wukong-hadoop">wukong-hadoop</a>: Run Wukong processors as mappers and reducers within the Hadoop framework.  Model Hadoop jobs locally before you run them.
+* <a href="http://github.com/infochimps-labs/wukong-storm>wukong-storm</a>: Run Wukong processors within the Storm framework.  Model flows locally before you run them.
+* <a href="http://github.com/infochimps-labs/wukong-load>wukong-load</a>: Load the output data from your local Wukong jobs and flows into a variety of different data stores.
 * <a href="http://github.com/infochimps-labs/wonderdog">wonderdog</a>: Connect Wukong processors running within Hadoop to Elasticsearch as either a source or sink for data.
 * <a href="http://github.com/infochimps-labs/wukong-deploy">wukong-deploy</a>: Orchestrate Wukong and other wu-tools together to support an application running on the Infochimps Platform.
 
@@ -36,7 +38,7 @@ processor is Ruby class which
 * subclasses `Wukong::Processor` (use the `Wukong.processor` method as sugar for this)
 * defines a `process` method which takes an input record, does something, and calls `yield` on the output
 
-Here's a processor that reverses all each input record:
+Here's a processor that reverses each of its input records:
 
 ```ruby
 # in string_reverser.rb
@@ -47,8 +49,8 @@ Wukong.processor(:string_reverser) do
 end
 ```
 
-When you're developing your application, run your processors on the
-command line on flat input files using `wu-local`:
+You can run this processor on the command line using text files as
+input using the `wu-local` tool that comes with Wukong:
 
 ```
 $ cat novel.txt
@@ -59,35 +61,46 @@ $ cat novel.txt | wu-local string_reverser.rb
 .semit fo tsrow eht saw ti ,semit fo tseb eht saw tI
 ```
 
-You can use yield as often (or never) as you need.  Here's a more
-complicated example to illustrate:
+The `wu-local` program consumes one line at at time from STDIN and
+calls your processor's `process` method with that line as a Ruby
+String object.  Each object you `yield` within your process method
+will be printed back out on STDOUT.
+
+### Multiple Processors, Multiple (Or No) Yields
+
+Processors are intended to be combined so they can be stored in the
+same file like these two, related processors:
 
 ```ruby
 # in processors.rb
 
-Wukong.processor(:tokenizer) do
+Wukong.processor(:splitter) do
   def process line
     line.split.each { |token| yield token }
   end
 end
   
-Wukong.processor(:starts_with) do
-
-  field :letter, String, :default => 'a'
-  
-  def process word
-    yield word if word =~ Regexp.new("^#{letter}", true)
+Wukong.processor(:normalizer) do
+  def process token
+    stripped = token.downcase.gsub(/\W/,'')
+	yield stripped if stripped.size > 0
   end
 end
 ```
 
-Let's start by running the `tokenizer`.  We've defined two processors
-in the file `processors.rb` and neither one is named `processors` so
-we have to tell `wu-local` the name of the processor we want to run
-explicitly.
+Notice how the `splitter` yields multiple tokens for each of its input
+tokens and that the `normalizer` may sometimes never yield at all,
+depending on its input.  Processors are under no obligations by the
+framework to yield or return anything so they can easily act as
+filters or even sinks in data flows.
+
+There are two processors in this file and neither shares a name with
+the basename of the file ("processors") so `wu-local` can't
+automatically choose a processor to run.  We can specify one
+explicitly with the `--run` option:
 
 ```
-$ cat novel.txt | wu-local processors.rb --run=tokenizer
+$ cat novel.txt | wu-local processors.rb --run=splitter
 It
 was
 the
@@ -97,39 +110,454 @@ times,
 ...
 ```
 
-You can combine the output of one processor with another right in the
-shell.  Let's add the `starts_with` filter and also pass in the
-*field* `letter`, defined in that processor:
+We can combine the two processors together
 
 ```
-$ cat novel.txt | wu-local processors.rb --run=tokenizer | wu-local processors.rb --run=starts_with --letter=t
-the
-times
+$ cat novel.txt | wu-local processors.rb --run=splitter | wu-local processors.rb --run=normalizer
+it
+was
 the
+best
+of
 times
 ...
 ```
 
-Wanting to match on a regular expression is such a common task that
-Wukong has a built-in "widget" called `regexp` that you can use
-directly:
+but there's an easier way of doing this with <a href="#flows">dataflows</a>.
+
+### Adding Configurable Options
+
+Processors can have options that can be set in Ruby code, from the
+command-line, a configuration file, or a variety of other places
+thanks to [Configliere](http://github.com/infochimps-labs/configliere).
+
+This processor calculates percentiles from observations assuming a
+normal distribution given a particular mean and standard deviation.
+It uses two *fields*, the mean or average of a distribution (`mean`)
+and its standard deviation (`std_dev`).  From this information, it
+will measure the percentile of all input values.
+
+```ruby
+# in percentile.rb
+Wukong.processor(:percentile) do
+
+  SQRT_1_HALF = Math.sqrt(0.5)
+
+  field :mean,    Float, :default => 0.0
+  field :std_dev, Float, :default => 1.0
+
+  def process value
+    observation = value.to_f
+    z_score     = (mean - observation) / std_dev
+    percentile  = 50 * Math.erfc(z_score * SQRT_1_HALF)
+    yield [observation, percentile].join("\t")
+  end
+end
+```
+
+These fields have default values but you can overide them on the
+command line.  If you scored a 95 on an exam where the mean score was
+80 points and the standard deviation of the scores was 10 points, for
+example, then you'd be in the 93rd percentile:
+
+```
+$ echo 95 | wu-local /tmp/percentile.rb --mean=80 --std_dev=10
+95.0	93.3192798731142
+```
+
+If the exam were more difficult, with a mean of 75 points and a
+standard deviation of 8 points, you'd be in the 99th percentile!
+
+```
+$ echo 95 | wu-local /tmp/percentile.rb --mean=75 --std_dev=8
+95.0	99.37903346742239
+```
+
+### The Lifecycle of a Processor
+
+Processors have a lifecycle that they execute when they are run within
+the context of a Wukong runner like `wu-local` or `wu-hadoop`.  Each
+lifecycle phase corresponds to a method of the processor that is
+called:
+
+* `setup` called *after* the Processor is initialized but *before* the first record is processed.  You cannot yield from this method.
+* `process` called once for each input record, may yield once, many, or no times.
+* `finalize` called after the the *last* record has been processed but while the processor still has an opportunity to yield records.
+* `stop` called to signal to the processor that all work should stop, open connections should be closed, &c.  You cannot yield from this method.
+
+The above examples have already focused on the `process` method.
+
+The `setup` and `stop` methods are often used together to handle
+external connections
+
+```ruby
+# in geolocator.rb
+Wukong.processor(:geolocator) do
+  field :host, String, :default => 'localhost'
+  attr_accessor :connection
+  
+  def setup
+    self.connection = Database::Connection.new(host)
+  end
+  def process record
+    record.added_value = connection.find("...some query...")
+  end
+  def stop
+    self.connection.close
+  end
+end
+```
+
+The `finalize` method is most useful when writing a "reduce"-type
+operation that involves storing or aggregating information till some
+criterion is met.  It will always be called after the last record has
+been given (to `process`) but you can call it whenever you want to
+within your own code.
+
+Here's an example of using the `finalize` method to implement a simple
+counter that counts all the input records:
+
+```ruby
+# in counter.rb
+Wukong.processor(:counter) do
+  attr_accessor :count
+  def setup
+    self.count = 0
+  end
+  def process thing
+    self.count += 1
+  end
+  def finalize
+    yield count
+  end
+end
+```
+
+It hinges on the fact that the last input record will be passed to
+`process` *first* and only then will `finalize` be called.  This
+allows the last input record to be counted/processed/aggregated and
+then the entire aggregate to be dealt with in finalize.
+
+Because of this emphasis on building and processing aggregates, the
+`finalize` method is often useful within processors meant to run as
+reducers in a Hadoop environment.
+
+Note:: Finalize is not guaranteed to be called by in every possible
+environment as it depends on the chosen runner.  In a local or Hadoop
+environment, the notion of "last record" makes sense and so the
+corresponding runners will call `finalize`.  In an environment like
+Storm, where the concept of last record is not (supposed to be)
+meaningful, the corresponding runner doesn't ever call it.
+
+### Serialization
+
+`wu-local` (and many similar tools) deal with inputs and outputs as
+strings.
+
+Processors want to process objects as close to their domain as is
+possible.  A processor which decorates address book entries with
+Twitter handles doesn't want to think of its inputs as Strings but
+Hashes or, better yet, Persons.
+
+Wukong makes it easy to wrap a processor with other processors
+dedicated to handling the common tasks of parsing records into or out
+of formats like JSON and turning them into Ruby model instances.
+
+#### De-serializing data formats like JSON or TSV
+
+Wukong can parse and emit common data formats like JSON and delimited
+formats like TSV or CSV so that you don't pollute or tie down your own
+processors with protocol logic.
+
+Here's an example of a processor that wants to deal with Hashes as
+input.
+
+```ruby
+# in extractor.rb
+Wukong.processor(:extractor) do
+  def process hsh
+    yield hsh["first_name"]
+  end
+end
+```
+
+Given JSON data,
+
+```
+$ cat input.json
+{"first_name": "John", "last_name":, "Smith"}
+{"first_name": "Sally", "last_name":, "Johnson"}
+...
+```
+
+you can feed it directly to a processor
+
+```
+$ cat input.json | wu-local --from=json extractor
+John
+Sally
+...
+```
+
+Other processors really like Arrays:
+
+```ruby
+Wukong.processor(:summer) do
+  def process values
+    yield values.map(&:to_f).inject(0.0) { |sum, summand| sum += summand }
+  end
+end
+```
+
+so you can feed them TSV data
+```
+$ cat data.tsv
+1	2	3
+4	5	6
+7	8	9
+...
+$ cat data.tsv | wu-local --from=tsv summer
+6
+15
+24
+...
+```
+
+but you can just as easily use the same code with CSV data
+
+```
+$ cat data.tsv | wu-local --from=csv summer
+```
+
+or a more general delimited format.
+
+```
+$ cat data.tsv | wu-local --from=delimited --delimiter='--' summer
+```
+
+#### Recordizing data structures into domain models
+
+Here's a contact validator that relies on a Person model to decide
+whether a contact entry should be yielded:
+
+```ruby
+# in contact_validator.rb
+require 'person'
+
+Wukong.processor(:contact_validator) do
+  def process person
+    yield person if person.valid?
+  end
+end
+```
+
+Relying on the (elsewhere-defined) Person model to define `valid?`
+means the processor can stay skinny and readable.  Wukong can, in
+combination with the deserializing features above, turn input text
+into instances of Person:
+
+```
+$ cat input.json | wu-local --consumes=Person --from=json contact_validator
+#<Person:0x000000020e6120>
+#<Person:0x000000020e6120>
+#<Person:0x000000020e6120>
+```
+
+`wu-local` can also serialize records from the `contact_validator`
+processor:
+
+```
+$ cat input.json | wu-local --consumes=Person --from=json contact_validator --to=json
+{"first_name": "John", "last_name":, "Smith", "valid": "true"}
+{"first_name": "Sally", "last_name":, "Johnson", "valid": "true"}
+...
+```
+
+Serialization formats work just like deserialization formats, with
+JSON as well as delimited formats available.
+
+Parsing records into model instances and serializing them out again
+puts constraints on the model class providing these instances.  Here's
+what the `Person` class needs to look like:
+
+
+```ruby
+# in person.rb
+class Person
+
+  # Create a new Person from the given attributes.  Supports usage of
+  # the `--consumes` flag on the command-line
+  # 
+  # @param [Hash] attrs
+  # @return [Person]
+  def self.receive attrs
+    new(attrs)
+  end
+  
+  # Turn this Person into a basic data structure.  Supports the usage
+  # of the `--to` flag on the command-line.
+  # 
+  # @return [Hash]
+  def to_wire
+    to_hash
+  end
+end
+```
+
+To support the `--consumes=Person` syntax, the `receive` class method
+must take a Hash produced from the operation of the `--from` argument
+and return a `Person` instance.
+
+To support the `--to=json` syntax, the `Person` class must implement
+the `to_wire` instance method.
+
+### Logging and Notifications
+
+Wukong comes with a logger that all processors have access to via
+their `log` attribute.  This logger has the following priorities:
+
+* debug (can be set as a log level)
+* info (can be set as a log level)
+* warn (can be set as a log level)
+* error
+* fatal
+
+and here's a processor which uses them all
+
+```ruby
+# in logs.rb
+Wukong.processor(:logs) do
+  def process line
+    log.debug line
+    log.info  line
+    log.warn  line
+    log.error line
+    log.fatal line
+  end
+end
+```
+
+The default log level is DEBUG.  
+
+```
+$ echo something | wu-local logs.rb
+DEBUG 2013-01-11 23:40:56 [Logs                ] -- event
+INFO 2013-01-11 23:40:56 [Logs                ] -- event
+WARN 2013-01-11 23:40:56 [Logs                ] -- event
+ERROR 2013-01-11 23:40:56 [Logs                ] -- event
+FATAL 2013-01-11 23:40:56 [Logs                ] -- event
+```
+
+though you can set it to something else globally
+
+```
+$ echo something | wu-local logs.rb --log.level=warn
+WARN 2013-01-11 23:40:56 [Logs                ] -- event
+ERROR 2013-01-11 23:40:56 [Logs                ] -- event
+FATAL 2013-01-11 23:40:56 [Logs                ] -- event
+```
+
+or on a per-class basis.
+
+### Creating Documentation
+
+`wu-local` includes a help message:
+
+```
+$ wu-local --help
+usage: wu-local [ --param=val | --param | -p val | -p ] PROCESSOR|FLOW
+
+wu-local is a tool for running Wukong processors and flows locally on
+the command-line.  Use wu-local by passing it a processor and feeding
+...
+
+
+Params:
+   -r, --run=String             Name of the processor or dataflow to use. Defaults to basename of the given path.
+   -t, --tcp_port=Integer       Consume TCP requests on the given port instead of lines over STDIN
+```
+
+You can generate custom help messages for your own processors.  Here's
+the percentile processor from before but made more usable with good
+documentation:
 
+```ruby
+# in percentile.rb
+Wukong.processor(:percentile) do
+
+  description <<-EOF.gsub(/^ {2}/,'')
+  This processor calculates percentiles from input scores based on a
+  given mean score and a given standard deviation for the scores.
+
+  The mean and standard deviation are given at run time and processed
+  scores will be compared against the given mean and standard
+  deviation.
+
+  The input is expected to consist of float values, one per line.
+
+  Example:
+
+    $ cat input.dat
+    88
+    89
+    77
+    ...
+
+    $ cat input.dat | wu-local percentile.rb --mean=85 --std_dev=7
+    88.0	66.58824291023753
+    89.0	71.61454169013237
+    77.0	12.654895447355777
+  EOF
+	
+  SQRT_1_HALF = Math.sqrt(0.5)
+
+  field :mean,    Float, :default => 0.0, :doc => "The mean of the assumed distribution"
+  field :std_dev, Float, :default => 1.0, :doc => "The standard deviation of the assumed distribution"
+
+  def process value
+    observation = value.to_f
+    z_score     = (mean - observation) / std_dev
+    percentile  = 50 * Math.erfc(z_score * SQRT_1_HALF)
+    yield [observation, percentile].join("\t")
+  end
+end
 ```
-$ cat novel.txt | wu-local processors.rb --run=tokenizer | wu-local regexp --match='^t'
+
+If you call `wu-local` with the file to this processor as an argument
+in addition to the original `--help` argument, you'll get custom
+documentation.
+
 ```
+$ wu-local percentile.rb --help
+usage: wu-local [ --param=val | --param | -p val | -p ] PROCESSOR|FLOW
+
+This processor calculates percentiles from input scores based on a
+given mean score and a given standard deviation for the scores.
+...
 
-There are many more simple <a href="#widgets">widgets</a> like these.
+
+Params:
+       --mean=Float             The mean of the assumed distribution [Default: 0.0]
+   -r, --run=String             Name of the processor or dataflow to use. Defaults to basename of the given path.
+       --std_dev=Float          The standard deviation of the assumed distribution [Default: 1.0]
+   -t, --tcp_port=Integer       Consume TCP requests on the given port instead of lines over STDIN
+
+```
 
 <a name="flows"></a>
 ## Combining Processors into Dataflows
 
 Combining processors which each do one thing well together in a chain
 is mimicing the tried and true UNIX pipeline.  Wukong lets you define
-these pipelines more formally as a dataflow.  Here's the dataflow for
+these pipelines more formally as a dataflow.
+
+Having written the `tokenizer` processor, we can use it in a dataflow
+along with the built-in `regexp` processor to replicate what we did in
 the last example:
 
 ```
 # in find_t_words.rb
+require_relative('processors')
 Wukong.dataflow(:find_t_words) do
   tokenizer | regexp(match: /^t/)
 end
@@ -148,7 +576,8 @@ times
 ...
 ```
 
-and it works exactly like before.
+and it works exactly like manually chaining the two processors
+together.
 
 <a name="serialization></a>
 ## Serialization
@@ -163,7 +592,14 @@ yield a String argument (or something that will `to_s` appropriately).
 ## Widgets
 
 Wukong has a number of built-in widgets that are useful for
-scaffolding your dataflows.
+scaffolding your dataflows or using as starting off points for your
+own processors.
+
+For any of these widgets you can get customized help, say
+
+```
+$ wu-local group --help
+```
 
 ### Serializers
 
@@ -350,10 +786,10 @@ describe :tokenizer do
     processor.given("Hi there.\nMy name is Wukong!").should emit(6).records
   end
   it "eliminates all punctuation" do
-    processor.given("Never!").output.first.should_not include(',')
+    processor(:tokenizer).given("Never!").should emit('Never')
   end
-  it "downcases all input text" do
-    processor.given("Whatever").output.first.should match(/^w/)
+  it "will not emit tokens in a stop list" do
+    processor(:tokenizer, :stop_list => ['apples', 'bananas']).given("I like apples and bananas").should emit('I', 'like', 'and')
   end
 end
 ```
@@ -364,8 +800,13 @@ Let's look at each kind of helper:
   `it_behaves_like` helper) adds some tests that ensure that the
   processor conforms to the API of a Wukong::Processor.
 
-* The `processor` method instantiates a processor very similarly to
-  the way `wu-local` instantiates one on the command-line.  It accepts
+* The `processor` method is actually an alias for the more aptly named
+  (but less convenient) `unit_test_runner`.  This method accepts a
+  processor name and options (just like `wu-local` and other
+  command-line tools) and returns a Wukong::UnitTestRunner instance.
+  This runner handles the
+
+
   a (registered) processor name and options and creates a new
   processor.  If no name is given, the argument of the enclosing
   `describe` or `context` block is used.  The object returned by
@@ -374,29 +815,38 @@ Let's look at each kind of helper:
   behavior.
 
 * The `given` method (and other helpers like `given_json`,
-  `given_tsv`, &c.) is added to the Processor class when
-  Wukong::SpecHelpers is required. It's a way of lazily feeding
-  records to a processor, without having to go through the `process`
-  method directly and having to handle the block or the processor's
-  lifecycle as in the prior example.
+  `given_tsv`, &c.) is a method on the runner. It's a way of lazily
+  feeding records to a processor, without having to go through the
+  `process` method directly and having to handle the block or the
+  processor's lifecycle as in the prior example.
 
 * The `output` and `emit` matchers will `process` all previously
   `given` records when they are called. This lets you separate
   instantiation, input, expectations, and output. Here's a more
-  complicated example:
+  complicated example.
 
 The same helpers can be used to test dataflows as well as
-processors. For complete details, see documentation for the
-Wukong::SpecHelpers module.
+processors.
+
+#### 
+
+#### Functions vs. Objects
+
+The above test helpers are designed to aid in testing processors
+functionally because:
+
+* they accept the 
 
 ### Integration Tests
 
-Sometimes unit tests aren't enough and you need to test your
-processors or flows as they will be run in production using
-`wu-local`.
+If you are implementing a new Wukong command (akin to `wu-local`) then
+you may also want to run integration tests.  Wukong comes with helpers
+for these, too.
 
-For these use cases, Wukong provides some integration helpers that
-make testing command line processes easier.
+You should almost always be able to test your processors without
+integration tests.  Your unit tests and the Wukong framework itself
+should ensure that your processors work correctly no matter what
+environment they are deployed in.
 
 ```ruby
 # spec/integration/tokenizer_spec.rb
@@ -415,7 +865,7 @@ context "interpreting its arguments" do
   end
   context "with a malformed --match argument" do
     # invalid b/c the regexp is broken...
-    subject { command("wu-local tokenizer --match='^[h'") < "hi there" }
+    subject { command("wu-local tokenizer --match='^(h'") < "hi there" }
 	it      { should exit_with(:non_zero)   }
 	it      { should have_stderr(/invalid/) }
   end
@@ -457,3 +907,192 @@ Let's go through the helpers:
 * The `have_stdout` and `have_stderr` matchers let you test the STDOUT or STDERR of the command for particular strings or regular expressions.
 
 * The `exit_with` matcher lets you test the exit code of the command.  You can pass the symbol `:non_zero` to set the expectation of _any_ non-zero exit code.
+
+## Plugins
+
+Wukong has a built-in plugin framework to make it easy to adapt Wukong
+processors to new backends or add other functionality.  The
+`Wukong::Local` module and the `wu-local` program it supports is
+itself a Wukong plugin.
+
+The following shows how you might build a simplified version of
+`Wukong::Local` as a new plugin.  We'll call this plugin `Cat` as it
+will implement a program `wu-cat` that is similar in function to
+`wu-local` (just simplified).
+
+The first thing to do is include the `Wukong::Plugin` module in your
+code:
+
+
+```Ruby
+# in lib/cat.rb
+#
+# This Wukong plugin works like wu-local but replicates some silly
+# features of cat like numbered lines.
+module Cat
+
+  # This registers Cat as a Wukong plugin.
+  include Wukong::Plugin
+
+  # Defines any settings specific to Cat.  Cat doesn't need to, but
+  # you can define global settings here if you want.  You can also
+  # check the `program` name to decide whether to apply your settings.
+  # This helps you not pollute other commands with your stuff.
+  def self.configure settings, program
+	case program
+	when 'wu-cat'
+	  settings.define(:input,  :description => "The input file to use")
+	  settings.define(:number, :description => "Prepend each input record with a consecutive number", :type => :boolean)
+	else
+	  # configure other programs if you need to
+	end
+  end
+
+  # Lets Cat boot up with settings that have already been resolved
+  # from the command-line or other sources like config files or remote
+  # servers added by other plugins.
+  #
+  # The `root` directory in which the program is executing is also
+  # provided.
+  def self.boot settings, root
+    puts "Cat booting up using resolved settings within directory #{root}"
+  end
+end
+```
+
+If your plugin doesn't interact directly with the command-line
+(through a wu-tool like `wu-local` or `wu-hadoop`) and doesn't
+directly interface with passing records to processors then you can
+just require the rest of your plugin's code at this point and be done.
+
+### Write a Runner to interact with the command-line
+
+If you need to implement a new command line tool then you should write
+a Runner.  A Runner is used to implement Wukong programs like
+`wu-local` or `wu-hadoop`. Here's what the actual program file would
+look like for our example plugin's `wu-cat` program.
+
+```ruby
+#!/usr/bin/env ruby
+# in bin/wu-cat
+require 'cat'
+Cat::Runner.run
+```
+
+The Cat::Runner class is implemented separately.
+
+```ruby
+# in lib/cat/runner.rb
+require_relative('driver')
+module Cat
+
+  # Implements the `wu-cat` command.
+  class Runner < Wukong::Runner
+
+    usage "PROCESSOR|FLOW"
+	
+	description <<-EOF
+	
+	wu-cat lets you run a Wukong processor or dataflow on the
+	command-line.  Try it like this.
+
+    $ wu-cat --input=data.txt
+	hello
+	my
+	friend
+
+    Connect the output to a processor in upcaser.rb
+	
+    $ wu-cat --input=data.txt upcaser.rb
+	HELLO
+	MY
+	FRIEND
+
+    You can also include add line numbers to the output.
+
+    $ wu-cat --number --input=data.txt upcaser.rb
+	1	HELLO
+	2	MY
+	3	FRIEND
+    EOF
+
+    # The name of the processor we're going to run.  The #args method
+    # is provided by the Runner class.
+	def processor_name
+	  args.first
+	end
+
+    # Validate that we were given the name of a registered processor
+	# to run.  Be careful to return true here or validation will fail.
+    def validate
+      raise Wukong::Error.new("Must provide a processor as the first argument") unless processor_name
+	  true
+	end
+
+    # Delgates to a driver class to run the processor.
+    def run
+	  Driver.new(processor_name, settings).start
+	end
+	
+  end
+end
+```
+
+### Write a Driver to interact with processors
+
+The `Cat::Runner#run` method delegates to the `Cat::Driver` class to
+handle instantiating and interacting with processors.
+
+```ruby
+# in lib/cat/driver.rb
+module Cat
+
+  # A class for driving a processor from `wu-cat`.
+  class Driver
+
+    # Lets us count the records.
+    attr_accessor :number
+
+    # Gives methods to construct and interact with dataflows.
+    include Wukong::DriverMethods
+
+    # Create a new Driver for a dataflow with the given `label` using
+    # the given `settings`.
+    #
+    # @param [String] label the name of the dataflow
+    # @param [Configliere::Param] settings the settings to use when creating the dataflow
+    def initialize label, settings
+      self.settings = settings
+      self.dataflow = construct_dataflow(label, settings)
+      self.number   = 1
+    end
+
+    # The file handle of the input file.
+    #
+    # @return [File]
+    def input_file
+      @input_file ||= File.new(settings[:input])
+    end
+
+    # Starts feeding records to the processor
+    def start
+      while line = input_file.readline rescue nil
+        driver.send_through_dataflow(line)
+      end
+    end
+
+    # Process each record that comes back from the dataflow.
+	#
+	# @param [Object] record the yielded record
+    def process record
+      if settings[:number]
+        puts [number, record].map(&:to_s).join("\t")
+      else
+        puts record.to_s
+      end
+	  self.number += 1
+    end
+
+  end
+end
+```