wukong 3.0.0.pre → 3.0.0.pre2

data/README.md

@@ -1,422 +1,283 @@
-# Wukong [![Build Status](https://secure.travis-ci.org/infochimps-labs/wukong.png)](http://travis-ci.org/infochimps-labs/wukong)
-
-Wukong is a toolkit for rapid, agile development of dataflows at any scale.
-
-(note: the syntax below is mostly false)
-
-
-<a name="design"></a>
-## Design Overview
-
-The fundamental principle of Wukong/Hanuman is *powerful black boxes, beautiful glue*. In general, they don't do the thing -- they coordinate the boxes that do the thing, to let you implement rapidly, nimbly and readably. Hanuman elegantly describes high-level data flows; Wukong is a pragmatic collection of dataflow primitives. They both emphasize scalability, readability and rapid development over performance or universality.
-
-Wukong/Hanuman are chiefly concerned with these specific types of graphs:
-
-* **dataflow**      -- chains of simple modules to handle continuous data processing -- coordinates Flume, Unix pipes, ZeroMQ, Esper, Storm.
-* **workflows**     -- episodic jobs sequences, joined by dependency links -- comparable to Rake, Azkaban or Oozie.
-* **map/reduce**    -- Hadoop's standard *disordered/partitioned stream > partition, sort & group > process groups* workflow. Comparable to MRJob and Dumbo.
-* **queue workers** -- pub/sub asynchronously triggered jobs -- comparable Resque, RabbitMQ/AMQP, Amazon Simple Worker, Heroku workers.
-
-In addition, wukong stages may be deployed into **http middlware**: lightweight distributed API handlers -- comparable to Rack, Goliath or Twisted.
-
-When you're describing a Wukong/Hanuman flow, you're writing pure expressive ruby, not some hokey interpreted language or clumsy XML format. Thanks to JRuby, it can speak directly to Java-based components like Hadoop, Flume, Storm or Spark.
-
-## What's where
-
-* Configliere -- Manage settings
-  - Layer - Project settings through a late-resolved stack of config objects.
-* Gorillib
-  - Type, RecordType
-  - TypeConversion
-  - Model
-  - PathHelpers
-* Wukong
-  - fs - Abstracts file hdfs s3n s3hdfs scp
-  - streamer - Black-box data transform
-  - job - Workflow definition
-  - flow - Dataflow definition
-  - widgets - Common data transforms
-  - RubyHadoop - Hadoop jobs using streamers
-  - RubyFlume - Flume decorators using streamers
-* Hanuman -- Elegant small graph assembly
-* Swineherd -- Common interface on ugly tools
-  - Turn readable hash into safe commandline (param conv, escaping)
-  - Execute command, capture stdin/stderr
-  - Summarize execution with a broham-able hash
-  - Common modules: Input/output, Java, gnu, configliere
-  - Template
-  - Hadoop, pig, flume
-  - ?? Cp, mv, rm, zip, tar, bz2, gz, ssh, scp
-  - ?? Remotely execute command
-
-<a name="design-rules"></a>
-### Story
-
-[Narrative Method Structure](http://avdi.org/talks/confident-code-rubymidwest-2011/confident-code.html)
-
-* Gather input
-* Perform work
-* Deliver results
-* Handle failure
-
-
-<a name="design-rules"></a>
-### Design Rules
-
-* **whiteboard rule**: the user-facing conceptual model should match the picture you would draw on the whiteboard in an engineering discussion. The fundamental goal is to abstract the necessary messiness surrounding the industrial-strength components it orchestrates while still providing their essential power.
-* **common cases are simple, complex cases are always possible**: The code should be as simple as the story it tells. For the things you do all the time, you only need to describe how this data flow is different from all other data flows. However, at no point in the project lifecycle should Wukong/Hanuman hit a brick wall or peat bog requiring its total replacement. A complex production system may, for example, require that you replace a critical path with custom Java code -- but that's a small set of substitutions in an otherwise stable, scalable graph. In the world of web programming, Ruby on Rails passes this test; Sinatra and Drupal do not.
-* **petabyte rule**: Wukong/Hanuman coordinate industrial-strength components that wort at terabyte- and petabyte-scale. Conceptual simplicity makes it an excellent tool even for small jobs, but scalability is key. All components must assume an asynchronous, unreliable and distributed system.
-* **laptop rule**:
-* **no dark magick**: the core libraries provide *elegant, predictable magic or no magic at all*. We use metaprogramming heavily, but always predictably, and only in service of making common cases simple. 
-  - Soupy multi-option `case` statements are a smell.
-  - Complex tasks will require code that is more explicit, but readable and organically connected to the typical usage. For example, many data flows will require a custom `Wukong::Streamer` class; but that class is no more complex than the built-in streamer models and receives all the same sugar methods they do.
-* **get shit done**: sometimes ugly tasks require ugly solutions. Shelling out to the hadoop process monitor and parsing its output is acceptable if it is robust and obviates the need for a native protocol handler.
-* **be clever early, boring late**: magic in service of having a terse language for assembling a graph is great. However, the assembled graph should be stomic and largely free of any conditional logic or dependencies.
-  - for example, the data flow `split` statement allows you to set a condition on each branch. The assembled graph, however, is typically a `fanout` stage followed by `filter` stages.
-  - the graph language has some helpers to refer to graph stages. The compiled graph uses explicit mostly-readable but unambiguous static handles.
-  - some stages offer light polymorphism -- for example, `select` accepts either a regexp or block. This is handled at the factory level, and the resulting stage is free of conditional logic.
-* **no lock-in**: needless to say, Wukong works seamlessly with the Infochimps platform, making robust, reliable massive-scale dataflows amazingly simple. However, wukong flows are not tied to the cloud: they project to Hadoop, Flume or any of the other open-source components that power our platform.
-
-__________________________________________________________________________
-
-<a name="stage"></a>
-## Stage
-
-A graph is composed of `stage`s.
-
-* *desc* (alias `description`)
-
-#### Actions
-
-each action 
-
-* the default action is `call`
-* all stages respond to `nothing`, and like ze goggles, do `nothing`.
-
-__________________________________________________________________________
-
-<a name="dataflows"></a>
-## Workflows
-
-Wukong workflows work somewhat differently than you may be familiar with Rake and such.
-
-In wukong, a stage corresponds to a resource; you can then act on that resource.
-
-Consider first compiling a c program:
-
-    to build the executable, run `cc -o cake eggs.o milk.o flour.o sugar.o -I./include -L./lib`
-    to build files like '{file}.o', run `cc -c -o {file}.o {file}.c -I./include`
-
-In this case, you define the *steps*, implying the resources.
-
-
-Something rake can't do (but we should be able to): make it so I can define a dependency that runs **last** 
-
-### Defining jobs
-
-    Wukong.job(:launch) do
-      task :aim do
-        #...
-      end
-      task :enter do
-      end
-      task :commit do
-        # ...
-      end
-    end
-
-    Wukong.job(:recall) do
-      task :smash_with_rock do
-        #...
-      end
-      task :reprogram do
-        # ...
-      end
-    end
-    
-* stages construct resources
-  - these have default actions
-* hanuman tracks defined order
-
-* do steps run in order, or is dependency explicit?
-* what about idempotency?
-
-* `task` vs `action` vs `resource`; `job`, `task`, `group`, `namespace`.
-
-### documenting
-
-Inline option (`:desc` or `:description`?)
-
-  ```ruby
-      task :foo, :description => "pity the foo" do 
-        # ...
-      end
-  ```
-
-DSL method option
-
-  ```ruby
-      task :foo do
-        description "pity the foo"
-        # ...
-      end
-  ```
-
-### actions
-
-default action:
-
-  ```ruby
-      script 'nukes/launch_codes.rb' do
-        # ...
-      end
-  ```
+# Wukong
+
+Wukong is a toolkit for rapid, agile development of data applications
+at any scale.
+
+The core concept in Wukong is a **Processor**.  Wukong processors are
+simple Ruby classes that do one thing and do it well.  This codebase
+implements processors and other core Wukong classes and provides a
+tool, `wu-local`, to run and combine processors on the command-line.
+
+Wukong's larger theme is *powerful black boxes, beautiful glue*. The
+Wukong ecosystem consists of other tools which run Wukong processors
+in various topologies across a variety of different backends.  Code
+written in Wukong can be easily ported between environments and
+frameworks: local command-line scripts on your laptop instantly turn
+into powerful jobs running in Hadoop.
+
+Here is a list of various other projects which you may also want to
+peruse when trying to understand the full Wukong experience:
+
+* <a href="http://github.com/infochimps-labs/wukong-hadoop">wukong-hadoop</a>: Run Wukong processors as mappers and reducers within the Hadoop framework.  Model Hadoop jobs locally before you run them.
+* <a href="http://github.com/infochimps-labs/wonderdog">wonderdog</a>: Connect Wukong processors running within Hadoop to Elasticsearch as either a source or sink for data.
+* <a href="http://github.com/infochimps-labs/wukong-deploy">wukong-deploy</a>: Orchestrate Wukong and other wu-tools together to support an application running on the Infochimps Platform.
+
+For a more holistic perspective also see the Infochimps Platform
+Community Edition (**FIXME: link to this**) which combines all the
+Wukong tools together into a jetpack which fits comfortably over the
+shoulders of developers.
+
+<a name="processors"></a>
+## Writing Simple Processors
+
+The fundamental unit of computation in Wukong is the processor.  A
+processor is Ruby class which
+
+* subclasses `Wukong::Processor` (use the `Wukong.processor` method as sugar for this)
+* defines a `process` method which takes an input record, does something, and calls `yield` on the output
+
+Here's a processor that reverses all each input record:
+
+```ruby
+# in string_reverser.rb
+Wukong.processor(:string_reverser) do
+  def process string
+    yield string.reverse
+  end
+end
+```
+
+When you're developing your application, run your processors on the
+command line on flat input files using `wu-local`:
+
+```
+$ cat novel.txt
+It was the best of times, it was the worst of times.
+...
+
+$ cat novel.txt | wu-local string_reverser.rb
+.semit fo tsrow eht saw ti ,semit fo tseb eht saw tI
+```
+
+You can use yield as often (or never) as you need.  Here's a more
+complicated example to illustrate:
+
+```ruby
+# in processors.rb
+
+Wukong.processor(:tokenizer) do
+  def process line
+    line.split.each { |token| yield token }
+  end
+end
   
-define the `:undo` action:
-
-  ```ruby
-      script 'nukes/launch_codes.rb', :undo do
-        # ...
-      end
-  ```      
-
-<a name="file-name-templates"></a>
-### File name templates
-
-* *timestamp*: timestamp of run. everything in this invocation will have the same timestamp.
-* *user*: username; `ENV['USER']` by default
-* *sources*: basenames of job inputs, minus extension, non-`\w` replaced with '_', joined by '-', max 50 chars.
-* *job*:  job flow name
-
-<a name="job-versioning-of-clobbered"></a>
-### versioning of clobbered files
-
-* when files are generated or removed, relocate to a timestamped location
-  - a file `/path/to/file.txt` is relocated to `~/.wukong/backups/path/to/file.txt.wukong-20110102120011` where `20110102120011` is the [job timestamp](#file-naming)
-  - accepts a `max_size` param
-  - raises if it can't write to directory -- must explicitly say `--safe_file_ops=false`
-
-<a name="job-running"></a>
-### running
-
-* `clobber` -- run, but clear all dependencies
-* `undo`    -- 
-* `clean`   -- 
-
-
-### Utility and Filesystem tasks
-
-The primitives correspond heavily with Rake and Chef. However, they extend them in many ways, don't cover all their functionality in many ways, and incompatible in several ways.
-
-### Configuration
-
-
-#### Commandline args
-
-* handled by configliere: `nukes launch --launch_code=GLG20`
-
-* TODO: configliere needs context-specific config vars, so I only get information about the `launch` action in the `nukes` job when I run `nukes launch --help`
-
-
-
-__________________________________________________________________________
-
-<a name="dataflows"></a>
-## Dataflows
-
-
-Data flows 
-
-* you can have a consumer connect to a provider, or vice versa
-  - producer binds to a port, consumers connect to it: pub/sub
-  - consumers open a port, producer connects to many: megaphone
+Wukong.processor(:starts_with) do
 
-* you can bring the provider on line first, and the consumers later, or vice versa.
-
-
-<a name="dataflow-syntax"></a>
-## Syntax 
-
-**note: this is a scratch pad; actual syntax evolving rapidly and currently looks not much like the following**
-   
-   read('/foo/bar')         # source( FileSource.new('/foo/bar') )
-   writes('/foo/bar')       # sink(   FileSink.new('/foo/bar') )
-
-   ... | file('/foo/bar')   # this we know is a source
-   file('/foo/bar') | ...   # this we know is a sink
-   file('/foo/bar')         # don't know; maybe we can guess later
-
-Here is an example Wukong script, `count_followers.rb`:
-
-    from :json
-    
-    mapper do |user|
-      year_month = Time.parse(user[:created_at]).strftime("%Y%M")
-      emit [ user[:followers_count], year_month ]
-    end   
-    
-    reducer do 
-      start{ @count = 0 }
-    
-      each do |followers_count, year_month|
-        @count += 1
-      end
-    
-      finally{ emit [*@group_key, @count] }
-    end
+  field :letter, String, :default => 'a'
+  
+  def process word
+    yield word if word =~ Regexp.new("^#{letter}", true)
+  end
+end
+```
+
+Let's start by running the `tokenizer`.  We've defined two processors
+in the file `processors.rb` and neither one is named `processors` so
+we have to tell `wu-local` the name of the processor we want to run
+explicitly.
+
+```
+$ cat novel.txt | wu-local processors.rb --run=tokenizer
+It
+was
+the
+best
+of
+times,
+...
+```
+
+You can combine the output of one processor with another right in the
+shell.  Let's add the `starts_with` filter and also pass in the
+*field* `letter`, defined in that processor:
+
+```
+$ cat novel.txt | wu-local processors.rb --run=tokenizer | wu-local processors.rb --run=starts_with --letter=t
+the
+times
+the
+times
+...
+```
+
+Wanting to match on a regular expression is such a common task that
+Wukong has a built-in "widget" called `regexp` that you can use
+directly:
+
+```
+$ cat novel.txt | wu-local processors.rb --run=tokenizer | wu-local regexp --match='^t'
+```
+
+There are many more simple <a href="#widgets">widgets</a> like these.
+
+<a name="flows"></a>
+## Combining Processors into Dataflows
+
+Combining processors which each do one thing well together in a chain
+is mimicing the tried and true UNIX pipeline.  Wukong lets you define
+these pipelines more formally as a dataflow.  Here's the dataflow for
+the last example:
+
+```
+# in find_t_words.rb
+Wukong.dataflow(:find_t_words) do
+  tokenizer > regexp(match: /^t/)
+end
+```
+
+The DSL Wukong provides for combining processors is designed to
+similar to the processing of developing them on the command line.  You
+can run this dataflow directly
+
+```
+$ cat novel.txt | wu-local find_t_words.rb
+the
+times
+the
+times
+...
+```
+
+and it works exactly like before.
+
+<a name="serialization></a>
+## Serialization
+
+The process method for a Processor must accept a String argument and
+yield a String argument (or something that will `to_s` appropriately).
+
+**Coming Soon:** The ability to define `consumes` and `emits` to
+  automatically handle serialization and deserialization.
+
+<a name="widgets></a>
+## Widgets
+
+Wukong has a number of built-in widgets that are useful for
+scaffolding your dataflows.
+
+### Serializers
+
+Serializers are widgets which don't change the semantic meaning of a
+record, merely its representation.  Here's a list:
+
+* `to_json`, `from_json` for turning records into JSON or parsing JSON into records
+* `to_tsv`, `from_tsv` for turning Array records into TSV or parsing TSV into Array records
+* `pretty` for pretty printing JSON inputs
+
+When you're writing processors that are capable of running in
+isolation you'll want to ensure that you deserialize and serialize
+records on the way in and out, like this
+
+```ruby
+Wukong.processor(:on_my_own) do
+  def process json
+    obj = MultiJson.load(json)
     
-You can run this from the commandline:
-
-    wukong count_followers.rb users.json followers_histogram.tsv
+    # do something with obj...
     
-It will run in local mode, effectively doing
-
-    cat users.json | {the map block} | sort | {the reduce block} > followers_histogram.tsv
-
-You can instead run it in Hadoop mode, and it will launch the job across a distributed Hadoop cluster
-
-    wukong --run=hadoop count_followers.rb users.json followers_histogram.tsv
-
-<a name="formatters"></a>
-#### Data Formats (Serialization / Deserialization)
-
-* tsv/csv
-* json
-* xml
-* avro
-* apache_log
-* flat 
-* regexp 
-* [Tagged Netstrings](http://tnetstrings.org/)
-* [ZeroMQ Property Language](http://rfc.zeromq.org/spec:4)
-
-* gz/bz2/zip/snappy 
-
-<a name="data-packets"></a>
-#### Data Packets
-
-Data consists of
-
-- record
-- schema
-- metadata
-
-## Delivery Guarantees
-
-Most messaging systems keep metadata about what messages have been consumed on the broker. That is, as a message is handed out to a consumer, the broker records that fact locally. This is a fairly intuitive choice, and indeed for a single machine server it is not clear where else it could go. Since the data structure used for storage in many messaging systems scale poorly, this is also a pragmatic choice--since the broker knows what is consumed it can immediately delete it, keeping the data size small.
-
-What is perhaps not obvious, is that getting the broker and consumer to come into agreement about what has been consumed is not a trivial problem. If the broker records a message as consumed immediately every time it is handed out over the network, then if the consumer fails to process the message (say because it crashes or the request times out or whatever) then that message will be lost. To solve this problem, many messaging systems add an acknowledgement feature which means that messages are only marked as sent not consumed when they are sent; the broker waits for a specific acknowledgement from the consumer to record the message as consumed. This strategy fixes the problem of losing messages, but creates new problems. First of all, if the consumer processes the message but fails before it can send an acknowledgement then the message will be consumed twice. The second problem is around performance, now the broker must keep multiple states about every single message (first to lock it so it is not given out a second time, and then to mark it as permanently consumed so that it can be removed). Tricky problems must be dealt with, like what to do with messages that are sent but never acknowledged.
-
-So clearly there are multiple possible message delivery guarantees that could be provided:
-
-* At most once—this handles the first case described. Messages are immediately marked as consumed, so they can't be given out twice, but many failure scenarios may lead to losing messages.
-* At least once—this is the second case where we guarantee each message will be delivered at least once, but in failure cases may be delivered twice.
-* Exactly once—this is what people actually want, each message is delivered once and only once.
-
-
-__________________________________________________________________________
-
-<a name="design-questions"></a>
-## Design Questions
-
-* **filename helpers**: 
-  - `':data_dir:/this/that/:script:-:user:-:timestamp:.:ext:'`?
-  - `path_to(:data_dir, 'this/that', "???")`?
-
-* `class Wukong::Foo::Base` vs `class Wukong::Foo` 
-  - the latter is more natural, and still allows 
-  - I'd like 
-
-
-
-__________________________________________________________________________
-__________________________________________________________________________
-__________________________________________________________________________
-
-
-<a name="references"></a>
-## References
-
-<a name="refs-workflow"></a>
-### Workflow
-
-* **Rake**
-
-  - [Rake Docs](http://rdoc.info/gems/rake/file/README.rdoc)
-  - [Rake Tutorial](http://jasonseifer.com/2010/04/06/rake-tutorial) by Jason Seifer -- 2010, with a good overview of why Rake is useful
-  - [Rake Tutorial](http://martinfowler.com/articles/rake.html) by Martin Fowler -- from 2005, so may lack some modernities
-  - [Rake Tutorial](http://onestepback.org/index.cgi/Tech/Rake/Tutorial/RakeTutorialRules.red) -- from 2005, so may lack some modernities
-  
-* **Rake Examples**
-
-  - [resque's redis.rake](https://github.com/defunkt/resque/blob/master/lib/tasks/redis.rake) and [resque/tasks](https://github.com/defunkt/resque/blob/master/lib/resque/tasks.rb)
-  - [rails' Rails Ties](https://github.com/rails/rails/tree/master/railties/lib/rails/tasks)
-  
-* **Thor**
-
-  - [Thor Wiki](https://github.com/wycats/thor/wiki)
-  - 
-  
-* **Chef**
-
-  - [Chef Wiki](http://wiki.opscode.com/display/chef/Home)
-  - specifically, [Chef Resources](http://wiki.opscode.com/display/chef/Resources)
-
-* **Other**
-
-  - [**Gradle**](http://gradle.org/) -- a modern take on `ant` + `maven`. The [Gradle overview](http://gradle.org/overview) states its case.
-
-<a name="refs-dataflow"></a>
-### Dataflow
-
-* **Esper**
-
-  - Must read: [StreamSQL Event Processing with Esper](http://www.igvita.com/2011/05/27/streamsql-event-processing-with-esper/)
-  - [Esper docs](http://esper.codehaus.org/esper-4.5.0/doc/reference/en/html_single/index.html#epl_clauses)
-  - [Esper EPL Reference](http://esper.codehaus.org/esper-4.5.0/doc/reference/en/html_single/index.html#epl_clauses)
-
-* **Storm**
-
-  - [A Storm is coming: more details and plans for release](http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html)
-  - [Storm: distributed and fault-tolerant realtime computation](http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolerant-realtime-computation) -- slideshare presentation
-  - [Storm: the Hadoop of Realtime Processing](http://tech.backtype.com/preview-of-storm-the-hadoop-of-realtime-proce)
-
-* **Kafka**: LinkedIn's high-throughput messaging queue
-
-  - [Kafka's Design: Why we built this](http://incubator.apache.org/kafka/design.html) 
-
-* **ZeroMQ**: tcp sockets like you think they should work
-
-  - [ZeroMQ: A Modern & Fast Networking Stack](http://www.igvita.com/2010/09/03/zeromq-modern-fast-networking-stack/)
-  - [ZeroMQ Guide](http://zguide.zeromq.org/page:all)
-  - [ZeroMQ: An Introduction](http://nichol.as/zeromq-an-introduction)
-  - [Routing with Ruby & ZeroMQ Devices](http://www.igvita.com/2010/11/17/routing-with-ruby-zeromq-devices/)
-  - [Ruby bindings for ZeroMQ](http://zeromq.github.com/rbzmq/) and the [Ruby-FFI bindings](http://www.zeromq.org/bindings:ruby-ffi)
-  - [Learn ruby ZeroMQ](https://github.com/andrewvc/learn-ruby-zeromq) by @andrewvc
-  
-* **Other**
-
-  - [Infopipes: An abstraction for multimedia streamin](http://web.cecs.pdx.edu/~black/publications/Mms062%203rd%20try.pdf) Black et al 2002
-  - [Yahoo Pipes](http://pipes.yahoo.com/pipes/) 
-  - [Yahoo Pipes wikipedia page](http://en.wikipedia.org/wiki/Yahoo_Pipes)
-  - [Streambase](http://www.streambase.com/products/streambasecep/faqs/) -- Why is is so goddamn hard to find out anything real about a project once it gets an enterprise version? Seriously, the consistent fundamental brokenness of enterprise product is astonishing. It's like they take inspiration from shitty major-label band websites but layer a whiteout of [web jargon bullshit](http://www.dack.com/web/bullshit.html) in place of inessential flash animation. Anyway I think Streambase is kinda similar but who the hell can tell.
-  - [Scribe](http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/)
-  - [Splunk Case Study](http://www.igvita.com/2008/10/22/distributed-logging-syslog-ng-splunk/)
-
-<a name="refs-dataflow"></a>
-### Messaging Queue
-
-- [DripDrop](https://github.com/andrewvc/dripdrop) - a message passing library with a unified API abstracting HTTP, zeroMQ and websockets.
-
-
-<a name="refs-dataflow"></a>
-### Data Processing
-
-* **Hadoop**
-
-  - [Hadoop]()
- 
-  
-* **Spark/Mesos**
-
-  - [Mesos](http://www.mesosproject.org/)
+    yield MultiJson.dump(obj)
+  end
+end
+```
+
+For processors which will only run inside a data flow, you can
+optimize by not doing any (de)serialization until except at the very
+beginning and at the end
+
+```ruby
+Wukong.dataflow(:complicated) do
+  from_json > proc_1 > proc_2 > proc_3 ... proc_n > to_json
+end
+```
+
+in this approach, no serialization will be done between processors.
+
+### General Purpose
+
+There are several general purpose processors which implement common
+patterns on input and output data.  These are most useful within the
+context of a dataflow definition.
+
+* `null` does what you think it doesn't
+* `map` perform some block on each
+* `flatten` flatten the input array
+* `filter`, `select`, `reject` only let certain records through based on a block
+* `regexp`, `not_regexp` only pass records matching (or not matching) a regular expression
+* `limit` only let some number of records pass
+* `logger` send events to the local log stream
+* `extract` extract some part of each input event
+
+Some of these widgets can be used directly, perhaps with some
+arguments
+
+```ruby
+Wukong.processor(:log_everything) do
+  proc_1 > proc_2 > ... > logger
+end
+
+Wukong.processor(:log_everything_important) do
+  proc_1 > proc_2 > ... > regexp(match: /important/i) > logger
+end
+```
+
+Other widgets require a block to define their action:
+
+```ruby
+Wukong.processor(:log_everything_important) do
+  parser > select { |record| record.priority =~ /important/i } > logger
+end
+```
+
+### Reducers
+
+There are a selection of widgets that do aggregative operations like
+counting, sorting, and summing.
+
+* `count` emits a final count of all input records
+* `sort` can sort input streams
+* `group` will group records by some extracting part and give a count of each group's size
+* `moments` will emit more complicated statistics (mean, std. dev.) on the group given some other value to measure
+
+Here's an example of sorting data right on the command line
+
+```
+$ head tokens.txt | wu-local sort
+abhor
+abide
+abide
+able
+able
+able
+about
+...
+```
+
+Try adding group:
+
+```
+$ head tokens.txt | wu-local sort | wu-local group
+{:group=>"abhor", :count=>1}
+{:group=>"abide", :count=>2}
+{:group=>"able", :count=>3}
+{:group=>"about", :count=>3}
+{:group=>"above", :count=>1}
+...
+```
+
+You can also use these within a more complicated dataflow:
+
+```ruby
+Wukong.dataflow(:word_count) do
+  tokenize > remove_stopwords > sort > group
+end
+```