wukong 3.0.1 → 4.0.0

data/.gitignore

@@ -57,3 +57,4 @@ away
 .rbx
 Gemfile.lock
 Backup*of*.numbers
+*.gem

data/Gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source 'https://rubygems.org'
 
 gemspec
 

data/README.md

@@ -5,8 +5,8 @@ at any scale.
 
 The core concept in Wukong is a **Processor**.  Wukong processors are
 simple Ruby classes that do one thing and do it well.  This codebase
-implements processors and other core Wukong classes and provides a
-tool, `wu-local`, to run and combine processors on the command-line.
+implements processors and other core Wukong classes and provides a way
+to run and combine processors on the command-line.
 
 Wukong's larger theme is *powerful black boxes, beautiful glue*. The
 Wukong ecosystem consists of other tools which run Wukong processors
@@ -293,7 +293,7 @@ $ cat input.json
 you can feed it directly to a processor
 
 ```
-$ cat input.json | wu-local --from=json extractor
+$ cat input.json | wu-local --from=json extractor.rb
 John
 Sally
 ...
@@ -302,9 +302,10 @@ Sally
 Other processors really like Arrays:
 
 ```ruby
+# in summer.rb
 Wukong.processor(:summer) do
   def process values
-    yield values.map(&:to_f).inject(0.0) { |sum, summand| sum += summand }
+    yield values.map(&:to_f).inject(&:+)
   end
 end
 ```
@@ -316,7 +317,7 @@ $ cat data.tsv
 4	5	6
 7	8	9
 ...
-$ cat data.tsv | wu-local --from=tsv summer
+$ cat data.tsv | wu-local --from=tsv summer.rb
 6
 15
 24
@@ -326,13 +327,13 @@ $ cat data.tsv | wu-local --from=tsv summer
 but you can just as easily use the same code with CSV data
 
 ```
-$ cat data.tsv | wu-local --from=csv summer
+$ cat data.tsv | wu-local --from=csv summer.rb
 ```
 
 or a more general delimited format.
 
 ```
-$ cat data.tsv | wu-local --from=delimited --delimiter='--' summer
+$ cat data.tsv | wu-local --from=delimited --delimiter='--' summer.rb
 ```
 
 #### Recordizing data structures into domain models
@@ -357,7 +358,7 @@ combination with the deserializing features above, turn input text
 into instances of Person:
 
 ```
-$ cat input.json | wu-local --consumes=Person --from=json contact_validator
+$ cat input.json | wu-local --consumes=Person --from=json contact_validator.rb
 #<Person:0x000000020e6120>
 #<Person:0x000000020e6120>
 #<Person:0x000000020e6120>
@@ -367,7 +368,7 @@ $ cat input.json | wu-local --consumes=Person --from=json contact_validator
 processor:
 
 ```
-$ cat input.json | wu-local --consumes=Person --from=json contact_validator --to=json
+$ cat input.json | wu-local --consumes=Person --from=json contact_validator.rb --to=json
 {"first_name": "John", "last_name":, "Smith", "valid": "true"}
 {"first_name": "Sally", "last_name":, "Johnson", "valid": "true"}
 ...
@@ -441,20 +442,20 @@ The default log level is DEBUG.
 
 ```
 $ echo something | wu-local logs.rb
-DEBUG 2013-01-11 23:40:56 [Logs                ] -- event
-INFO 2013-01-11 23:40:56 [Logs                ] -- event
-WARN 2013-01-11 23:40:56 [Logs                ] -- event
-ERROR 2013-01-11 23:40:56 [Logs                ] -- event
-FATAL 2013-01-11 23:40:56 [Logs                ] -- event
+DEBUG 2013-01-11 23:40:56 [Logs                ] -- something
+INFO 2013-01-11 23:40:56 [Logs                ] -- something
+WARN 2013-01-11 23:40:56 [Logs                ] -- something
+ERROR 2013-01-11 23:40:56 [Logs                ] -- something
+FATAL 2013-01-11 23:40:56 [Logs                ] -- something
 ```
 
 though you can set it to something else globally
 
 ```
 $ echo something | wu-local logs.rb --log.level=warn
-WARN 2013-01-11 23:40:56 [Logs                ] -- event
-ERROR 2013-01-11 23:40:56 [Logs                ] -- event
-FATAL 2013-01-11 23:40:56 [Logs                ] -- event
+WARN 2013-01-11 23:40:56 [Logs                ] -- something
+ERROR 2013-01-11 23:40:56 [Logs                ] -- something
+FATAL 2013-01-11 23:40:56 [Logs                ] -- something
 ```
 
 or on a per-class basis.
@@ -474,7 +475,6 @@ the command-line.  Use wu-local by passing it a processor and feeding
 
 Params:
    -r, --run=String             Name of the processor or dataflow to use. Defaults to basename of the given path.
-   -t, --tcp_port=Integer       Consume TCP requests on the given port instead of lines over STDIN
 ```
 
 You can generate custom help messages for your own processors.  Here's
@@ -540,22 +540,23 @@ Params:
        --mean=Float             The mean of the assumed distribution [Default: 0.0]
    -r, --run=String             Name of the processor or dataflow to use. Defaults to basename of the given path.
        --std_dev=Float          The standard deviation of the assumed distribution [Default: 1.0]
-   -t, --tcp_port=Integer       Consume TCP requests on the given port instead of lines over STDIN
 
 ```
 
 <a name="flows"></a>
 ## Combining Processors into Dataflows
 
-Combining processors which each do one thing well together in a chain
-is mimicing the tried and true UNIX pipeline.  Wukong lets you define
-these pipelines more formally as a dataflow.
+Wukong provides a DSL for combining processors together into
+dataflows.  This DSL is designed to make it easy to replicate the
+tried and true UNIX philosophy of building simple tools which do one
+thing well and then combining them together to create more complicated
+flows.
 
-Having written the `tokenizer` processor, we can use it in a dataflow
-along with the built-in `regexp` processor to replicate what we did in
-the last example:
+For example, having written the `tokenizer` processor, we can use it
+in a dataflow along with the built-in `regexp` processor to replicate
+what we did in the last example:
 
-```
+```ruby
 # in find_t_words.rb
 require_relative('processors')
 Wukong.dataflow(:find_t_words) do
@@ -563,9 +564,13 @@ Wukong.dataflow(:find_t_words) do
 end
 ```
 
-The DSL Wukong provides for combining processors is designed to
-similar to the processing of developing them on the command line.  You
-can run this dataflow directly
+The `|` operator connects the output of one processor (what it
+`yield`s) with the input of another (its `process` method).  In this
+example, every record emitted by `tokenizer` will be subsequently
+processed by `regexp`.
+
+You can run this dataflow directly (mimicing what we did above with
+single processors chained together on the command-line):
 
 ```
 $ cat novel.txt | wu-local find_t_words.rb
@@ -576,8 +581,85 @@ times
 ...
 ```
 
-and it works exactly like manually chaining the two processors
-together.
+### More complicated dataflow topologies
+
+The Wukong dataflow DSL allows for more complicated topologies than
+just chaining processors together in a linear pipeline.
+
+The `|` operator, used in the above examples to connect two processors
+together into a chain, can also be used to connect a single processor
+to *multiple* processors, creating a branch-point in the dataflow.
+Each branch of the flow will receive the same records.
+
+This can be used to perform multiple actions with the same record, as
+in the following example:
+
+```ruby
+# in book_reviews.rb
+Wukong.dataflow(:complicated) do
+  from_json | recordize(model: BookReview) | 
+  [
+    map(&:author) | do_author_stuff | ... | to_json,
+	map(&:book)   | do_book_stuff   | ... | to_json,
+  ]
+end
+```
+
+Each `BookReview` record yielded by the `recordize` processor will be
+passed to both subsequent branches of the flow, with each branch doing
+a different kind of processing.  Output records from both branches
+(which are here turned `to_json` first) will be interspersed in the
+final output when run.
+
+A processor like `select`, which filters its inputs, can be used to
+split a flow into records of two types:
+
+```ruby
+# in complicated.rb
+Wukong.dataflow(:complicated) do
+  from_json | parser | 
+  [
+    select(&:valid?)   | further_processing | ... | to_json,
+	select(&:invalid?) | track_errors | null
+  ]
+end
+```
+
+Here, only records which respond true to the method `valid?` will pass
+through the first flow (applying `further_processing` and so on) while
+only records which respond true to `invalid?` will pass through the
+second flow (with `track_errors`).  The `null` processor at the end of
+this second branch ensures that only records from the first branch
+will be emitted in the final output.
+
+Flows can be split over and over again, allowing for rich semantics
+when processing an input source:
+
+```ruby
+# in many_splits.rb
+Wukong.dataflow(:many_splits) do
+  from_json | parser | recordize(model: BookReview) |
+  [ 
+    map(&:author) | ... | to_json,
+	map(&:publisher) |
+	[
+	  select(&:domestic?) | ... | to_json,
+	  select(&:international?) | 
+	  [
+	    select(&:north_american?) | ... | 
+		[
+		  select(&:american?) | ... | to_json,
+		  select(&:canadian?) | ... | to_json,
+		  select(&:mexican?)  | ... | to_json,
+		],
+		select(&:asian?)    | ... | to_json,
+		select(&:european?) | ... | to_json,
+	  ],
+	],
+	map(&:title) | ... | to_json
+  ]
+end
+```
 
 <a name="serialization></a>
 ## Serialization
@@ -612,19 +694,9 @@ record, merely its representation.  Here's a list:
 
 When you're writing processors that are capable of running in
 isolation you'll want to ensure that you deserialize and serialize
-records on the way in and out, like this
-
-```ruby
-Wukong.processor(:on_my_own) do
-  def process json
-    obj = MultiJson.load(json)
-    
-    # do something with obj...
-    
-    yield MultiJson.dump(obj)
-  end
-end
-```
+records on the way in and out, using the serialization/deserialization
+options `--to` and `--from` on the command-line, as <a
+href="#serialization">defined above</a>.
 
 For processors which will only run inside a data flow, you can
 optimize by not doing any (de)serialization until except at the very
@@ -636,7 +708,12 @@ Wukong.dataflow(:complicated) do
 end
 ```
 
-in this approach, no serialization will be done between processors.
+in this approach, no serialization will be done between processors,
+only at the beginning and end.
+
+(This is actually the implementation behind the serialization options
+themselves -- they dynamically prepend/append the appropriate
+deserializers/serializers.)
 
 ### General Purpose
 
@@ -718,6 +795,137 @@ Wukong.dataflow(:word_count) do
 end
 ```
 
+## Commands
+
+Wukong comes with a few commands built-in.
+
+### wu-local
+
+You've seen one already, `wu-local`, in many of the examples above.
+`wu-local` is used to model dataflows locally, using `STDIN` and
+`STDOUT` for input and output.
+
+`wu-local` is a "core" Wukong command in the sense that more
+complicated commands like `wu-hadoop` and `wu-storm`, implemented by
+Wukong plugins, ultimately invoke some `wu-local` process.
+
+### wu-source
+
+Wukong also comes with another basic command `wu-source`.  This
+command works very similarly to `wu-local` except that it doesn't read
+any input from `STDIN`.  Instead it generates its *own* input records
+in an easy to configure, periodic way.  It thus acts as a *source* of
+data for other processes in a UNIX pipeline.
+
+Here's an example using the `identity` processor which will have the
+effect of printing to `STDOUT` the exact input received:
+
+```
+$ wu-source identity
+1
+2
+3
+...
+```
+
+From this example it's clear that the records produced by `wu-source`
+are consecutive integers starting at 1 and that they are produced at a
+rate of one record per second.
+
+`wu-source` can thus be used to turn any processor (or dataflow) into
+a source of data:
+
+```ruby
+# in random_numbers.rb
+Wukong.processor(:random_numbers) do
+  def process index
+    yield rand() * index.to_i
+  end
+end
+```
+
+Run `random_numbers` like this:
+
+```
+$ wu-source random_numbers.rb
+0.7671364694830113
+0.5958089791553307
+1.8284806932633886
+3.707189931235327
+4.106618048255548
+...
+```
+
+Which produces random numbers with an ever greater ceiling.
+
+You can also completely ignore the input record from `wu-source` in
+your processor:
+
+```ruby
+# in generator.rb
+Wukong.processor(:generator) do
+  def process _
+    yield new_record
+  end
+  def new_record
+    MyRecord.new(...)
+  end
+end
+```
+
+which can produce `MyRecord` instances as it's driven by `wu-source`.
+
+It's easy to generate several thousand events per second using
+`wu-source` this way:
+
+```
+$ wu-source generator.rb --per_sec=2000
+```
+
+or use the `--period` (which is the inverse of `--per_sec`) to spit
+out records at a regular interval (every 5 minutes in this example):
+
+```
+$ wu-source generator.rb --period=300
+```
+
+`wu-source` can naturally combine with other dataflows or programs you
+might write:
+
+```
+$ wu-source generator.rb --per_sec=200 | wu-local my_flow
+```
+### wu
+
+The `wu` command is a convenience command useful when using any of the
+other `wu-` commands in the context of a Ruby project with a
+[`Gemfile`](http://bundler.io/v1.3/gemfile.html).
+
+Instead of typing
+
+```
+$ bundle exec wu-local my_flow --option=value ...
+```
+
+which would run `wu-local` using the exact version of `wukong` (and
+any other dependencies) as declared in your project's `Gemfile` and
+`Gemfile.lock`, the `wu` command lets you type
+
+```
+$ wu local my_flow --option=value ...
+```
+
+essentially adding the `bundle exec` prefix and munging `wu local` to
+`wu-local` for you.  This can be very helpful when doing lots of work
+with Wukong.
+
+**Note:** If `bundle exec wu-whatever` works in your project but `wu
+  whatever` fails it is probably because Bundler is resolving `wu-`
+  commands to some installation that is not on your `$PATH` (often the
+  case if you ran `bundle install --standalone`).  Ensure that the
+  `wukong` gem is installed on your system and that it's binaries are
+  your `$PATH` to use the `wu` command.
+
 ## Testing
 
 Wukong comes with several helpers to make writing specs using

data/bin/wu

@@ -0,0 +1,34 @@
+#!/usr/bin/env ruby
+require 'shellwords'
+now=Time.now.strftime("%Y-%m-%d %H:%M:%S")
+if ARGV.empty?
+    abort "ERROR #{now} [wu                  ] -- Must provide a Wukong command to run. Try the --help option."
+else
+    if ARGV.size == 1 && ARGV.first == '--help'
+	abort <<EOF
+usage: wu COMMAND [OPTIONS] [ARG] ...
+
+wu is a wrapper for easy use of Wukong's command-line tools.  It takes
+your arguments, constructs the name of the proper wu-tool to call, and
+prepends a call to bundle exec.
+
+  $ wu local ...
+
+is equivalent to
+
+  $ bundle exec wu-local ...
+
+You can run any of the wu-tools this way:
+	
+  wu-local    wu-source
+  wu-hadoop   wu-storm
+  wu-deploy   wu-load
+EOF
+    else
+      if ARGV.first =~ /^-/
+        abort "ERROR ${now} [wu                  ] -- First argument must be the name of a wu tool to run, got <${1}>"
+      else
+        Kernel.exec "bundle exec wu-#{Shellwords.join(ARGV)}"
+      end
+    end
+end