wukong 3.0.0.pre2 → 3.0.0.pre3

data/Gemfile

@@ -1,3 +1,16 @@
 source :rubygems
 
 gemspec
+
+group :development do
+  gem 'rake',        '>= 0.9'
+  gem 'rspec',       '>= 2.8'
+  gem 'guard',       '>= 1.0'
+  gem 'guard-rspec', '>= 0.6'
+  gem 'simplecov',   '>= 0.5'
+  gem 'pry'
+  gem 'yard'
+  gem 'redcarpet'
+  gem 'addressable'
+  gem 'htmlentities'
+end

data/README.md

@@ -131,7 +131,7 @@ the last example:
 ```
 # in find_t_words.rb
 Wukong.dataflow(:find_t_words) do
-  tokenizer > regexp(match: /^t/)
+  tokenizer | regexp(match: /^t/)
 end
 ```
 
@@ -196,7 +196,7 @@ beginning and at the end
 
 ```ruby
 Wukong.dataflow(:complicated) do
-  from_json > proc_1 > proc_2 > proc_3 ... proc_n > to_json
+  from_json | proc_1 | proc_2 | proc_3 ... proc_n | to_json
 end
 ```
 
@@ -222,11 +222,11 @@ arguments
 
 ```ruby
 Wukong.processor(:log_everything) do
-  proc_1 > proc_2 > ... > logger
+  proc_1 | proc_2 | ... | logger
 end
 
 Wukong.processor(:log_everything_important) do
-  proc_1 > proc_2 > ... > regexp(match: /important/i) > logger
+  proc_1 | proc_2 | ... | regexp(match: /important/i) | logger
 end
 ```
 
@@ -234,7 +234,7 @@ Other widgets require a block to define their action:
 
 ```ruby
 Wukong.processor(:log_everything_important) do
-  parser > select { |record| record.priority =~ /important/i } > logger
+  parser | select { |record| record.priority =~ /important/i } | logger
 end
 ```
 
@@ -278,6 +278,182 @@ You can also use these within a more complicated dataflow:
 
 ```ruby
 Wukong.dataflow(:word_count) do
-  tokenize > remove_stopwords > sort > group
+  tokenize | remove_stopwords | sort | group
 end
 ```
+
+## Testing
+
+Wukong comes with several helpers to make writing specs using
+[RSpec](http://rspec.info/) easier.
+
+The only method that you need to test in a Processor is the `process`
+method.  The rest of the processor's methods and functionality are
+provided by Wukong and are already tested.
+
+You may want to test this process method in two ways:
+
+* unit tests of the class itself in various contexts
+* integration tests of running the class with the `wu-local` (or other) command-line runner
+
+### Unit Tests
+
+Let's start with a simple processor
+
+```ruby
+# in tokenizer.rb
+Wukong.processor(:tokenizer) do
+  def process text
+    text.downcase.gsub(/[^\s\w]/,'').split.each do |token|
+      yield token
+    end
+  end
+end
+```
+
+You could test this processor directly:
+
+```ruby
+# in spec/tokenizer_spec.rb
+require 'spec_helper'
+describe :tokenizer do
+  subject { Wukong::Processor::Tokenizer.new }
+  before  { subject.setup                    }
+  after   { subject.finalize ; subject.stop  }
+  it "correctly counts tokens" do
+    expect { |b| subject.process("Hi there, Wukong!", &b) }.to yield_successive_args('hi', 'there', 'wukong')
+  end
+end
+```
+
+but having to handle the yield from the block yourself can lead to
+verbose and unreadable tests.  Wukong defines some helpers for this
+case.  Require and include them first in your `spec_helper.rb`:
+
+```ruby
+# spec/spec_helper.rb
+require 'wukong'
+require 'wukong/spec_helpers'
+RSpec.configure do |config|
+  config.include(Wukong::SpecHelpers)
+end
+```
+
+and then use them in your test
+
+```ruby
+# in spec/tokenizer_spec.rb
+require 'spec_helper'
+describe :tokenizer do
+  it_behaves_like 'a processor', :named => :tokenizer
+  it "emits the correct number of tokens" do
+    processor.given("Hi there.\nMy name is Wukong!").should emit(6).records
+  end
+  it "eliminates all punctuation" do
+    processor.given("Never!").output.first.should_not include(',')
+  end
+  it "downcases all input text" do
+    processor.given("Whatever").output.first.should match(/^w/)
+  end
+end
+```
+
+Let's look at each kind of helper:
+
+* The `a processor` shared example (invoked with RSpec's
+  `it_behaves_like` helper) adds some tests that ensure that the
+  processor conforms to the API of a Wukong::Processor.
+
+* The `processor` method instantiates a processor very similarly to
+  the way `wu-local` instantiates one on the command-line.  It accepts
+  a (registered) processor name and options and creates a new
+  processor.  If no name is given, the argument of the enclosing
+  `describe` or `context` block is used.  The object returned by
+  `processor` is the Wukong::Processor you're testing so you can
+  directly declare introspect on it or declare expectations about its
+  behavior.
+
+* The `given` method (and other helpers like `given_json`,
+  `given_tsv`, &c.) is added to the Processor class when
+  Wukong::SpecHelpers is required. It's a way of lazily feeding
+  records to a processor, without having to go through the `process`
+  method directly and having to handle the block or the processor's
+  lifecycle as in the prior example.
+
+* The `output` and `emit` matchers will `process` all previously
+  `given` records when they are called. This lets you separate
+  instantiation, input, expectations, and output. Here's a more
+  complicated example:
+
+The same helpers can be used to test dataflows as well as
+processors. For complete details, see documentation for the
+Wukong::SpecHelpers module.
+
+### Integration Tests
+
+Sometimes unit tests aren't enough and you need to test your
+processors or flows as they will be run in production using
+`wu-local`.
+
+For these use cases, Wukong provides some integration helpers that
+make testing command line processes easier.
+
+```ruby
+# spec/integration/tokenizer_spec.rb
+context "running the tokenizer with wu-local" do
+  subject { command("wu-local tokenizer") < "hi there" }
+  it { should exit_with(0)               }
+  it { should have_stdout("hi", "there") }
+end
+
+context "interpreting its arguments" do
+  context "with a valid --match argument" do
+    subject { command("wu-local tokenizer --match='^hi'") < "hi there" }
+	it      { should     exit_with(0) }
+	it      { should     have_stdout("hi")    }
+	it      { should_not have_stdout("there") }
+  end
+  context "with a malformed --match argument" do
+    # invalid b/c the regexp is broken...
+    subject { command("wu-local tokenizer --match='^[h'") < "hi there" }
+	it      { should exit_with(:non_zero)   }
+	it      { should have_stderr(/invalid/) }
+  end
+end
+```
+
+Let's go through the helpers:
+
+* The `command` helper creates a wrapper around a command-line that will be launched.  The command's environment and working directory will be taken from the current values of `ENV` and `Dir.pwd`, unless
+
+  * The `in` or `using` arguments are chained with `command` to specify the working directory and environment:
+
+  ```ruby
+  command("some-command with --args").in("/my/working/directory").using("THIS" => "ENV_HASH", "WILL_BE" => "MERGED_OVER_EXISTING_ENV")
+  ```
+
+  * The scope in which the `command` helper is called defines methods `integration_cwd` and `integration_env`.  This can be done through including a module in your `spec_helper.rb`:
+
+  ```ruby
+  # in spec/support/integration_helper.rb
+  module IntegrationHelper
+    def integration_cwd
+	  "/my/working/directory"
+	end
+	def integration_env
+	  { "THIS" => "ENV_HASH", "WILL_BE" => "MERGED_OVER_EXISTING_ENV" }
+	end
+  end
+
+  # in spec/spec_helper.rb
+  require_relative("support/integration_helper")
+  RSpec.configure do |config|
+    config.include(IntegrationHelper)
+  end
+  ```  
+
+* The `command` helper can accept input with the `<` method.  Input can be either a String or an Array of strings.  It will be passed to the command over STDIN.
+
+* The `have_stdout` and `have_stderr` matchers let you test the STDOUT or STDERR of the command for particular strings or regular expressions.
+
+* The `exit_with` matcher lets you test the exit code of the command.  You can pass the symbol `:non_zero` to set the expectation of _any_ non-zero exit code.

data/bin/wu-local

@@ -42,8 +42,8 @@ again test locally:
   clever
 EOF
 
-settings.define :run, description: "Name of the processor or dataflow to use.  Defaults to basename of the given path.", flag: 'r'
-
+settings.define :run,        description: "Name of the processor or dataflow to use.  Defaults to basename of the given path.", flag: 'r'
+# settings.define :tcp_server, description: "Run locally as a tcp server on a specified port", default: false,                    flag: 't'
 require 'wukong/boot' ; Wukong.boot!(settings)
 
 thing = settings.rest.first
@@ -60,10 +60,18 @@ else
   settings.dump_help
   exit(2)
 end
-# p settings
+
+
+
 begin
-  Wukong::LocalDriver.run(processor.to_sym, settings)
-rescue Wukong::Error => e
+  # EM.run do
+  #   settings.tcp_server ? Wu::TCPServer.start(processor.to_sym, settings) : Wu::StdioServer.start(processor.to_sym, settings)
+  # end
+  StupidServer.new(processor.to_sym, settings).run!
+rescue Wu::Error => e
   $stderr.puts e.message
   exit(3)
 end
+
+# One day, it will be this easy...
+# Wukong::LocalRunner.run!

data/bin/wu-server

@@ -67,4 +67,4 @@ end
 EM::run {
   EM::start_server(Settings[:host], Settings[:port], WukongMachine)
   puts "Listening on #{Settings[:host]}:#{Settings[:port]}"
-}
+}

data/examples/Gemfile

@@ -11,7 +11,7 @@ gem "log4r"
 group :examples do
   gem "forgery"
   gem "nokogiri"
-  # gem "sanitize"
+  gem "sanitize"
   gem "addressable"
   gem "forgery"
   gem "crack"
@@ -28,6 +28,7 @@ group :development do
   gem "simplecov",          '>= 0.5'
   gem "pry"
   gem "ap"
+  gem "ruby-progressbar"
 end
 
 group :docs do

data/examples/basic/string_reverser.rb

@@ -0,0 +1,23 @@
+Wukong.processor(:string_reverser) do
+
+  def setup
+    log.info("Inside the setup method")
+    @count = 0
+    EM.add_periodic_timer(10){ notify('metrics', count: @count) }
+  end
+
+  def process(record) 
+    @count += 1
+    yield record.reverse
+    yield nil
+  end
+
+  def finalize
+    log.info("Finalizing flow")
+  end
+  
+  def stop
+    log.info("Inside the stop method")
+  end
+
+end

data/examples/deploy_pack/Gemfile

@@ -0,0 +1,7 @@
+# This is a minimal Gemfile that lets Wukong boot up (but not work).
+gem 'configliere'
+gem 'gorillib'
+gem 'multi_json'
+gem 'vayacondios-client'
+gem 'log4r'
+gem 'eventmachine'

data/examples/deploy_pack/README.md

@@ -0,0 +1,6 @@
+This directory contains the skeleton of a Wukong deploy pack.  It's
+used to test the behavior of Wukong's command line tools within such a
+context.
+
+The Gemfile here contains the minimal subset of gems that let Wukong
+boot.

data/examples/deploy_pack/app/processors/string_reverser.rb

@@ -0,0 +1,5 @@
+Wukong.processor(:string_reverser) do
+  def process string
+    yield string.reverse
+  end
+end

data/examples/deploy_pack/config/environment.rb

@@ -0,0 +1 @@
+require_relative("../app/processors/string_reverser.rb")

data/examples/dsl/dataflow/scraper_macro_flow.rb

@@ -0,0 +1,28 @@
+require 'wukong/widgets/sinks/hbase_record_sink.rb'
+
+Wukong.chain(:friend_graph) do
+  tail(:scrapables) do
+    directory   'scrapables/ids-%{t:ymd}.tsv'
+  end
+
+  requester = decorator('tw_requester.rb') do
+    input  :scrape_url,       Url
+    output :raw_json_request, JsonString
+    config do
+      define :request_types, :default => [:follower_ids, :friend_ids], :doc => 'which requests to make: follower_ids, user_timeline, etc'
+    end
+  end
+
+  retriable_requester = retriable do
+    with        :timeouts => [1,2,3]
+    on_failure  :sleep
+    guest       requester
+  end
+
+  tail(:scrapables)> retriable_requester > processor('tw_parse.rb') > hbase_record_sink
+end
+
+Wukong.processor(:tw_parse) do
+  def process
+  end
+end

data/examples/graph/implied_geolocation/README.md

@@ -0,0 +1,63 @@
+# Implied Geolocation
+
+* Some objects are explicitly geolocated: "Austin, Texas", "Cornell University", the "USS_Constitution".
+* Some objects are not only geolocated, they are 'places' -- present as well in the geonames dataset.
+
+The estimator is as follows:
+
+* a best-estimate longitude and latitude
+* the radius of uncertainty for the point
+* the likelihood the point is erroneous
+
+	 12000 krec articles
+	  7000 krec geonames
+	   400 krec dbpedia-geo_coordinates_en.json
+	    87 krec dbpedia-geonames_links.json
+	    
+
+
+### dispatch geolocation estimates along links
+
+* Send every neighbor your geoestimate
+
+accumulate all neighbors' geoestimates.
+
+
+In this drawing, the vertical bars show implied locations; six reasonably nearby each other and two with large error.
+
+          |      | |       |  ||               |          |
+      ----+------+-+-------+--++------- // ----+---- // --+-----
+
+But of course in some places I _know_ the location
+
+          |    X | |       |  ||               |          |
+      ----+----X-+-+-------+--++------- // ----+---- // --+-----
+               X
+                `-- actual location
+
+
+Why are the estimates spread from the actual?
+
+* intrinsic size of the actual: the graph neighbors of "Texas" are spread over a much larger area than the graph neighbors of "Yee-Haw Junction, FL".
+* strength of the relationship: for example, this naive model can't tell the difference between "X is located in Y" and "X borders Y"
+* errors in the relationship: the link might be irrelevant or not explanatory for any reason -- anything from "X has the same area as Virginia" to a hacked page.
+* multi-modal location: Davey Crockett (TODO: verify) was from XXX to XXX the representative of Tennesee (location #1) to the US Congress in Washington, DC (locaton #2). Upon losing re-election, he famously said "You can all go to hell, I am going to Texas"; he died during the battle of the Alamo. The most robust assignment of a geolocation to "Davey Crockett" would look something like the following cartoon:
+ 
+         ____          	 
+        /    \	      ------
+       /      \	     /      \	   +-+
+       |       |_____|       |____/   \
+       
+       Tennesee        Texas        DC
+
+
+So what we're going to do is track two separate types of error:
+
+* the likelihood the estimate is drawn from purely irrelevant points
+* assuming the estimates are relevant, the fuzziness of the implied geolocation.
+
+
+
+* ?? only use estimates with some strength ??
+* For all known points, the number of neighbors that are irrelevant
+