wukong 0.1.1

data/doc/TODO.textile

@@ -0,0 +1,61 @@
+Utility
+
+* columnizing / reconstituting
+
+* Set up with JRuby
+* Allow for direct HDFS operations
+* Make the dfs commands slightly less stupid
+* add more standard options
+* Allow for combiners
+* JobStarter / JobSteps
+* might as well take dumbo's command line args
+
+BUGS:
+
+* Can't do multiple input files in local mode
+
+Patterns to implement:
+
+* Stats reducer (takes sum, avg, max, min, std.dev of a numeric field)
+* Make StructRecordizer work generically with other reducers (spec. AccumulatingReducer)
+
+Example graph scripts:
+
+* Multigraph
+* Pagerank 		(done)
+* Breadth-first search  
+* Triangle enumeration  
+* Clustering
+
+Example example scripts (from http://www.cloudera.com/resources/learning-mapreduce): 
+
+1. Find the [number of] hits by 5 minute timeslot for a website given its access logs.
+
+2. Find the pages with over 1 million hits in day for a website given its access logs.
+
+3. Find the pages that link to each page in a collection of webpages.
+
+4. Calculate the proportion of lines that match a given regular expression for a collection of documents.
+
+5. Sort tabular data by a primary and secondary column.
+
+6. Find the most popular pages for a website given its access logs.
+
+/can use
+
+
+---------------------------------------------------------------------------
+
+Add statistics helpers
+
+* including "running standard deviation":http://www.johndcook.com/standard_deviation.html
+
+
+---------------------------------------------------------------------------
+
+Make wutils: tsv-oriented implementations of the coreutils (eg uniq, sort, cut, nl, wc, split, ls, df and du) to instrinsically accept and emit tab-separated records.
+
+More example hadoop algorithms:
+Bigram counts: http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/exercises/bigrams.html
+* Inverted index construction: http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/exercises/indexer.html
+* Pagerank : http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/exercises/pagerank.html

data/doc/UsingWukong-part1-setup.textile

@@ -0,0 +1,2 @@
+h1. Using Wukong and Wuclan, Part 1 - Setup
+

data/doc/UsingWukong-part2-scraping.textile

@@ -0,0 +1,2 @@
+h1. Using Wukong and Wuclan, Part 2 - Scraping
+

data/doc/UsingWukong-part3-parsing.textile

@@ -0,0 +1,132 @@
+h1. Using Wukong and Wuclan, Part 3 - Parsing
+
+In part 2 we begain a scraper to trawl our desired part of the social web. Now
+we're ready to start using Wukong to process the files.
+
+Files come off the wire as
+
+    :url	:scraped_at	:response_code	:response_message	:contents
+    String      DateTime (flat) Integer          String                 String (JSON-formatted, tab&newline-munged)
+
+The contents field is a JSON-formatted mix of records:
+
+* TwitterFollowersRequest and TwitterFriendsRequest yield an @Array[Hash{user => raw_tweet}]@. We want to extract a stream of AFollowsB (with the request user as user_a for a friends request and user_b for a followers request) along with the included Tweet, TwitterUser, TwitterUserProfile and TwitterUserStyle records.
+* TwitterFavoritesRequest yields an array of @Array[Hash{tweet_hash => user_hash}]. We want to extract a stream of AFavoritesB along with the included Tweet, TwitterUser, TwitterUserProfile and TwitterUserStyle records
+* TwitterUser yields a single @user_hash@ making one each of TwitterUser, TwitterUserProfile and TwitterUserStyle.
+* UserTimelineRequest and PublicTimelineRequest yield an Array[Hash{tweet => user}]. We want to extract the included Tweet, TwitterUser, TwitterUserProfile and TwitterUserStyle records.
+* TwitterFollowerIdsRequest and TwitterFriendIdsRequest return an Array[user_ids] (each user_id is a simple Integer). We extract a series of AFollowsB (using the request's user_id as user_a_id or user_b_id)
+
+We want to split each API response into a stream of those TwitterUser, Tweet, etc. records.
+
+# Stream in each line (each line holds one request)
+# turn the line into the corresponding TwitterRequest
+# have the TwitterRequest parse its JSON contents and construct the TwitterUser, Tweet, etc.
+# seriealize those records back out as tab-separated lines suitable for further processing with Wukong
+
+h4. The basics of StructStreamer
+
+Wukong handles the first and last steps through its StructStreamer and the standard .to_flat method. So the actual structure is really simple:
+
+        #
+        # Instantiate each incoming request.
+        # Stream out the contained classes it generates.
+        #
+        class TwitterRequestParser < Wukong::Streamer::StructStreamer
+          def process request
+            request.parse do |obj|
+              yield obj
+            end
+          end
+        end
+
+        # This makes the script go.
+        Wukong::Script.new(TwitterRequestParser, nil).run
+
+In practice, all you need to know is that a StructStreamer gets a stream of objects to parse.  Here's an outline of its internals. The Wukong StructStreamer:
+
+# takes each flattened line:
+
+    "twitter_friends_request	http://.... 	20090701123456	...fields...	[{...}, {...}, ...json..., {...}]"
+
+# splits by tabs to create an array of fields
+
+    ["twitter_friends_request", "http://...", ... "[{...}, {...}, ...json..., {...}]"]
+
+# constructs the class name indicated in the first field,
+  using the values extracted from the remaining fields.
+
+    TwitterFriendsRequest.new "http://...", "20090701123456", ... "[{...}, {...}, ...json..., {...}]"
+
+The last (contents) field is still just a string: there's nothing special about it to Wukong.
+
+h4. Parsing
+
+Since each requests' contents are handled in a slightly (and brittle-ly) different manner, we just ask each request object to parse itself and feed out all the TwitterXXXX objects it generates.
+
+    class TwitterFollowersRequest
+        # ...
+
+        def parse &block
+          return unless healthy?
+          # for each raw user/tweet pair in the parsed JSON contents,
+          parsed_contents.each do |hsh|
+            json_obj = JsonUserWithTweet.new(hsh, 'scraped_at' => scraped_at)
+            next unless json_obj && json_obj.healthy?
+            # Extract user, tweet and relationship
+            yield AFollowsB.new(json_obj.user.id, self.twitter_user_id) if json_obj.user
+            json_obj.each(&block)
+          end
+        end
+        
+        # ...
+     end
+
+The TwitterXXXRequest objects consist of one or many hashes with (a raw user hash, and possibly its latest raw tweet hash) or (a raw tweet hash and its raw user hash).  The user hash might have only the fields for a TwitterPartialUser or it might have the fields for a full set of TwitterUser, TwitterUserProfile, TwitterUserStyle. Besides which, the fields themselves need some massaging to be compatible with Wukong and other tools in our Map/Reduce toolkit (details explained in a later section).
+
+The fiddly little details are handled by a JsonUserWithTweet or JsonTweetWithUser (as appropriate) adapter pattern:
+
+    class JsonUserTweetPair
+      def initialize raw, moreinfo
+        # clean up fields in entries (flatten date, true/false -> 1/0, etc)
+        fix_raw_user!
+        fix_raw_tweet!
+      end
+
+      # generate all the contained TwitterXXX objects
+      def each
+        # 
+      end
+
+      # create TwitterUser object from raw info
+      def user
+      end
+      # create Tweet object from raw tweet hash
+      def tweet
+      end
+      # ... and so forth
+    end
+      
+I'll ignore the gory details; view the source if you're interested.
+
+
+h4. Running the script
+
+Here, again, is the code (in full!) for the twitter_request_parser.rb script.
+
+    #
+    # Instantiate each incoming request.
+    # Stream out the contained classes it generates.
+    #
+    class TwitterRequestParser < Wukong::Streamer::StructStreamer
+      def process request
+        request.parse do |obj|
+          yield obj
+        end
+      end
+    end
+
+    # This makes the script go.
+    Wukong::Script.new(TwitterRequestParser, nil).run
+
+That last line is the runner: it makes this a Wukong script with a map phase only.  (We'll add in a reducer later on.)
+

data/doc/code/api_response_example.txt

@@ -0,0 +1,20 @@
+[
+  { // TwitterUser
+    "id":123456789,
+    // Basic fields
+      "screen_name":"nena", "protected":false, "created_at":"Thu Apr 23 02:00:00 +0000 2009",
+      "followers_count":0, "friends_count":1, "statuses_count":1, "favourites_count":0,
+    // TwitterUserProfile fields
+      "name":"nena", "url":null, "location":null, "description":null,"time_zone":null,"utc_offset":null,
+    // TwitterUserStyle
+      "profile_background_color":"9ae4e8", "profile_text_color":"000000", "profile_link_color":"0000ff", "profile_sidebar_border_color":"87bc44", "profile_sidebar_fill_color":"e0ff92", "profile_background_tile":false,
+      "profile_background_image_url":"http:\/\/static.twitter.com\/images\/themes\/theme1\/bg.gif"
+      "profile_image_url":"http:\/\/s3.amazonaws.com\/twitter_production\/profile_images\/123456789\/crane_normal.JPG",
+    // with enclosed Tweet
+    "status": {
+      "id":123456789,
+      // the twitter_user_id is implied
+      "created_at":"Thu Apr 23 02:00:00 +0000 2009", "favorited":false, "truncated":false, "source":"web", 
+      "in_reply_to_user_id":null, "in_reply_to_status_id":null, "in_reply_to_screen_name":null
+      "text":"My cat's breath smells like cat food." },
+  },

data/doc/code/parser_skeleton.rb

@@ -0,0 +1,38 @@
+# extract each record from request contents
+# and stream it to output
+class TwitterRequestParser < Wukong::Streamer::StructStreamer
+  def process request
+    request.parse do |obj|
+      yield obj
+    end
+  end
+end
+
+# Incoming Request:
+class TwitterFollowersRequest < Struct.new(
+    :url, :scraped_at, :response_code, :response_message, :moreinfo, :contents)
+  include Monkeyshines::ScrapeRequest
+end
+
+# Outgoing classes:
+class TwitterUser < TypedStruct.new( :id, :scraped_at, :screen_name, :protected, :created_at,
+    :followers_count, :friends_count, :statuses_count, :favourites_count )
+end
+class Tweet < TypedStruct.new(:id, :created_at, :twitter_user_id, :favorited, :truncated,
+    :text, :source, :in_reply_to_user_id, :in_reply_to_status_id, :in_reply_to_screen_name)
+end
+
+# Parsing code:
+TwitterFollowersRequest.class_eval do
+  include Monkeyshines::RawJsonContents
+  def parse &block
+    parsed_contents.each do |user_tweet_hash|
+      yield AFollowsB.new         user_tweet_hash["id"], self.moreinfo[:request_user_id]
+      yield TwitterUser.from_hash user_tweet_hash
+      yield Tweet.from_hash       user_tweet_hash
+    end
+  end
+end
+
+# This makes the script go.
+Wukong::Script.new(TwitterRequestParser, TwitterRequestUniqer).run

data/doc/hadoop-nfs.textile

@@ -0,0 +1,51 @@
+The "Cloudera Hadoop AMI Instances":http://www.cloudera.com/hadoop-ec2 for Amazon's EC2 compute cloud are the fastest, easiest way to get up and running with hadoop. Unfortunately, doing streaming scripts can be a pain, especially if you're doing iterative development.
+
+Installing NFS to share files along the cluster gives the following conveniences:
+
+* You don't have to bundle everything up with each run: any path in ~coder/ will refer back via NFS to the filesystem on master.
+ 
+* The user can now passwordless ssh among the nodes, since there's only one shared home directory and since we included the user's own public key in the authorized_keys2 file.  This lets you easily rsync files among the nodes.
+
+First, you need to take note of the _internal_ name for your master, perhaps something like @domU-xx-xx-xx-xx-xx-xx.compute-1.internal@.
+   
+As root, on the master (change @compute-1.internal@ to match your setup):
+
+<pre>
+    apt-get install nfs-kernel-server 
+    echo "/home *.compute-1.internal(rw)" >> /etc/exports ;
+    /etc/init.d/nfs-kernel-server stop ;
+</pre>
+
+(The @*.compute-1.internal@ part limits host access, but you should take a look at the security settings of both EC2 and the built-in portmapper as well.)
+
+Next, set up a regular user account on the *master only*. In this case our user will be named 'chimpy':
+
+<pre>
+  visudo # uncomment the last line, to allow group sudo to sudo
+  groupadd admin 
+  adduser chimpy
+  usermod -a -G sudo,admin chimpy
+  su chimpy                  # now you are the new user
+  ssh-keygen -t rsa          # accept all the defaults
+  cat ~/.ssh/id_rsa.pub      # can paste this public key into your github, etc
+  cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys2
+</pre>    
+
+Then on each slave (replacing domU-xx-... by the internal name for the master node):
+
+<pre>
+    apt-get install nfs-common ;
+    echo "domU-xx-xx-xx-xx-xx-xx.compute-1.internal:/home  /mnt/home  nfs  rw  0 0"  >> /etc/fstab
+    /etc/init.d/nfs-common restart
+    mkdir /mnt/home
+    mount /mnt/home
+   ln -s /mnt/home/chimpy /home/chimpy
+</pre>
+
+You should now be in business. 
+
+Performance tradeoffs should be small as long as you're just sending code files and gems around.  *Don't* write out log entries or data to NFS partitions, or you'll effectively perform a denial-of-service attack on the master node.
+
+------------------------------
+
+The "Setting up an NFS Server HOWTO":http://nfs.sourceforge.net/nfs-howto/index.html was an immense help, and I recommend reading it  carefully.

data/doc/hadoop-setup.textile

@@ -0,0 +1,29 @@
+
+h2. Hadoop on EC2
+
+* http://www.cloudera.com/hadoop-ec2
+* http://www.cloudera.com/hadoop-ec2-ebs-beta
+
+
+h3. Setup NFS within the cluster
+
+* 
+* http://nfs.sourceforge.net/nfs-howto/ar01s03.html
+
+
+h3. Miscellaneous Hadoop Tips
+
+* The Cloudera AMIs and distribution include BZip2 support.  This means that if you have input files with a .bz2 extension, they will be naturally un-bzipped and streamed. (Note that there is a non-trivial penalty for doing so: each bzip'ed file must go, in whole, to a single mapper; and the CPU load for un-bzipping is sizeable.)
+
+* To _produce_ bzip2 files, specify the new @--compress_output=@ flag.  If you have the BZip2 patches installed, you can give @--compress_output=bz2@; everyone should be able to use @--compress_output=gz@.
+
+* For excellent performance you can patch your install for "Parallel LZO Splitting":http://www.cloudera.com/blog/2009/06/24/parallel-lzo-splittable-compression-for-hadoop/
+
+
+h3. Tools for EC2 and S3 Management
+
+* http://s3sync.net/wiki
+* http://jets3t.s3.amazonaws.com/applications/applications.html#uploader
+* "ElasticFox"
+* "S3Fox (S3 Organizer)":
+* "FoxyProxy":

data/doc/index.textile

@@ -0,0 +1,124 @@
+---
+layout: default
+title:  mrflip.github.com/wukong
+collapse: false
+---
+
+h1(gemheader). wukong %(small):: hadoop made easy%
+
+
+p(description). {{ site.description }}
+
+
+Treat your dataset like a
+* stream of lines when it's efficient to process by lines
+* stream of field arrays when it's efficient to deal directly with fields
+* stream of lightweight objects when it's efficient to deal with objects
+
+Wukong is friends with "Hadoop":http://hadoop.apache.org/core the elephant, "Pig":http://hadoop.apache.org/pig/ the query language, and the @cat@ on your command line.
+
+<notextile><div class="toggle"></notextile>
+
+h2. How to write a Wukong script
+
+Here's a script to count words in a text stream:
+
+<pre><code>
+    require 'wukong'
+    module WordCount
+      class Mapper < Wukong::Streamer::LineStreamer
+        # Emit each word in the line.
+        def process line
+          words = line.strip.split(/\W+/).reject(&:blank?)
+          words.each{|word| yield [word, 1] }
+        end
+      end
+      
+      class Reducer < Wukong::Streamer::ListReducer
+        def finalize
+          yield [ key, values.map(&:last).map(&:to_i).sum ]
+        end
+      end
+    end
+    
+    Wukong::Script.new(
+      WordCount::Mapper,
+      WordCount::Reducer
+      ).run # Execute the script
+</code></pre>
+
+The first class, the Mapper, eats lines and craps @[word, count]@ records: word is the /key/, its count is the /value/.
+
+In the reducer, the values for each key are stacked up into a list; then the record(s) yielded by @#finalize@ are emitted. There are many other ways to write the reducer (most of them are better) -- see the ["examples":examples/]
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. Structured data stream
+
+You can also use structs to treat your dataset as a stream of objects:
+
+<pre><code>
+    require 'wukong'
+    require 'my_blog' #defines the blog models
+    # structs for our input objects
+    Tweet = Struct.new( :id, :created_at, :twitter_user_id,
+      :in_reply_to_user_id, :in_reply_to_status_id, :text )
+    TwitterUser  = Struct.new( :id, :username, :fullname,
+      :homepage, :location, :description )
+    module TwitBlog
+      class Mapper < Wukong::Streamer::RecordStreamer
+        # Watch for tweets by me
+        MY_USER_ID = 24601
+        #
+        # If this is a tweet is by me, convert it to a Post.
+        #
+        # If it is a tweet not by me, convert it to a Comment that
+        # will be paired with the correct Post.
+        #
+        # If it is a TwitterUser, convert it to a User record and
+        # a user_location record
+        #
+        def process record
+          case record
+          when TwitterUser
+            user     = MyBlog::User.new.merge(record) # grab the fields in common
+            user_loc = MyBlog::UserLoc.new(record.id, record.location, nil, nil)
+            yield user
+            yield user_loc
+          when Tweet
+            if record.twitter_user_id == MY_USER_ID
+              post = MyBlog::Post.new.merge record
+              post.link = "http://twitter.com/statuses/show/#{record.id}"
+              post.body = record.text
+              post.title = record.text[0..65] + "..."
+              yield post
+            else
+              comment = MyBlog::Comment.new.merge record
+              comment.body    = record.text
+              comment.post_id = record.in_reply_to_status_id
+              yield comment
+            end
+          end
+        end
+      end
+    end
+    Wukong::Script.new( TwitBlog::Mapper, nil ).run # identity reducer
+</code></pre>
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. More info
+
+There are many useful examples (including an actually-useful version of the WordCount script) in examples/ directory.
+
+h3. Authors
+
+Philip (flip) Kromer (flip@infochimps.org)
+
+Patches submitted by:
+* gemified by Ben Woosley (ben.woosley@gmail.com)
+* ruby interpreter path fix by "Yuichiro MASUI":http://github.com/masuidrive - masui@masuidrive.jp - http://blog.masuidrive.jp/
+
+<notextile></div></notextile>
+
+{% include news.html %}