wukong 0.1.4 → 1.4.0

data/doc/hadoop-setup.textile

@@ -1,29 +0,0 @@
-
-h2. Hadoop on EC2
-
-* http://www.cloudera.com/hadoop-ec2
-* http://www.cloudera.com/hadoop-ec2-ebs-beta
-
-
-h3. Setup NFS within the cluster
-
-* 
-* http://nfs.sourceforge.net/nfs-howto/ar01s03.html
-
-
-h3. Miscellaneous Hadoop Tips
-
-* The Cloudera AMIs and distribution include BZip2 support.  This means that if you have input files with a .bz2 extension, they will be naturally un-bzipped and streamed. (Note that there is a non-trivial penalty for doing so: each bzip'ed file must go, in whole, to a single mapper; and the CPU load for un-bzipping is sizeable.)
-
-* To _produce_ bzip2 files, specify the new @--compress_output=@ flag.  If you have the BZip2 patches installed, you can give @--compress_output=bz2@; everyone should be able to use @--compress_output=gz@.
-
-* For excellent performance you can patch your install for "Parallel LZO Splitting":http://www.cloudera.com/blog/2009/06/24/parallel-lzo-splittable-compression-for-hadoop/
-
-
-h3. Tools for EC2 and S3 Management
-
-* http://s3sync.net/wiki
-* http://jets3t.s3.amazonaws.com/applications/applications.html#uploader
-* "ElasticFox"
-* "S3Fox (S3 Organizer)":
-* "FoxyProxy":

data/doc/index.textile

@@ -1,124 +0,0 @@
----
-layout: default
-title:  mrflip.github.com/wukong
-collapse: false
----
-
-h1(gemheader). wukong %(small):: hadoop made easy%
-
-
-p(description). {{ site.description }}
-
-
-Treat your dataset like a
-* stream of lines when it's efficient to process by lines
-* stream of field arrays when it's efficient to deal directly with fields
-* stream of lightweight objects when it's efficient to deal with objects
-
-Wukong is friends with "Hadoop":http://hadoop.apache.org/core the elephant, "Pig":http://hadoop.apache.org/pig/ the query language, and the @cat@ on your command line.
-
-<notextile><div class="toggle"></notextile>
-
-h2. How to write a Wukong script
-
-Here's a script to count words in a text stream:
-
-<pre><code>
-    require 'wukong'
-    module WordCount
-      class Mapper < Wukong::Streamer::LineStreamer
-        # Emit each word in the line.
-        def process line
-          words = line.strip.split(/\W+/).reject(&:blank?)
-          words.each{|word| yield [word, 1] }
-        end
-      end
-      
-      class Reducer < Wukong::Streamer::ListReducer
-        def finalize
-          yield [ key, values.map(&:last).map(&:to_i).sum ]
-        end
-      end
-    end
-    
-    Wukong::Script.new(
-      WordCount::Mapper,
-      WordCount::Reducer
-      ).run # Execute the script
-</code></pre>
-
-The first class, the Mapper, eats lines and craps @[word, count]@ records: word is the /key/, its count is the /value/.
-
-In the reducer, the values for each key are stacked up into a list; then the record(s) yielded by @#finalize@ are emitted. There are many other ways to write the reducer (most of them are better) -- see the ["examples":examples/]
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. Structured data stream
-
-You can also use structs to treat your dataset as a stream of objects:
-
-<pre><code>
-    require 'wukong'
-    require 'my_blog' #defines the blog models
-    # structs for our input objects
-    Tweet = Struct.new( :id, :created_at, :twitter_user_id,
-      :in_reply_to_user_id, :in_reply_to_status_id, :text )
-    TwitterUser  = Struct.new( :id, :username, :fullname,
-      :homepage, :location, :description )
-    module TwitBlog
-      class Mapper < Wukong::Streamer::RecordStreamer
-        # Watch for tweets by me
-        MY_USER_ID = 24601
-        #
-        # If this is a tweet is by me, convert it to a Post.
-        #
-        # If it is a tweet not by me, convert it to a Comment that
-        # will be paired with the correct Post.
-        #
-        # If it is a TwitterUser, convert it to a User record and
-        # a user_location record
-        #
-        def process record
-          case record
-          when TwitterUser
-            user     = MyBlog::User.new.merge(record) # grab the fields in common
-            user_loc = MyBlog::UserLoc.new(record.id, record.location, nil, nil)
-            yield user
-            yield user_loc
-          when Tweet
-            if record.twitter_user_id == MY_USER_ID
-              post = MyBlog::Post.new.merge record
-              post.link = "http://twitter.com/statuses/show/#{record.id}"
-              post.body = record.text
-              post.title = record.text[0..65] + "..."
-              yield post
-            else
-              comment = MyBlog::Comment.new.merge record
-              comment.body    = record.text
-              comment.post_id = record.in_reply_to_status_id
-              yield comment
-            end
-          end
-        end
-      end
-    end
-    Wukong::Script.new( TwitBlog::Mapper, nil ).run # identity reducer
-</code></pre>
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. More info
-
-There are many useful examples (including an actually-useful version of the WordCount script) in examples/ directory.
-
-h3. Authors
-
-Philip (flip) Kromer (flip@infochimps.org)
-
-Patches submitted by:
-* gemified by Ben Woosley (ben.woosley@gmail.com)
-* ruby interpreter path fix by "Yuichiro MASUI":http://github.com/masuidrive - masui@masuidrive.jp - http://blog.masuidrive.jp/
-
-<notextile></div></notextile>
-
-{% include news.html %}

data/doc/links.textile

@@ -1,42 +0,0 @@
-h3. Setting up your cluster:
-
-* "Running Hadoop On Ubuntu Linux (Single-Node Cluster)":http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster) and "unning Hadoop On Ubuntu Linux (Multi-Node Cluster).":http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
-* "Running Hadoop MapReduce on Amazon EC2 and S3":http://developer.amazonwebservices.com/connect/entry.jspa?externalID=873
-
-
-
-
-* "Hadoop Overview by Doug Cutting":http://video.google.com/videoplay?docid=-4912926263813234341 - the founder of the Hadoop project. 49m
-
-* "Cluster Computing and Map|Reduce":http://www.youtube.com/results?search_query=cluster+computing+and+mapreduce
-** "Lecture 1: Overview":http://www.youtube.com/watch?v=yjPBkvYh-ss
-** "Lecture 2 (technical): Map|Reduce":http://www.youtube.com/watch?v=-vD6PUdf3Js
-** "Lecture 3 (technical): GFS (Google File System)":http://www.youtube.com/watch?v=5Eib_H_zCEY
-** "Lecture 4 (theoretical): Canopy Clustering":http://www.youtube.com/watch?v=1ZDybXl212Q
-** "Lecture 5 (theoretical): Breadth-First Search":http://www.youtube.com/watch?v=BT-piFBP4fE
-
-
-* http://www.cloudera.com/hadoop-training
-
-** "Thinking at Scale":http://www.cloudera.com/hadoop-training-thinking-at-scale
-** "Mapreduce and HDFS":http://www.cloudera.com/hadoop-training-mapreduce-hdfs
-** "A Tour of the Hadoop Ecosystem":http://www.cloudera.com/hadoop-training-ecosystem-tour
-** "Programming with Hadoop":http://www.cloudera.com/hadoop-training-programming-with-hadoop
-
-** "Hadoop and Hive: introduction":http://www.cloudera.com/hadoop-training-hive-introduction
-** "Hadoop and Hive: tutorial":http://www.cloudera.com/hadoop-training-hive-tutorial
-** "Hadoop and Pig: Introduction":http://www.cloudera.com/hadoop-training-pig-introduction
-** "Hadoop and Pig: Tutorial":http://www.cloudera.com/hadoop-training-pig-tutorial
-
-** "Mapreduce Algorithms":http://www.cloudera.com/hadoop-training-mapreduce-algorithms
-** "Exercise: Getting started with Hadoop":http://www.cloudera.com/hadoop-training-exercise-getting-started-with-hadoop
-** "Exercise: Writing mapreduce programs":http://www.cloudera.com/hadoop-training-exercise-writing-mapreduce-programs
-
-
-
-
----------------------------------------------------------------------------
-
-* "Hadoop Wiki: Hadoop Streaming":http://wiki.apache.org/hadoop/HadoopStreaming
-* "Hadoop Docs: Hadoop Streaming":http://hadoop.apache.org/common/docs/current/streaming.html
-

data/doc/usage.textile

@@ -1,102 +0,0 @@
----
-layout: default
-title:  Usage notes
----
-
-h1(gemheader). {{ site.gemname }} %(small):: usage%
-
-
-<notextile><div class="toggle"></notextile>
-
-h2. How to run a Wukong script
-
-To run your script using local files and no connection to a hadoop cluster,
-
-pre. your/script.rb --run=local path/to/input_files path/to/output_dir
-
-To run the command across a Hadoop cluster,
-
-pre. your/script.rb --run=hadoop path/to/input_files path/to/output_dir
-
-You can set the default in the config/wukong-site.yaml file, and then just use @--run@ instead of @--run=something@ --it will just use the default run mode.
-
-If you're running @--run=hadoop@, all file paths are HDFS paths. If you're running @--run=local@, all file paths are local paths.  (your/script path, of course, lives on the local filesystem).
-
-You can supply arbitrary command line arguments (they wind up as key-value pairs in the options path your mapper and reducer receive), and you can use the hadoop syntax to specify more than one input file:
-
-pre. ./path/to/your/script.rb --any_specific_options --options=can_have_vals \
-   --run "input_dir/part_*,input_file2.tsv,etc.tsv" path/to/output_dir
-
-Note that all @--options@ must precede (in any order) all non-options.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. How to test your scripts
-
-To run mapper on its own:
-
-pre. cat ./local/test/input.tsv | ./examples/word_count.rb --map | more
-
-or if your test data lies on the HDFS,
-
-pre. hdp-cat test/input.tsv | ./examples/word_count.rb --map | more
-
-Next graduate to running @--run=local@ mode so you can inspect the reducer.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. What tools does Wukong work with?
-
-Wukong is friends with "Hadoop":http://hadoop.apache.org/core the elephant, "Pig":http://hadoop.apache.org/pig/ the query language, and the @cat@ on your command line.  We're looking forward to being friends with "martinis":http://datamapper.org and "express trains":http://wiki.rubyonrails.org/rails/pages/ActiveRecord down the road.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. Design
-
-...
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. Caveats
-
-...
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. TODOs
-
-...
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. Note on Patches/Pull Requests
-
-* Fork the project.
-* Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a future version unintentionally.
-* Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but  bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. Endnotes
-
-
-h3. Why is it called Wukong?
-
-Hadoop, as you may know, is "named after a stuffed elephant.":http://en.wikipedia.org/wiki/Hadoop Since Wukong was started by the "infochimps":http://infochimps.org team, we needed a simian analog.  A Monkey King who journeyed to the land of the Elephant seems to fit the bill:
-
-bq. Sun Wukong (孙悟空), known in the West as the Monkey King, is the main character in the classical Chinese epic novel Journey to the West. In the novel, he accompanies the monk Xuanzang on the journey to retrieve Buddhist sutras from India.
-
-bq. Sun Wukong possesses incredible strength, being able to lift his 13,500 jīn (8,100 kg) Ruyi Jingu Bang with ease. He also has superb speed, traveling 108,000 li (54,000 kilometers) in one somersault. Sun knows 72 transformations, which allows him to transform into various animals and objects; he is, however, shown with slight problems transforming into other people, since he is unable to complete the transformation of his tail. He is a skilled fighter, capable of holding his own against the best generals of heaven. Each of his hairs possesses magical properties, and is capable of transforming into a clone of the Monkey King himself, or various weapons, animals, and other objects. He also knows various spells in order to command wind, part water, conjure protective circles against demons, freeze humans, demons, and gods alike. -- ["Sun Wukong's Wikipedia entry":http://en.wikipedia.org/wiki/Wukong]
-
-The "Jaime Hewlett / Damon Albarn short":http://news.bbc.co.uk/sport1/hi/olympics/monkey that the BBC made for their 2008 Olympics coverage gives the general idea.
-
-
-* What's up with Wukong::AndPig?
-** @Wukong::AndPig@ is a small library to more easily generate code for the "Pig":http://hadoop.apache.org/pig data analysis language.  See its "README":wukong/and_pig/README.textile for more.
-
-
-
-<notextile></div></notextile>
-

data/doc/utils.textile

@@ -1,48 +0,0 @@
-
-<something to tab and align table>
-
-
-* uniq                  - report or filter out repeated lines in a file
-**  -c                    produces line<tab>count
-**  --ignore f1,f2,...    discards given fields from consideration. field syntax same as for cut, etc.
-
-* sort                  - sort lines of text files
-** columns indexed as tab-separated
-** can specify any column order, uses same field spec as cut
-* tsort                 - topological sort of a directed graph
-
-* cut                   - select portions of each line of a file
-** can reorder columns
-* nl                    - line numbering filter
-** takes prefix, suffix
-** count \t line -OR- line \t count 
-
-* wc                    - word, line, character, and byte count
-** field count (tab-separated fields)
-* paste                 - merge corresponding or subsequent lines of files
-* expand, unexpand      - expand tabs to spaces, and vice versa
-* seq
-* simple row, column sums
-* join                  - relational database operator
-* tac
-
-* cat                   - concatenate and print files
-* head                  - display first lines of a file
-* tail                  - display the last part of a file
-* shuf
-* split                 - split a file into pieces
-* csplit                - split files based on context
-* tee                   - pipe fitting
-
-* ls                    - list directory contents. 
-* df                    - display free disk space
-* du                    - display disk usage statistics
-** tab-delimited, space aligned
-
-* od                    - octal, decimal, hex, ASCII dump
-* printf                - formatted output
-* cksum, sum            - display file checksums and block counts
-* md5sum
-
-* diff
-* comm

data/examples/and_pig/sample_queries.rb

@@ -1,128 +0,0 @@
-#!/usr/bin/env ruby
-$: << File.dirname(__FILE__) + '/../../lib'
-require 'wukong'         ; include Wukong
-require 'wukong/and_pig' ; include Wukong::AndPig
-
-# PIG_DIR = '/usr/local/share/pig'
-PIG_DIR = '/public/share/pig'
-# full pathname to the pig executable
-# Wukong::AndPig::PIG_EXECUTABLE = "#{PIG_DIR}/bin/pig"
-Wukong::AndPig::PIG_EXECUTABLE = "/public/bin/pig -x local"
-
-#
-HDFS_BASE_DIR = 'foo/meta/lang'
-Wukong::AndPig::PigVar.working_dir = HDFS_BASE_DIR
-Wukong::AndPig.comments    = false
-# Wukong::AndPig.emit_dest   = :captured
-
-Wukong::AndPig::PigVar.emit "REGISTER #{PIG_DIR}/contrib/piggybank/java/piggybank.jar"
-
-#
-# Load basic types
-#
-
-# class Token < Struct.new(:rsrc, :context, :user_id, :token, :usages)
-# end
-# :tokens_users_0 << Token.pig_load('meta/datanerds/token_count/users_tokens')
-# :tokens_users_0 << Token.pig_load('/tmp/users_tokens.tsv')
-# :tokens_users   << :tokens_users_0.generate(:user_id, :token, :usages)
-# :tokens_users.checkpoint!
-
-class Token < TypedStruct.new(
-      [:user_id, Integer], [:token, String], [:usages, Integer])
-end
-:tokens_users << Token.pig_load('/tmp/users_tokens.tsv')
-:tokens_users.describe
-
-pig_comment %Q{
-# ***************************************************************************
-#
-# Global totals
-#
-# Each row in Tokens lists a (user, token, usages)
-# We want
-#   Sum of all usage counts = total tokens seen in tweet stream.
-#   Number of distinct tokens
-#   Number of distinct users <- different than total in twitter_users.tsv
-#                               because we want only users that say stuff.
-}
-
-def count_distinct relation, field, options={}
-  result_name = options[:as] || "#{relation.name}_#{field}_count".to_sym
-  a = relation.
-    generate(field).set!.describe.
-    distinct(options).set!
-  result_name << a.
-    group(:all).set!.
-    generate(["COUNT(#{a.relation}.#{field})", :u_count, Integer]).set!
-end
-
-pig_comment "Count Users"
-tok_users_count = count_distinct(:tokens_users, :user_id).checkpoint!
-
-pig_comment "Count Tokens"
-tok_tokens_count = count_distinct(:tokens_users, :token, :parallel => 10).checkpoint!
-
-
-pig_comment %Q{
-# ***************************************************************************
-#
-# Statistics for each user
-}
-
-def user_stats users_tokens
-  users_tokens.describe.
-    group(   :user_id).set!.describe.
-    generate(
-      [:group, :user_id],
-      ["(int)COUNT(#{users_tokens.relation})",          :tot_tokens, Integer],
-      [  "(int)SUM(#{users_tokens.relation}.usages)",   :tot_usages, Integer],
-      [   "FLATTEN(#{users_tokens.relation}.token",     :token,      String ],
-      [   "FLATTEN(#{users_tokens.relation}.usages",    :usages,     Integer]).set!.describe.
-    # [   "FLATTEN(#{users_tokens.relation}.(token, usages) )", [:token, :usages], TypedStruct.new([:token, String], [:usages, Integer])]).set!.
-    generate(:user_id, :token, :usages,
-         ["(float)(1.0*usages / tot_usages)", :usage_pct, Float],
-         ["(float)(1.0*usages / tot_usages) * (1.0*(float)usages / tot_usages)", :usage_pct_sq, Float]).set!
-end
-
-:user_stats << user_stats(:tokens_users)
-:user_stats.describe.checkpoint!
-puts "UserStats               = LOAD    'foo/meta/lang/user_stats' AS (user_id, token, usages, usage_pct, usage_pct_sq) ;"
-
-UserStats = TypedStruct.new([:user_id, Integer],
-  [:token, String],
-  [:usages, Integer],
-  [:usage_pct, Float],
-  [:usage_pct_sq, Float])
-:user_stats << UserStats.pig_load('foo/meta/lang/user_stats')
-
-def range_and_dispersion user_stats
-
-  n_users  = 436
-  n_tokens = 61630
-
-  token_stats = user_stats.group(:token).set!
-  token_stats = token_stats.foreach(
-      ["(float)SUM(#{user_stats.relation}.usage_pct)    / #{n_users.to_f}",   :avg_uspct ],
-    ["(float)SUM(#{user_stats.relation}.usage_pct_sq)",                     :sum_uspct_sq],
-    ["org.apache.pig.piggybank.evaluation.math.SQRT(
-                        (sum_uspct_sq /436) -
-                        ( (SUM(#{user_stats.relation}.usage_pct)/436.0) * (SUM(#{user_stats.relation}.usage_pct)/436.0) )
-                        )",     :stdev_uspct],
-    ["1 - ( ( stdev_uspct / avg_uspct ) / org.apache.pig.piggybank.evaluation.math.SQRT(436.0 - 1.0) )",  :dispersion],
-    [
-      [:group,                                                                :token,     String     ],
-      ["(int)COUNT(#{user_stats.relation}) ",                                 :range,     Integer     ],
-      ["(int)COUNT(#{user_stats.relation})              / #{n_users.to_f}",   :pct_range,  Integer ],
-      ["(int)SUM(  #{user_stats.relation}.usages)",                           :tot_usages, Integer],
-      ["(float)( 1.0e6*SUM(#{user_stats.relation}.usages) / #{n_tokens.to_f})", :ppm_usages, Float],
-      [:avg_uspct,   :avg_uspct],
-      [:stdev_uspct, :stdev_uspct],
-      [:dispersion,  :dispersion]
-    ]
-    ).set!
-end
-
-range_and_dispersion(:user_stats).checkpoint!
-
-Wukong::AndPig.finish