mrflip-wukong 0.1.0

data/doc/tips.textile

@@ -0,0 +1,65 @@
+http://cluster-fork.info/index.php?title=Main_Page
+
+
+h3. For Big Data, instead of "ACID" you use "ACID*"
+
+* A -- Associative
+* C -- Commutative
+* I -- Idempotent
+* D -- Distributed
+* (*) -- (and where possible, left in sort order)
+
+
+Finally, where possible leave things in sort order by some appropriate index. Clearly I'm not talking about introducing extra unnecessary sorts on ephemeral data.  For things that will be read (and experimented with) much more often than they're written, though, it's worth running a final sort.  Now you can
+
+* Efficiently index into a massive dataset with binary search
+* Do a direct merge sort on two files with the same sort order
+* Run a reducer directly across the data
+* Assign a synthetic key by just serially numbering lines (either distribute a unique prefix to each mapper 
+
+Note: for files that will live on the DFS, you should usually *not* do a total sort, 
+
+h3. Encode once, and carefully.
+
+Encoding violates idempotence.
+
+Is there a lightweight, mostly-transparent, ASCII-compatible *AND* idempotent encoding scheme lurking in a back closet of some algorithms book?
+
+
+
+h3. Keys
+
+Natural keys are right for big data
+
+Synthetic keys suck. They demand locality or a central keymaster.
+
+See About Keys
+
+* Use the natural key
+* Hash the natural key. This has some drawbacks
+
+OK, fine. you need a synthetic key
+
+* Do a total sort, and use nl
+* Generate
+* Use a single reducer to reduce locality. YUCK.
+* have each mapper generate a unique prefix; number each line as "prefix#{line_number}" or whatever.
+
+How do you get a unique prefix?
+
+* Distribute a unique prefix to each mapper out-of-band. People using Streaming are out of luck.
+
+* Use a UUID -- that's what they're for. Drawback: ridiculously long
+
+* Hash the machine name, PID and timestamp to something short. Check after the
+  fact that uniqueness was achieved. Use the birthday party formula to find out
+  how often this will happen. (In practice, almost never.)
+
+h3. Epistemology and exeption handling
+
+something that goes wrong 1/1000 time will happen
+
+
+h3. Real hackers use the command line as an IDE
+
+

data/doc/utils.textile

@@ -0,0 +1,48 @@
+
+<something to tab and align table>
+
+
+* uniq                  - report or filter out repeated lines in a file
+**  -c                    produces line<tab>count
+**  --ignore f1,f2,...    discards given fields from consideration. field syntax same as for cut, etc.
+
+* sort                  - sort lines of text files
+** columns indexed as tab-separated
+** can specify any column order, uses same field spec as cut
+* tsort                 - topological sort of a directed graph
+
+* cut                   - select portions of each line of a file
+** can reorder columns
+* nl                    - line numbering filter
+** takes prefix, suffix
+** count \t line -OR- line \t count 
+
+* wc                    - word, line, character, and byte count
+** field count (tab-separated fields)
+* paste                 - merge corresponding or subsequent lines of files
+* expand, unexpand      - expand tabs to spaces, and vice versa
+* seq
+* simple row, column sums
+* join                  - relational database operator
+* tac
+
+* cat                   - concatenate and print files
+* head                  - display first lines of a file
+* tail                  - display the last part of a file
+* shuf
+* split                 - split a file into pieces
+* csplit                - split files based on context
+* tee                   - pipe fitting
+
+* ls                    - list directory contents. 
+* df                    - display free disk space
+* du                    - display disk usage statistics
+** tab-delimited, space aligned
+
+* od                    - octal, decimal, hex, ASCII dump
+* printf                - formatted output
+* cksum, sum            - display file checksums and block counts
+* md5sum
+
+* diff
+* comm

data/examples/README.txt

@@ -0,0 +1,17 @@
+Examples:
+
+
+* sample_records -- extract a random sample from a collection of data
+
+* word_count
+
+* apache_log_parser -- example for parsing standard apache webserver log files.
+
+* wordchains -- solving a word puzzle using breadth-first search of a graph
+
+* graph -- some generic graph
+
+* pagerank -- use the pagerank algorithm to find the most 'interesting'
+  (central) nodes of a network graph
+
+  

data/examples/and_pig/sample_queries.rb

@@ -0,0 +1,128 @@
+#!/usr/bin/env ruby
+$: << File.dirname(__FILE__) + '/../../lib'
+require 'wukong'         ; include Wukong
+require 'wukong/and_pig' ; include Wukong::AndPig
+
+# PIG_DIR = '/usr/local/share/pig'
+PIG_DIR = '/public/share/pig'
+# full pathname to the pig executable
+# Wukong::AndPig::PIG_EXECUTABLE = "#{PIG_DIR}/bin/pig"
+Wukong::AndPig::PIG_EXECUTABLE = "/public/bin/pig -x local"
+
+#
+HDFS_BASE_DIR = 'foo/meta/lang'
+Wukong::AndPig::PigVar.working_dir = HDFS_BASE_DIR
+Wukong::AndPig.comments    = false
+# Wukong::AndPig.emit_dest   = :captured
+
+Wukong::AndPig::PigVar.emit "REGISTER #{PIG_DIR}/contrib/piggybank/java/piggybank.jar"
+
+#
+# Load basic types
+#
+
+# class Token < Struct.new(:rsrc, :context, :user_id, :token, :usages)
+# end
+# :tokens_users_0 << Token.pig_load('meta/datanerds/token_count/users_tokens')
+# :tokens_users_0 << Token.pig_load('/tmp/users_tokens.tsv')
+# :tokens_users   << :tokens_users_0.generate(:user_id, :token, :usages)
+# :tokens_users.checkpoint!
+
+class Token < TypedStruct.new(
+      [:user_id, Integer], [:token, String], [:usages, Integer])
+end
+:tokens_users << Token.pig_load('/tmp/users_tokens.tsv')
+:tokens_users.describe
+
+pig_comment %Q{
+# ***************************************************************************
+#
+# Global totals
+#
+# Each row in Tokens lists a (user, token, usages)
+# We want
+#   Sum of all usage counts = total tokens seen in tweet stream.
+#   Number of distinct tokens
+#   Number of distinct users <- different than total in twitter_users.tsv
+#                               because we want only users that say stuff.
+}
+
+def count_distinct relation, field, options={}
+  result_name = options[:as] || "#{relation.name}_#{field}_count".to_sym
+  a = relation.
+    generate(field).set!.describe.
+    distinct(options).set!
+  result_name << a.
+    group(:all).set!.
+    generate(["COUNT(#{a.relation}.#{field})", :u_count, Integer]).set!
+end
+
+pig_comment "Count Users"
+tok_users_count = count_distinct(:tokens_users, :user_id).checkpoint!
+
+pig_comment "Count Tokens"
+tok_tokens_count = count_distinct(:tokens_users, :token, :parallel => 10).checkpoint!
+
+
+pig_comment %Q{
+# ***************************************************************************
+#
+# Statistics for each user
+}
+
+def user_stats users_tokens
+  users_tokens.describe.
+    group(   :user_id).set!.describe.
+    generate(
+      [:group, :user_id],
+      ["(int)COUNT(#{users_tokens.relation})",          :tot_tokens, Integer],
+      [  "(int)SUM(#{users_tokens.relation}.usages)",   :tot_usages, Integer],
+      [   "FLATTEN(#{users_tokens.relation}.token",     :token,      String ],
+      [   "FLATTEN(#{users_tokens.relation}.usages",    :usages,     Integer]).set!.describe.
+    # [   "FLATTEN(#{users_tokens.relation}.(token, usages) )", [:token, :usages], TypedStruct.new([:token, String], [:usages, Integer])]).set!.
+    generate(:user_id, :token, :usages,
+         ["(float)(1.0*usages / tot_usages)", :usage_pct, Float],
+         ["(float)(1.0*usages / tot_usages) * (1.0*(float)usages / tot_usages)", :usage_pct_sq, Float]).set!
+end
+
+:user_stats << user_stats(:tokens_users)
+:user_stats.describe.checkpoint!
+puts "UserStats               = LOAD    'foo/meta/lang/user_stats' AS (user_id, token, usages, usage_pct, usage_pct_sq) ;"
+
+UserStats = TypedStruct.new([:user_id, Integer],
+  [:token, String],
+  [:usages, Integer],
+  [:usage_pct, Float],
+  [:usage_pct_sq, Float])
+:user_stats << UserStats.pig_load('foo/meta/lang/user_stats')
+
+def range_and_dispersion user_stats
+
+  n_users  = 436
+  n_tokens = 61630
+
+  token_stats = user_stats.group(:token).set!
+  token_stats = token_stats.foreach(
+      ["(float)SUM(#{user_stats.relation}.usage_pct)    / #{n_users.to_f}",   :avg_uspct ],
+    ["(float)SUM(#{user_stats.relation}.usage_pct_sq)",                     :sum_uspct_sq],
+    ["org.apache.pig.piggybank.evaluation.math.SQRT(
+                        (sum_uspct_sq /436) -
+                        ( (SUM(#{user_stats.relation}.usage_pct)/436.0) * (SUM(#{user_stats.relation}.usage_pct)/436.0) )
+                        )",     :stdev_uspct],
+    ["1 - ( ( stdev_uspct / avg_uspct ) / org.apache.pig.piggybank.evaluation.math.SQRT(436.0 - 1.0) )",  :dispersion],
+    [
+      [:group,                                                                :token,     String     ],
+      ["(int)COUNT(#{user_stats.relation}) ",                                 :range,     Integer     ],
+      ["(int)COUNT(#{user_stats.relation})              / #{n_users.to_f}",   :pct_range,  Integer ],
+      ["(int)SUM(  #{user_stats.relation}.usages)",                           :tot_usages, Integer],
+      ["(float)( 1.0e6*SUM(#{user_stats.relation}.usages) / #{n_tokens.to_f})", :ppm_usages, Float],
+      [:avg_uspct,   :avg_uspct],
+      [:stdev_uspct, :stdev_uspct],
+      [:dispersion,  :dispersion]
+    ]
+    ).set!
+end
+
+range_and_dispersion(:user_stats).checkpoint!
+
+Wukong::AndPig.finish

data/examples/apache_log_parser.rb

@@ -0,0 +1,53 @@
+#!/usr/bin/env ruby
+$: << File.dirname(__FILE__)+'/../lib'
+require 'wukong'
+
+module ApacheLogParser
+  class Mapper < Wukong::Streamer::LineStreamer
+
+
+    def parse_request req
+      m = %r{\A(\w+) (.*) (\w+/[\w\.]+)\z}.match(req)
+      if m
+        [''] + m.captures
+      else
+        [req, '', '', '']
+      end
+    end
+
+
+    # regular expression to match on apache-style log lines
+    #             IP addr              -         -        [07/Jun/2008:20:37:11 +0000]               400   "GET /faq" + gaJsHost + "google-analytics.com/ga.js HTTP/1.1" 173 "-" "-" "-"
+    LOG_RE = %r{\A(\d+\.\d+\.\d+\.\d+) ([^\s]+) ([^\s]+) \[(\d\d/\w+/\d+):(\d\d:\d\d:\d\d)([^\]]*)\] (\d+) "([^\"]*(?:\" \+ gaJsHost \+ \"[^\"]*)?)" (\d+) "([^\"]*)" "([^\"]*)" "([^\"]*)"\z}
+
+    def process line
+      line.chomp
+      m = LOG_RE.match(line)
+      if m
+        ip, j1, j2, datepart, timepart, tzpart, resp, req, j3, ref, ua, j4 = m.captures
+        req_date = DateTime.parse("#{datepart} #{timepart} #{tzpart}").to_flat
+        req, method, path, protocol = parse_request(req)
+        yield [:logline, method, path, protocol, ip, j1, j2, req_date, resp, req, j3, ref, ua, j4]
+      else
+        yield [:unparseable, line]
+      end
+    end
+  end
+
+  class Reducer < Wukong::Streamer::LineStreamer
+  end
+
+  # Execute the script
+  class Script < Wukong::Script
+    def reduce_command
+      "/usr/bin/uniq"
+    end
+    def default_options
+      super.merge :sort_fields => 8 # , :reduce_tasks => 0
+    end
+  end
+
+  Script.new(Mapper,nil).run
+end
+
+# 55.55.155.55 - - [04/Feb/2008:11:37:52 +0000] 301 "GET /robots.txt HTTP/1.1" 185 "-" "WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)" "-"

data/examples/count_keys.rb

@@ -0,0 +1,56 @@
+#!/usr/bin/env ruby
+$: << File.dirname(__FILE__)+'/../lib'
+require 'wukong'
+require 'wukong/streamer/count_keys'
+require 'wukong/streamer/count_lines'
+
+#
+#
+class CountKeysReducer < Wukong::Streamer::CountLines
+  #
+  # Taken from the actionpack Rails component ('action_view/helpers/number_helper')
+  #
+  # Formats a +number+ with grouped thousands using +delimiter+. You
+  # can customize the format using optional <em>delimiter</em> and <em>separator</em> parameters.
+  # * <tt>delimiter</tt>  - Sets the thousands delimiter, defaults to ","
+  # * <tt>separator</tt>  - Sets the separator between the units, defaults to "."
+  #
+  #  number_with_delimiter(12345678)      => 12,345,678
+  #  number_with_delimiter(12345678.05)   => 12,345,678.05
+  #  number_with_delimiter(12345678, ".")   => 12.345.678
+  def number_with_delimiter(number, delimiter=",", separator=".")
+    begin
+      parts = number.to_s.split('.')
+      parts[0].gsub!(/(\d)(?=(\d\d\d)+(?!\d))/, "\\1#{delimiter}")
+      parts.join separator
+    rescue
+      number
+    end
+  end
+
+  # Override to look nice
+  def formatted_count item, key_count
+    key_count_str = number_with_delimiter(key_count.to_i)
+    "%-25s\t%12s" % [item, key_count_str]
+  end
+end
+
+#
+class CountKeysScript < Wukong::Script
+  def map_command
+    # Use `cut` to extract the first field
+    %Q{ cut -d"\t" -f1 }
+  end
+
+  #
+  # There's just the one field
+  #
+  def default_options
+    super.merge :sort_fields => 1
+  end
+end
+
+# Executes the script when run from command line
+if __FILE__ == $0
+  CountKeysScript.new(nil, CountKeysReducer).run
+end

data/examples/count_keys_at_mapper.rb

@@ -0,0 +1,57 @@
+#!/usr/bin/env ruby
+$: << File.dirname(__FILE__)+'/../lib'
+require 'wukong'
+
+#
+#
+module CountKeys
+  #
+  class Mapper < Wukong::Streamer::Base
+    attr_accessor :keys_count
+    def initialize *args
+      self.keys_count = {}
+    end
+    def process key, *args
+      key.gsub!(/-.*/, '')  # kill off the slug
+      self.keys_count[key] ||= 0
+      self.keys_count[key]  += 1
+    end
+    def stream *args
+      super *args
+      self.keys_count.each do |key, count|
+        emit [key, count].to_flat
+      end
+    end
+  end
+  # Identity Mapper
+  class Reducer < Wukong::Streamer::AccumulatingReducer
+    attr_accessor :key_count
+    require 'active_support'
+    require 'action_view/helpers/number_helper'; include ActionView::Helpers::NumberHelper
+
+    # Override to look nice
+    def formatted_count item, key_count
+      key_count_str = number_with_delimiter(key_count.to_i, :delimiter => ',')
+      "%-25s\t%12s" % [item, key_count_str]
+    end
+    def start!  *args
+      self.key_count = 0
+    end
+    def accumulate key, count
+      self.key_count += count.to_i
+    end
+    def finalize
+      yield formatted_count(key, key_count)
+    end
+  end
+
+  #
+  class Script < Wukong::Script
+    # There's just the one field
+    def default_options
+      super.merge :sort_fields => 1, :reduce_tasks => 1
+    end
+  end
+end
+
+CountKeys::Script.new(CountKeys::Mapper, CountKeys::Reducer).run

data/examples/graph/adjacency_list.rb

@@ -0,0 +1,74 @@
+#!/usr/bin/env ruby
+$: << '/home/flip/ics/wukong/lib' # ENV['WUKONG_PATH']
+require 'wukong'
+
+#
+# Given an adjacency pairs (from \t to) representation of a directed graph:
+#
+#    1     2
+#    1     7
+#    2     7
+#    2     9
+#    7     2
+#
+# It produces an "adjacency list":http://en.wikipedia.org/wiki/Adjacency_list representation:
+#
+#    1  >   2   7
+#    2  >   7   9
+#    7  >   2
+#    9  >
+#
+# and
+#
+#    1  <
+#    2  <   1   7
+#    7  <   1   2
+#    9  <   2
+#
+# (each column is tab-separated in the actual output)
+#
+#
+#
+module Gen1HoodEdges
+  class Mapper < Wukong::Streamer::Base
+    def process rsrc, src, dest, *_
+      src = src.to_i ; dest = dest.to_i
+      yield [ src,  '>', dest ]
+      yield [ dest, '<', src  ]
+    end
+  end
+
+  #
+  # Accumulate links onto single line.
+  #
+  # The reduce key is the target node and direction; we just stream through all
+  # pairs for each target node and output its neighbor nodes on the same line.
+  #
+  # To control memory usage, we will print directly to the output (and not run
+  # through the Emitter)
+  #
+  class Reducer < Wukong::Streamer::AccumulatingReducer
+    # clear the list of incoming paths
+    def start! target, dir, *args
+      print target + "\t" + dir  # start line with target and list type
+    end
+    def accumulate target, dir, neighbor
+      print "\t" + neighbor      # append neighbor to output, same line
+    end
+    def finalize
+      puts ''                    # start new line
+    end
+  end
+
+  class Script < Wukong::Script
+    def default_options
+      super.merge :sort_fields => 1, :partition_fields => 1
+    end
+  end
+end
+
+# Execute the script
+Gen1HoodEdges::Script.new(
+  Gen1HoodEdges::Mapper,
+  Gen1HoodEdges::Reducer
+  ).run