wukong 3.0.0.pre → 3.0.0.pre2

data/old/examples/corpus/words_to_bigrams.rb

@@ -1,53 +0,0 @@
-#!/usr/bin/env ruby
-require 'rubygems'
-require 'wukong/script'
-
-#
-# Bigram counts
-#
-# head -n 100 /usr/share/dict/words | ./examples/corpus/words_to_bigrams.rb  | sort |  /tmp/words_to_bigrams.rb
-#
-
-
-#
-# Kludge to work in Elastic map reduce:
-#
-# If your script is ./examples/corpus/words_to_bigrams.rb, make symlinks
-# to it from ./examples/corpus/words_to_bigrams__map.rb and
-# ./examples/corpus/words_to_bigrams__reduce.rb
-#
-if $0 =~ /__(map|reduce)\.rb$/
-  Settings[$1.to_sym] = true
-end
-
-
-#
-# given one word per line
-# emits all successive pairs of characters in that word
-# eg 'boooo-urns' yields
-#   bo oo oo oo o- -u ur rn ns
-#
-class WordNGrams < Wukong::Streamer::Base
-  def process word
-    word[0..-2].chars.zip(word[1..-1].chars).each do |ngram_2|
-      yield ngram_2.join('')
-    end
-  end
-end
-
-#
-# number of unique keys in a row
-#
-class KeyCountStreamer < Wukong::Streamer::AccumulatingReducer
-  def start! *args
-    @count = 0
-  end
-  def accumulate *args
-    @count += 1
-  end
-  def finalize
-    yield [key, @count]
-  end
-end
-
-Wukong::Script.new(WordNGrams, KeyCountStreamer).run

data/old/examples/emr/README.textile

@@ -1,110 +0,0 @@
-h1. Using Elastic Map-Reduce in Wukong
-
-h2. Initial Setup
-
-# Sign up for elastic map reduce and S3 at Amazon AWS.
-
-# Download the Amazon elastic-mapreduce runner: either the official version at http://elasticmapreduce.s3.amazonaws.com/elastic-mapreduce-ruby.zip or the infochimps fork (which has support for Ruby 1.9) at http://github.com/infochimps/elastic-mapreduce .
-
-# Create a bucket and path to hold your EMR logs, scripts and other ephemera. For instance you might choose 'emr.yourdomain.com' as the bucket and '/wukong' as a scoping path within that bucket. In that case you will refer to it with a path like s3://emr.yourdomain.com/wukong (see notes below about s3n:// vs. s3:// URLs).
-
-# Copy the contents of wukong/examples/emr/dot_wukong_dir to ~/.wukong
-
-# Edit emr.yaml and credentials.json, adding your keys where appropriate and  following the other instructions. Start with a single-node m1.small cluster as you'll probably have some false starts beforethe flow of logging in, checking the logs, etc becomes clear.
-
-# You should now be good to launch a program. We'll give it the @--alive@ flag so that the machine sticks around if there were any issues:
-
-   ./elastic_mapreduce_example.rb --run=emr --alive s3://emr.yourdomain.com/wukong/data/input s3://emr.yourdomain.com/wukong/data/output
-
-# If you visit the "AWS console":http://bit.ly/awsconsole you should now see a jobflow with two steps. The first sets up debugging for the job; the second is your hadoop task.
-
-# The "AWS console":http://bit.ly/awsconsole also has the public IP of the master node. You can log in to the machine directly:
-
-<pre>
-  ssh -i /path/to/your/keypair.pem hadoop@ec2-148-37-14-128.compute-1.amazonaws.com
-</pre>
-
-h3. Lorkbong
-
-Lorkbong (named after the staff carried by Sun Wukong) is a very very simple example Heroku app that lets you trigger showing job status or launching a new job, either by visiting a special URL or by triggering a rake task. Get its code from
-
-  http://github.com/mrflip/lorkbong
-
-h3. s3n:// vs. s3:// URLs
-
-Many external tools use a URI convention to address files in S3; they typically use the 's3://' scheme, which makes a lot of sense:
-  s3://emr.yourcompany.com/wukong/happy_job_1/logs/whatever-20100808.log
-
-Hadoop can maintain an HDFS on the Amazon S3: it uses a block structure and has optimizations for streaming, no file size limitation, and other goodness. However, only hadoop tools can interpret the contents of those blocks -- to everything else it just looks like a soup of blocks labelled block_-8675309 and so forth.  Hadoop unfortunately chose the 's3://' scheme for URIs in this filesystem:
-  s3://s3hdfs.yourcompany.com/path/to/data
-
-Hadoop is happy to read s3 native files -- 'native' as in, you can look at them with a browser and upload them an download them with any S3 tool out there. There's a 5GB limit on file size, and in some cases a performance hit (but not in our experience enough to worry about).  You refer to these files with the 's3n://' scheme ('n' as in 'native'):
-  s3n://emr.yourcompany.com/wukong/happy_job_1/code/happy_job_1-mapper.rb
-  s3n://emr.yourcompany.com/wukong/happy_job_1/code/happy_job_1-reducer.rb
-  s3n://emr.yourcompany.com/wukong/happy_job_1/logs/whatever-20100808.log
-
-Wukong will coerce things to the right scheme when it knows what that scheme should be (eg. code should be s3n://). It will otherwise leave the path alone. Specifically, if you use a URI scheme for input and output paths you must use 's3n://' for normal s3 files.
-
-h2. Advanced Tips n' Tricks for common usage
-
-h3. Direct access to logs using your browser
-
-Each Hadoop component exposes a web dashboard for you to access. Use the following ports:
-
-* 9100: Job tracker (master only)
-* 9101: Namenode (master only)
-* 9102: Datanodes
-* 9103: Task trackers
-
-They will only, however, respond to web requests from within the private cluster
-subnet. You can browse the cluster by creating a persistent tunnel to the hadoop master node, and configuring your
-browser to use it as a proxy.
-
-h4. Create a tunneling proxy to your cluster
-
-To create a tunnel from your local machine to the master node, substitute the keypair and the master node's address into this command:
-
-<pre><code>
-  ssh -i ~/.wukong/keypairs/KEYPAIR.pem -f -N -D 6666 -o StrictHostKeyChecking=no -o "ConnectTimeout=10" -o "ServerAliveInterval=60" -o "ControlPath=none" ubuntu@MASTER_NODE_PUBLIC_IP
-</code></pre>
-
-The command will silently background itself if it worked.
-
-h4. Make your browser use the proxy (but only for cluster machines)
-
-You can access basic information by pointing your browser to "this Proxy
-Auto-Configuration (PAC)
-file.":http://github.com/infochimps/cluster_chef/raw/master/config/proxy.pac
-You'll have issues if you browse around though, because many of the in-page
-links will refer to addresses that only resolve within the cluster's private
-namespace.
-
-h4. Setup Foxy Proxy
-
-To fix this, use "FoxyProxy":https://addons.mozilla.org/en-US/firefox/addon/2464
-It allows you to manage multiple proxy configurations and to use the proxy for
-DNS resolution (curing the private address problem).
-
-Once you've installed the FoxyProxy extension and restarted Firefox,
-
-* Set FoxyProxy to 'Use Proxies based on their pre-defined patterns and priorities'
-* Create a new proxy, called 'EC2 Socks Proxy' or something
-* Automatic proxy configuration URL: http://github.com/infochimps/cluster_chef/raw/master/config/proxy.pac
-* Under 'General', check yes for 'Perform remote DNS lookups on host'
-* Add the following URL patterns as 'whitelist' using 'Wildcards' (not regular expression):
-
-* <code>*.compute-*.internal*</code>
-* <code>*ec2.internal*</code>
-* <code>*domu*.internal*</code>
-* <code>*ec2*.amazonaws.com*</code>
-* <code>*://10.*</code>
-
-And this one as blacklist:
-
-* <code>https://us-*st-1.ec2.amazonaws.com/*</code>
-
-
-h3. Pulling to your local machine
-
-s3cmd sync s3://s3n.infinitemonkeys.info/emr/elastic_mapreduce_example/log/ /tmp/emr_log/
-

data/old/examples/emr/dot_wukong_dir/credentials.json

@@ -1,7 +0,0 @@
-{
-  "key-pair":      "gibbon",
-  "key-pair-file": "/home/your/.wukong/keypairs/gibbon.pem",
-  "access-id":     "YOURACCESSID",
-  "private-key":   "YOURPRIVATEKEY",
-  "region":        "us-east-1",
-}

data/old/examples/emr/dot_wukong_dir/emr.yaml

@@ -1,69 +0,0 @@
-#
-# Elastic MapReduce config in wukong
-#
-
-# ===========================================================================
-#
-# Infrastructure options
-#
-
-# == Fill all your information into yet another file with your amazon key
-#    It needs to be in so many stupid places because nobody can agree on a
-#    filename or format.
-#
-:emr_credentials_file:          ~/.wukong/credentials.json
-
-#
-# == Set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars, or enter them here
-#  
-# :access_key:                  ASDFAHKHASDF
-# :secret_access_key:           ADSGHASDFJASDFASDF
-
-# == Path to your keypair file.
-#
-:key_pair_file:                 ~/.wukong/keypairs/gibbon.pem
-
-# == Keypair will be named after your file, or force the name
-#
-# :key_pair:                    ~
-
-# == Path to the Amazon elastic-mapreduce runner. Get a copy from
-#    http://elasticmapreduce.s3.amazonaws.com/elastic-mapreduce-ruby.zip
-#
-:emr_runner:                    ~/ics/hadoop/elastic-mapreduce/elastic-mapreduce
-
-# ===========================================================================
-#
-# Remote Paths
-#
-
-# == Wukong is opinionated about the paths and locations of scripts and
-#    everything. It will organize files by job name within the following path:
-#
-:emr_root:                      s3://s3n.infinitemonkeys.info/emr
-
-# == If you specify the :emr_data_root path, then relative pathnames -- ones that
-#    do not look like a URI (s3://yadda/yada) and do not start with a '/' -- will
-#    be prefixed with this path prefix.
-:emr_data_root:                 s3n://s3n.infinitemonkeys.info/data
-
-
-# ===========================================================================
-#
-# Cluster Config
-#
-:num_instances:                 1
-:instance_type:                 m1.small
-:master_instance_type:          ~
-:hadoop_version:                '0.20'
-:availability_zone:             us-east-1b
-
-# ===========================================================================
-#
-# Running and reporting options
-#
-:alive:                         true
-:enable_debugging:              true
-:emr_runner_verbose:            true
-:emr_runner_debug:              ~
-:step_action:                   CANCEL_AND_WAIT         # CANCEL_AND_WAIT, TERMINATE_JOB_FLOW or CONTINUE

data/old/examples/emr/dot_wukong_dir/emr_bootstrap.sh

@@ -1,33 +0,0 @@
-#!/usr/bin/env bash
-set -x  # turn on tracing
-
-# A url directory with the scripts you'd like to stuff into the machine
-REMOTE_FILE_URL_BASE="http://github.com/infochimps/wukong"
-
-# echo "`date` Broaden the apt universe"
-# sudo bash -c 'echo "deb http://ftp.us.debian.org/debian  lenny  multiverse restricted universe" >> /etc/apt/sources.list.d/multiverse.list'
-
-# Do a non interactive apt-get so the user is never prompted for input
-export DEBIAN_FRONTEND=noninteractive
-
-# Update package index and update the basic system files to newest versions
-echo "`date` Apt update" 
-sudo apt-get -y update  ;
-sudo dpkg --configure -a
-echo "`date` Apt upgrade, could take a while" 
-sudo apt-get -y safe-upgrade
-echo "`date` Apt install" 
-sudo apt-get -f install ;
-
-echo "`date` Installing base packages"
-# libopenssl-ruby1.8 ssl-cert 
-sudo apt-get install -y unzip build-essential git-core ruby ruby1.8-dev rubygems ri irb build-essential wget git-core zlib1g-dev libxml2-dev;
-echo "`date` Unchaining rubygems from the tyrrany of ubuntu" 
-sudo gem install --no-rdoc --no-ri rubygems-update --version=1.3.7 ; sudo /var/lib/gems/1.8/bin/update_rubygems; sudo gem update --no-rdoc --no-ri --system ; gem --version ;
-
-echo "`date` Installing wukong and related gems" 
-sudo gem install --no-rdoc --no-ri addressable extlib htmlentities configliere yard wukong right_aws uuidtools cheat
-sudo gem list 
-
-echo "`date` Wukong bootstrap complete: `date`" 
-true

data/old/examples/emr/elastic_mapreduce_example.rb

@@ -1,28 +0,0 @@
-#!/usr/bin/env ruby
-Dir[File.dirname(__FILE__)+'/vendor/**/lib'].each{|dir| $: << dir }
-require 'rubygems'
-require 'wukong/script'
-require 'wukong/script/emr_command'
-
-#
-# * Copy the emr.yaml from here into ~/.wukong/emr.yaml
-#   and edit it to suit.
-# * Download the Amazon elastic-mapreduce runner. Get a copy from
-#   http://elasticmapreduce.s3.amazonaws.com/elastic-mapreduce-ruby.zip
-# * Find out what breaks, fix it or ask us for help (coders@infochimps.org) and
-#   submit a patch
-#
-
-class FooStreamer < Wukong::Streamer::LineStreamer
-  def initialize *args
-    super *args
-    @line_no = 0
-  end
-
-  def process *args
-    yield ["%5d" % @line_no, *args]
-    @line_no += 1
-  end
-end
-
-Wukong::Script.new(FooStreamer, FooStreamer).run

data/old/examples/network_graph/adjacency_list.rb

@@ -1,74 +0,0 @@
-#!/usr/bin/env ruby
-require 'rubygems'
-require 'wukong/script'
-
-#
-# Given an adjacency pairs (from \t to) representation of a directed graph:
-#
-#    1     2
-#    1     7
-#    2     7
-#    2     9
-#    7     2
-#
-# It produces an "adjacency list":http://en.wikipedia.org/wiki/Adjacency_list representation:
-#
-#    1  >   2   7
-#    2  >   7   9
-#    7  >   2
-#    9  >
-#
-# and
-#
-#    1  <
-#    2  <   1   7
-#    7  <   1   2
-#    9  <   2
-#
-# (each column is tab-separated in the actual output)
-#
-#
-#
-module Gen1HoodEdges
-  class Mapper < Wukong::Streamer::Base
-    def process rsrc, src, dest, *_
-      src = src.to_i ; dest = dest.to_i
-      yield [ src,  '>', dest ]
-      yield [ dest, '<', src  ]
-    end
-  end
-
-  #
-  # Accumulate links onto single line.
-  #
-  # The reduce key is the target node and direction; we just stream through all
-  # pairs for each target node and output its neighbor nodes on the same line.
-  #
-  # To control memory usage, we will print directly to the output (and not run
-  # through the Emitter)
-  #
-  class Reducer < Wukong::Streamer::AccumulatingReducer
-    # clear the list of incoming paths
-    def start! target, dir, *args
-      print target + "\t" + dir  # start line with target and list type
-    end
-    def accumulate target, dir, neighbor
-      print "\t" + neighbor      # append neighbor to output, same line
-    end
-    def finalize
-      puts ''                    # start new line
-    end
-  end
-
-  class Script < Wukong::Script
-    def default_options
-      super.merge :sort_fields => 1, :partition_fields => 1
-    end
-  end
-end
-
-# Execute the script
-Gen1HoodEdges::Script.new(
-  Gen1HoodEdges::Mapper,
-  Gen1HoodEdges::Reducer
-  ).run

data/old/examples/network_graph/breadth_first_search.rb

@@ -1,72 +0,0 @@
-#!/usr/bin/env ruby
-require 'rubygems'
-require 'wukong/script'
-
-#
-# Use this script to do a Breadth-First Search (BFS) of a graph.
-#
-# Usage:
-#   ./make_paths --head=[path_in_key] --tail=[path_out_key] --out_rsrc=[combined_path_key]
-#
-# For example, given an edge list in the file '1path.tsv' that looks like
-#   1path       n1      n2
-#   1path       n1      n3
-#   ... and so forth ...
-# you can run
-#   for t in 1 2 3 4 5 6 7 8 9 ; do next=$((t+1)) ; time cat 1path.tsv "${t}path.tsv" | ./make_paths.rb --map --head="1path"  --tail="${t}path" | sort -u | ./make_paths.rb --reduce --out_rsrc="${next}path" | sort -u > "${next}path.tsv" ; done
-# to do a 9-deep breadth-first search.
-#
-module Gen1HoodEdges
-  class Mapper < Wukong::Streamer::RecordStreamer
-    def initialize
-      @head = Settings[:head]
-      @tail = Settings[:tail]
-    end
-    def process rsrc, *nodes
-      yield [ nodes.last,  'i', nodes[0..-2] ] if (rsrc == self.head)
-      yield [ nodes.first, 'o', nodes[1..-1] ] if (rsrc == self.tail)
-    end
-  end
-
-  #
-  # Accumulate ( !!in memory!!) all inbound links onto middle node
-  #
-  # Then for each outbound link, loop over those inbound links and emit the
-  # triple (in, mid,out)
-  #
-  class Reducer < Wukong::Streamer::AccumulatingReducer
-    attr_accessor :paths_in, :out_rsrc
-    def initialize
-      self.out_rsrc = Settings[:out_rsrc]
-    end
-    # clear the list of incoming paths
-    def start! *args
-      self.paths_in  = []
-    end
-    def accumulate mid, dir, *nodes
-      case dir
-      when 'i'
-        self.paths_in << nodes
-        if (self.paths_in.length % 1000 == 0) && (self.paths_in.length > 10000)
-          $stderr.puts ["Accumulating:", mid, self.paths_in.length].join("\t")
-        end
-      when 'o'
-        paths_in.each do |path_in|
-          yield [self.out_rsrc, path_in, mid, *nodes]
-        end
-      end
-    end
-    def finalize
-    end
-    def get_key mid, *_
-      mid
-    end
-  end
-end
-
-# Execute the script
-Wukong.run(
-  Gen1HoodEdges::Mapper,
-  Gen1HoodEdges::Reducer,
-   :sort_fields => 2, :partition_fields => 1
-  )