wukong 0.1.1

data/bin/wu-sum

@@ -0,0 +1,30 @@
+#!/usr/bin/env ruby
+require 'wukong'
+require 'wukong/streamer/summing_reducer'
+
+#
+#
+class Summer < Wukong::Streamer::SummingReducer
+  attr_accessor :sample_line
+
+  def initialize *args
+    self.summing_elements = [0]
+    super *args
+  end
+
+  def start! *args
+    self.sample_line = args
+    super *args
+  end
+
+  def get_key *fields
+    fields.values_at(2,3)
+  end
+
+  def finalize
+    summing_elements.each{|idx| sample_line[idx] = sums[idx]}
+    yield sample_line
+  end
+end
+
+Wukong::Script.new(Summer, nil).run

data/doc/INSTALL.textile

@@ -0,0 +1,41 @@
+---
+layout: default
+title:  Install Notes
+---
+
+
+h1(gemheader). {{ site.gemname }} %(small):: install%
+
+<notextile><div class="toggle"></notextile>
+
+h2. Get the code
+
+This code is available as a gem:
+
+pre. $ sudo gem install mrflip-{{ site.gemname }}
+
+You can instead download this project in either "zip":http://github.com/mrflip/{{ site.gemname }}/zipball/master or "tar":http://github.com/mrflip/{{ site.gemname }}/tarball/master formats.
+
+Better yet, you can also clone the project with "Git":http://git-scm.com by running:
+
+pre. $ git clone git://github.com/mrflip/{{ site.gemname }}
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. Get the Dependencies
+
+* Hadoop, pig
+* extlib, YAML, JSON
+* Optional gems: trollop, addressable/uri, htmlentities
+
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. Setup
+
+1. Allow Wukong to discover where his elephant friend lives: either
+** set a $HADOOP_HOME environment variable,
+** or create a file 'config/wukong-site.yaml' with a line that points to the top-level directory of your hadoop install: @:hadoop_home: /usr/local/share/hadoop@
+2. Add wukong's @bin/@ directory to your $PATH, so that you may use its filesystem shortcuts.
+
+<notextile></div></notextile>

data/doc/LICENSE.textile

@@ -0,0 +1,107 @@
+---
+layout: default
+title:  Apache License
+---
+
+
+h1(gemheader). {{ site.gemname }} %(small):: license%
+
+
+The wukong code is __Copyright (c) 2009 Philip (flip) Kromer__
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+
+h1. Apache License
+
+                              Apache License
+                        Version 2.0, January 2004
+                     http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+<notextile><div class="toggle"></notextile>
+
+h2. 1. Definitions.
+
+* **License** shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+* **Licensor** shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+* **Legal Entity** shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, **control** means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+* **You** (or **Your**) shall mean an individual or Legal Entity exercising permissions granted by this License.
+* **Source** form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+* **Object** form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+* **Work** shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+* **Derivative Works** shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+* **Contribution** shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, **submitted** means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+* **Contributor** shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 2. Grant of Copyright License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 3. Grant of Patent License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 4. Redistribution.
+
+You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+# You must give any other recipients of the Work or Derivative Works a copy of this License; and
+# You must cause any modified files to carry prominent notices stating that You changed the files; and
+# You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+# If the Work includes a __NOTICE__ text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 5. Submission of Contributions.
+
+Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 6. Trademarks.
+
+This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 7. Disclaimer of Warranty.
+
+Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 8. Limitation of Liability.
+
+In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+<notextile></div><div class="toggle"></notextile>
+
+h2. 9. Accepting Warranty or Additional Liability.
+
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+<notextile></div><div class="toggle"></notextile>
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+
+<notextile></div></notextile>

data/doc/README-tutorial.textile

@@ -0,0 +1,163 @@
+Here's a script to count words in a text stream:
+
+  require 'wukong'
+  module WordCount
+    class Mapper < Wukong::Streamer::LineStreamer
+      # Emit each word in the line.
+      def process line
+        words = line.strip.split(/\W+/).reject(&:blank?)
+        words.each{|word| yield [word, 1] }
+      end
+    end
+
+    class Reducer < Wukong::Streamer::ListReducer
+      def finalize
+        yield [ key, values.map(&:last).map(&:to_i).sum ]
+      end
+    end
+  end
+
+  Wukong::Script.new(
+    WordCount::Mapper,
+    WordCount::Reducer
+    ).run # Execute the script
+
+The first class, the Mapper, eats lines and craps @[word, count]@ records. Here
+the /key/ is the word, and the /value/ is its count.  
+
+The second class is an example of an accumulated list reducer. The values for
+each key are stacked up into a list; then the record(s) yielded by @#finalize@
+are emitted.
+
+Here's another way to write the Reducer: accumulate the count of each line, then
+yield the sum in @#finalize@:
+
+    class Reducer2 < Wukong::Streamer::AccumulatingReducer
+      attr_accessor :key_count
+      def start! *args
+        self.key_count = 0
+      end
+      def accumulate(word, count)
+        self.key_count += count.to_i
+      end
+      def finalize
+        yield [ key, key_count ]
+      end
+    end
+
+Of course you can be really lazy (that is, smart) and write your script instead as
+
+  class Script < Wukong::Script
+    def reducer_command
+      'uniq -c'
+    end
+  end
+
+  
+h2. Structured data
+  
+All of these deal with unstructured data.  Wukong also lets you view your data
+as a stream of structured objects.
+
+Let's say you have a blog; its records look like
+
+  Post    = Struct.new( :id, :created_at, :user_id, :title, :body, :link )
+  Comment = Struct.new( :id, :created_at, :post_id, :user_id, :body )
+  User    = Struct.new( :id, :username, :fullname, :homepage, :description )
+  UserLoc = Struct.new( :user_id, :text, :lat, :lng )
+
+You've been using "twitter":http://twitter.com for a long time, and you've
+written something that from now on will inject all your tweets as Posts, and all
+replies to them as Comments (by a common 'twitter_bot' account on your blog).
+What about the past two years' worth of tweets?  Let's assume you're so chatty that
+a Map/Reduce script is warranted to handle the volume.
+
+Cook up something that scrapes your tweets and all replies to your tweets:
+
+  Tweet = Struct.new( :id, :created_at, :twitter_user_id,
+    :in_reply_to_user_id, :in_reply_to_status_id, :text )
+  TwitterUser  = Struct.new( :id, :username, :fullname,
+    :homepage, :location, :description )
+
+Now we'll just process all those in a big pile, converting to Posts, Comments
+and Users as appropriate. Serialize your scrape results so that each Tweet and
+each TwitterUser is a single lines containing first the class name ('tweet' or
+'twitter_user') followed by its constituent fields, in order, separated by tabs.
+
+The RecordStreamer takes each such line, constructs its corresponding class, and
+instantiates it with the 
+  
+  require 'wukong'
+  require 'my_blog' #defines the blog models
+  module TwitBlog
+    class Mapper < Wukong::Streamer::RecordStreamer
+      # Watch for tweets by me
+      MY_USER_ID = 24601
+      # structs for our input objects
+      Tweet = Struct.new( :id, :created_at, :twitter_user_id,
+        :in_reply_to_user_id, :in_reply_to_status_id, :text )
+      TwitterUser  = Struct.new( :id, :username, :fullname,
+        :homepage, :location, :description )
+      #
+      # If this is a tweet is by me, convert it to a Post.
+      #
+      # If it is a tweet not by me, convert it to a Comment that 
+      # will be paired with the correct Post.
+      #
+      # If it is a TwitterUser, convert it to a User record and
+      # a user_location record
+      #
+      def process record
+        case record
+	when TwitterUser
+	  user     = MyBlog::User.new.merge(record) # grab the fields in common
+	  user_loc = MyBlog::UserLoc.new(record.id, record.location, nil, nil)
+	  yield user
+	  yield user_loc
+	when Tweet
+	  if record.twitter_user_id == MY_USER_ID
+	    post = MyBlog::Post.new.merge record
+	    post.link = "http://twitter.com/statuses/show/#{record.id}"
+	    post.body = record.text
+	    post.title = record.text[0..65] + "..."
+	    yield post
+	  else
+   	    comment = MyBlog::Comment.new.merge record
+	    comment.body    = record.text
+	    comment.post_id = record.in_reply_to_status_id
+	    yield comment
+	  end
+	end
+      end
+    end
+  end
+  Wukong::Script.new( TwitBlog::Mapper, nil ).run # identity reducer
+
+h2. Uniqifying
+
+The script above uses the identity reducer: every record from the mapper is sent
+to the output.  But what if you had grabbed the replying user's record every time
+you saw a reply?
+
+Fine, so pass it through @uniq@. But what if a user updated their location or
+description during this time? You'll want to probably use UniqByLastReducer
+
+Location might want to take the most /frequent/, and might want as well to
+geolocate the location text. Use a ListReducer, find the most frequent element,
+then finally call the expensive geolocation method.
+  
+h2. A note about keys
+
+Now we're going to write this using the synthetic keys already extant in the
+twitter records, making the unwarranted assumption that they won't collide with
+the keys in your database.
+
+Map/Reduce paradigm does badly with synthetic keys. Synthetic keys demand
+locality, and map/reduce's remarkable scaling comes from not assuming
+locality. In general, write your map/reduce scripts to use natural keys (the scre
+
+h1. More info
+
+There are many useful examples (including an actually-useful version of this
+WordCount script) in examples/ directory.
+

data/doc/README-wulign.textile

@@ -0,0 +1,59 @@
+h1. wu-lign -- format a tab-separated file as aligned columns
+
+wu-lign will intelligently reformat a tab-separated file into a tab-separated, space aligned file that is still suitable for further processing. For example, given the log-file input
+
+    <pre><code>
+    2009-07-21T21:39:40 day     65536   3.15479 68750   1171316
+    2009-07-21T21:39:45 doing   65536   1.04533 26230   1053956
+    2009-07-21T21:41:53 hapaxlegomenon  65536   0.87574e-05     23707   10051141
+    2009-07-21T21:44:00 concert 500     0.29290 13367   9733414
+    2009-07-21T21:44:29 world   65536   1.09110 32850   200916
+    2009-07-21T21:44:39 world+series    65536   0.49380 9929    7972025
+    2009-07-21T21:44:54 iranelection    65536   2.91775 14592   136342
+    </code></pre>
+
+wu-lign will reformat it to read
+
+    <pre><code>
+    2009-07-21T21:39:40 day                   65536   3.154791234 68750    1171316
+    2009-07-21T21:39:45 doing                 65536   1.045330000 26230    1053956
+    2009-07-21T21:41:53 hapaxlegomenon        65536   0.000008757 23707   10051141
+    2009-07-21T21:44:00 concert                 500   0.292900000 13367    9733414
+    2009-07-21T21:44:29 world                 65536   1.091100000 32850     200916
+    2009-07-21T21:44:39 world+series          65536   0.493800000  9929    7972025
+    2009-07-21T21:44:54 iranelection          65536   2.917750000 14592     136342
+    </code></pre>
+
+The fields are still tab-delimited by exactly one tab -- only spaces are used to pad out fields. You can still use cuttab and friends to manipulate columns.
+
+wu-lign isn't intended to be smart, or correct, or reliable -- only to be useful for previewing and organizing tab-formatted files. In general @wu-lign(foo).split("\t").map(&:strip)@ *should* give output semantically equivalent to its input. (That is, the only changes should be insertion of spaces and re-formatting of numerics.) But still -- reserve its use for human inspection only.
+
+(Note: tab characters in this source code file have been converted to spaces; replace whitespace with tab in the first example if you'd like to play along at home.)
+
+h2. How it works 
+
+Wu-Lign takes the first 1000 lines, splits by TAB characters into fields, and tries to guess the format -- int, float, or string -- for each. It builds a consensus of the width and type for corresponding columns in the chunk.  If a column has mixed numeric and string formats it degrades to :mixed, which is basically treated as :string. If a column has mixed :float and :int elements all of them are formatted as float.
+
+h2. Command-line arguments
+
+You can give sprintf-style positional arguments on the command line that will be applied to the corresponding columns. (Blank args are used for placeholding and auto-formatting is still applied).  So with the example above,
+
+    @cat foo | wu-lign  '' '' '' '%8.4e'@
+
+will format the fourth column with "%8.4e", while the first three columns and fifth-and-higher columns are formatted as usual.
+
+    <pre><code>
+    ...
+    2009-07-21T21:39:45 doing           65536   1.0453e+00      26230    1053956
+    2009-07-21T21:41:53 hapaxlegomenon  65536   8.7574e-06      23707   10051141
+    2009-07-21T21:44:00 concert           500   2.9290e-01      13367    9733414
+    ....
+    </code></pre>
+
+h2. Notes
+
+* It has no knowledge of header rows. An all-text first line will screw everything up.
+* It also requires a unanimous vote. One screwy line can coerce the whole mess to :mixed; width formatting will still be applied, though.
+* It won't set columns wider than 70 chars -- this allows for the occasional super-wide column without completely breaking your screen.
+* For :float values, wu-lign tries to guess at the right number of significant digits to the left and right of the decimal point.
+* wu-lign does not parse 'TSV files' in their strict sense -- there is no quoting or escaping; every tab delimits a field, every newline a record.

data/doc/README-wutils.textile

@@ -0,0 +1,128 @@
+h1. Wukong Utility Scripts
+
+h2. Stupid command-line tricks
+
+h3. Histogram
+
+Given data with a date column:
+
+   message	235623	20090423012345	Now is the winter of our discontent Made glorious summer by this son of York
+   message	235623	20080101230900	These pretzels are making me THIRSTY!
+   ...
+
+You can calculate number of messages sent by day with
+
+    cat messages | cuttab 3 | cutc 8 | sort | uniq -c
+
+(see the wuhist command, below.)
+
+h3. Simple intersection, union, etc
+
+For two datasets (batch_1 and batch_2) with unique entries (no repeated lines),
+
+* Their union is simple:
+
+      cat batch_1 batch_2 | sort -u
+
+
+* Their intersection:
+
+      cat batch_1 batch_2 | sort | uniq -c | egrep -v '^ *1 '
+
+  This concatenates the two sets and filters out everything that only occurred once.
+
+* For the complement of the intersection, use "... | egrep '^ *1 '"
+  
+* In both cases, if the files are each internally sorted, the commandline sort takes a --merge flag:
+
+      sort --merge -u batch_1 batch_2 
+
+h2. Command Listing
+
+h3. cutc
+
+@cutc [colnum]@
+
+Ex.
+
+  echo -e 'foo\tbar\tbaz' | cutc 6
+  foo	ba
+
+Cuts from beginning of line to given column (default 200). A tab is one character, so right margin can still be ragged.
+ 
+h3. cuttab
+
+  @cuttab [colspec]@
+
+Cuts given tab-separated columns. You can give a comma separated list of numbers
+or ranges 1-4. columns are numbered from 1.
+
+Ex.
+
+  echo -e 'foo\tbar\tbaz' | cuttab 1,3
+  foo	baz
+
+h3. hdp-*
+
+These perform the corresponding commands on the HDFS filesystem.  In general,
+where they accept command-line flags, they go with the GNU-style ones, not the
+hadoop-style: so, @hdp-du -s dir@ or @hdp-rm -r foo/@
+
+* @hdp-cat@
+* @hdp-catd@ -- cats the files that don't start with '_' in a directory. Use this for a pile of @.../part-00000@ files
+* @hdp-du@
+* @hdp-get@
+* @hdp-kill@
+* @hdp-ls@
+* @hdp-mkdir@
+* @hdp-mv@
+* @hdp-ps@
+* @hdp-put@
+* @hdp-rm@
+* @hdp-sync@
+
+h3. hdp-sort, hdp-stream, hdp-stream-flat
+
+* @hdp-sort@
+* @hdp-stream@
+* @hdp-stream-flat@
+
+    <code><pre>
+    hdp-stream input_filespec output_file map_cmd reduce_cmd num_key_fields
+    </pre></code>
+
+h3. tabchar
+
+Outputs a single tab character.
+ 
+h3. wuhist
+
+Occasionally useful to gather a lexical histogram of a single column:
+
+Ex.
+
+    <code><pre>
+    $ echo -e 'foo\nbar\nbar\nfoo\nfoo\nfoo\n7' | ./wuhist
+    4       foo
+    2       bar
+    1       7
+    </pre></code>
+
+(the output will have a tab between the first and second column, for futher processing.)
+
+h3. wulign
+
+Intelligently format a tab-separated file into aligned columns (while remaining tab-separated for further processing). See README-wulign.textile.
+ 
+h3. hdp-parts_to_keys.rb
+
+A *very* clumsy script to rename reduced hadoop output files by their initial key.
+
+If your output file has an initial key in the first column and you pass it
+through hdp-sort, they will be distributed across reducers and thus output
+files. (Because of the way hadoop hashes the keys, there's no guarantee that
+each file will get a distinct key. You could have 2 keys with a million entries
+and they could land sequentially on the same reducer, always fun.)
+
+If you're willing to roll the dice, this script will rename files according to
+the first key in the first line.