wukong 3.0.0.pre → 3.0.0.pre2

data/docpages/intro.textile

@@ -1,8 +0,0 @@
----
-layout: default
-title:  mrflip.github.com/wukong
-collapse: false
----
-h1(gemheader). Intro %(small):: 3 simple examples%
-
-{% include intro.textile %}

data/docpages/moreinfo.textile

@@ -1,174 +0,0 @@
----
-layout: default
-title:  mrflip.github.com/wukong - TODO
-collapse: false
----
-
-
-h1(gemheader). Wukong More Info
-
-** "Why is it called Wukong?":#name
-** "Don't Use Wukong, use this instead":#whateverdude
-** "Further Reading and useful links":#links
-** "Note on Patches/Pull Requests":#patches
-** "What's up with Wukong::AndPig?":#andpig
-** "Map/Reduce Algorithms":#algorithms
-** "TODOs":#TODO                
-
-
-<notextile><div class="toggle"></notextile>
-
-h2(#name). Why is it called Wukong?
-
-Hadoop, as you may know, is "named after a stuffed elephant.":http://en.wikipedia.org/wiki/Hadoop Since Wukong was started by the "infochimps":http://infochimps.org team, we needed a simian analog.  A Monkey King who journeyed to the land of the Elephant seems to fit the bill:
-
-bq. Sun Wukong (孙悟空), known in the West as the Monkey King, is the main character in the classical Chinese epic novel Journey to the West. In the novel, he accompanies the monk Xuanzang on the journey to retrieve Buddhist sutras from India.
-
-bq. Sun Wukong possesses incredible strength, being able to lift his 13,500 jīn (8,100 kg) Ruyi Jingu Bang with ease. He also has superb speed, traveling 108,000 li (54,000 kilometers) in one somersault. Sun knows 72 transformations, which allows him to transform into various animals and objects; he is, however, shown with slight problems transforming into other people, since he is unable to complete the transformation of his tail. He is a skilled fighter, capable of holding his own against the best generals of heaven. Each of his hairs possesses magical properties, and is capable of transforming into a clone of the Monkey King himself, or various weapons, animals, and other objects. He also knows various spells in order to command wind, part water, conjure protective circles against demons, freeze humans, demons, and gods alike. -- ["Sun Wukong's Wikipedia entry":http://en.wikipedia.org/wiki/Wukong]
-
-The "Jaime Hewlett / Damon Albarn short":http://news.bbc.co.uk/sport1/hi/olympics/monkey that the BBC made for their 2008 Olympics coverage gives the general idea.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#algorithms). Map/Reduce Algorithms
-
-Example graph scripts:
-
-* Multigraph
-* Pagerank 		(done)
-* Breadth-first search  
-* Triangle enumeration  
-* Clustering
-
-h3. K-Nearest Neighbors
-
-More example hadoop algorithms:
-* Bigram counts: http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/exercises/bigrams.html
-* Inverted index construction: http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/exercises/indexer.html
-* Pagerank : http://www.umiacs.umd.edu/~jimmylin/cloud9/docs/exercises/pagerank.html
-* SIPs, Median, classifiers and more : http://matpalm.com/
-* Brad Heintz's "Distributed Computing with Ruby":http://www.bradheintz.com/no1thing/talks/ demonstrates Travelling Salesman in map/reduce.
-
-* "Clustering billions of images with large scale nearest neighbor search":http://scholar.google.com/scholar?cluster=2473742255769621469&hl=en uses three map/reduce passes:
-** Subsample to build a "spill tree" that roughly localizes each object
-** Use the spill tree on the full dataset to group each object with its potential neighbors
-** Calculate the metrics and emit only the k-nearest neighbors
-
-Example example scripts (from http://www.cloudera.com/resources/learning-mapreduce): 
-
-1. Find the [number of] hits by 5 minute timeslot for a website given its access logs.
-2. Find the pages with over 1 million hits in day for a website given its access logs.
-3. Find the pages that link to each page in a collection of webpages.
-4. Calculate the proportion of lines that match a given regular expression for a collection of documents.
-5. Sort tabular data by a primary and secondary column.
-6. Find the most popular pages for a website given its access logs.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#whateverdude). Don't Use Wukong, use this instead
-
-There are several worthy Hadoop|Streaming Frameworks:
-
-* infochimps.org's "Wukong":http://github.com/mrflip/wukong -- ruby; object-oriented *and* record-oriented
-* NYTimes' "MRToolkit":http://code.google.com/p/mrtoolkit/ -- ruby; much more log-oriented
-* Freebase's "Happy":http://code.google.com/p/happy/ -- python; the most performant, as it can use Jython to make direct API calls.
-* Last.fm's "Dumbo":http://wiki.github.com/klbostee/dumbo -- python
-
-Most people use Wukong / one of the above (or straight Java Hadoop, poor souls) for heavy lifting, and several of the following hadoop tools for efficiency:
-
-* Pig  OR
-* Hive -- hive is more SQL-ish, Pig is more elegant (in a brushed-metal kind of way).  I greatly prefer Pig, because I hate SQL; you may feel differently.
-* Sqoop
-* Mahout
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#links). Further Reading and useful links:
-
-* "Ruby Hadoop Quickstart":http://blog.pdatasolutions.com/post/191978092/ruby-on-hadoop-quickstart - dive right in with Wukong, Hadoop and the Amazon Elastic MapReduce cloud. Once you get bored with the command line, this is the fastest path to Wukong power.
-* "Distributed Computing with Ruby":http://www.bradheintz.com/no1thing/talks/ has some raw ruby, some Wukong and some JRuby/Hadoop integration -- it demonstrates a Travelling Salesman in map/reduce. Cool!
-
-* "Hadoop, The Definitive Guide":http://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/0596521979
-
-* "Running Hadoop On Ubuntu Linux (Single-Node Cluster)":http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster) and "unning Hadoop On Ubuntu Linux (Multi-Node Cluster).":http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
-* "Running Hadoop MapReduce on Amazon EC2 and S3":http://developer.amazonwebservices.com/connect/entry.jspa?externalID=873
-
-* "Hadoop Overview by Doug Cutting":http://video.google.com/videoplay?docid=-4912926263813234341 - the founder of the Hadoop project. (49m video)
-
-* "Cluster Computing and Map|Reduce":http://www.youtube.com/results?search_query=cluster+computing+and+mapreduce
-** "Lecture 1: Overview":http://www.youtube.com/watch?v=yjPBkvYh-ss
-** "Lecture 2 (technical): Map|Reduce":http://www.youtube.com/watch?v=-vD6PUdf3Js
-** "Lecture 3 (technical): GFS (Google File System)":http://www.youtube.com/watch?v=5Eib_H_zCEY
-** "Lecture 4 (theoretical): Canopy Clustering":http://www.youtube.com/watch?v=1ZDybXl212Q
-** "Lecture 5 (theoretical): Breadth-First Search":http://www.youtube.com/watch?v=BT-piFBP4fE
-
-* "Cloudera Hadoop Training:":http://www.cloudera.com/hadoop-training
-** "Thinking at Scale":http://www.cloudera.com/hadoop-training-thinking-at-scale
-** "Mapreduce and HDFS":http://www.cloudera.com/hadoop-training-mapreduce-hdfs
-** "A Tour of the Hadoop Ecosystem":http://www.cloudera.com/hadoop-training-ecosystem-tour
-** "Programming with Hadoop":http://www.cloudera.com/hadoop-training-programming-with-hadoop
-** "Hadoop and Hive: introduction":http://www.cloudera.com/hadoop-training-hive-introduction
-** "Hadoop and Hive: tutorial":http://www.cloudera.com/hadoop-training-hive-tutorial
-** "Hadoop and Pig: Introduction":http://www.cloudera.com/hadoop-training-pig-introduction
-** "Hadoop and Pig: Tutorial":http://www.cloudera.com/hadoop-training-pig-tutorial
-** "Mapreduce Algorithms":http://www.cloudera.com/hadoop-training-mapreduce-algorithms
-** "Exercise: Getting started with Hadoop":http://www.cloudera.com/hadoop-training-exercise-getting-started-with-hadoop
-** "Exercise: Writing mapreduce programs":http://www.cloudera.com/hadoop-training-exercise-writing-mapreduce-programs
-** "Cloudera Blog":http://www.cloudera.com/blog/
-
-* "Hadoop Wiki: Hadoop Streaming":http://wiki.apache.org/hadoop/HadoopStreaming
-* "Hadoop Docs: Hadoop Streaming":http://hadoop.apache.org/common/docs/current/streaming.html
-
-* A "dimwitted screed on Ruby, Hadoop and Starling":http://www.theregister.co.uk/2008/08/11/hadoop_dziuba/ seemingly written with jockstrap on head.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#patches). Note on Patches/Pull Requests
-
-* Fork the project.
-* Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a future version unintentionally.
-* Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but  bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#andpig). What's up with Wukong::AndPig?
-
-@Wukong::AndPig@ is a small library to more easily generate code for the "Pig":http://hadoop.apache.org/pig data analysis language.  See its "README":http://github.com/mrflip/wukong/tree/master/lib/wukong/and_pig/README.textile for more.
-
-It's **not really being worked on**, and you should probably **ignore it**.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#todo). TODOs
-
-Utility
-
-* columnizing / reconstituting
-
-* Set up with JRuby
-* Allow for direct HDFS operations
-* Make the dfs commands slightly less stupid
-* add more standard options
-* Allow for combiners
-* JobStarter / JobSteps
-* might as well take dumbo's command line args
-
-BUGS:
-
-* Can't do multiple input files in local mode
-
-Patterns to implement:
-
-* Stats reducer
-** basic sum, avg, max, min, std.dev of a numeric field
-** the "running standard deviation":http://www.johndcook.com/standard_deviation.html
-
-* Efficient median (and other order statistics)
-
-* Make StructRecordizer work generically with other reducers (spec. AccumulatingReducer)
-
-Make wutils: tsv-oriented implementations of the coreutils (eg uniq, sort, cut, nl, wc, split, ls, df and du) to instrinsically accept and emit tab-separated records.
-
-<notextile></div></notextile>

data/docpages/news.html

@@ -1,24 +0,0 @@
----
-layout: default
-title:  edamame news
-collapse: true
----
-<h1 class="gemheader">{% if site.gemname %}{{ site.gemname }}{% else %}mrflip{% endif %}<span class="small">:: news</span></h1>
-
-<div id="news">
-  {% for t in site.posts %} {% assign has_posts = true %}{% endfor %}{% if has_posts %}
-  {% for post in site.posts %}
-  <div class="toggle" id="news-{{ post.id }}">
-    
-    <h2><a href="{{ post.url }}">{{ post.title }}</a><span class="postdate"> &raquo; {{ post.date | date_to_string }}</span></h2>
-    
-    {{ post.content }}
-    
-  </div>
-  {% endfor %}
-  {% else %}
-  <p class="heavy">
-    <em>(no news. good news?)</em>
-  </p>
-  {% endif %}
-</div>

data/docpages/pig/PigLatinExpressionsList.txt

@@ -1,122 +0,0 @@
-
-h2. RelationalOperators
-
-  * foreach
-
-  * cogroup
-  * group
-  * join
-
-  * cross
-  
-  * distinct
-  * filter
-  * limit
-  * order
-  * split
-  * union
-
-  * load
-  * store
-  
-h2. Streaming Operator
-  * stream
-
-h2. UDF Statements
-  * define
-  * register
-
-h2. Diagnostic Statements
-  * describe
-  * dump
-  * explain
-  * illustrate
-
-h2. Built-in Functions
-  * EvalFunctions
-  * AVG
-  * CONCAT
-  * COUNT
-  * DIFF
-  * MIN
-  * MAX
-  * SIZE
-  * SUM
-  * TOKENIZE
-  * Load/StoreFunctions
-  * BinaryDeserializer
-  * BinarySerializer
-  * BinStorage
-  * PigStorage
-  * PigDump
-  * TextLoader
-
-h1. Operators
-  * ArithmeticOperators
-    - addition+
-    - subtraction-
-    - multiplication*
-    - division/
-    - modulo%
-    - bincond?
-  * ComparisonOperators
-    - Equal==
-    - notequal!=
-    - lessthan<
-    - greaterthan>
-    - lessthanorequalto<=
-    - greaterthanorequalto>=
-    - patternmatchingmatches
-  * NullOperators
-    - isnull
-    - isnotnull
-  * BooleanOperators
-    - and
-    - or
-    - not
-  * DereferenceOperators
-    - tupledereference.
-    - mapdereference#
-  * SignOperators
-    - positive+
-    - negative-
-  * CastOperators
-    - (type)$0
-    - (type)alias
-    - Nulls
-    - Constants
-    - Expressions
-    - Schemas
-    - Keywords
-
-h1. DataTypes 
-  h2. SimpleDataTypes
-    - int
-    - long
-    - double
-    - arrays
-    - chararray
-    - bytearray
-  h2. ComplexDataTypes
-    - tuple
-    - bag
-    - map
-
-h1. FileCommands
-  * cat
-  * cd
-  * copyFromLocal
-  * copyToLocal
-  * cp
-  * ls
-  * mkdir
-  * mv
-  * pwd
-  * rm
-  * rmf
-
-h1. UtilityCommands
-  * help
-  * kill
-  * quit
-  * set

data/docpages/pig/PigLatinReferenceManual.txt

@@ -1,1640 +0,0 @@
-# ---------------------------------------------------------------------------
-# 
-# = CROSS
-# 
-# Computes the cross product of two or more relations.
-# 
-# == Syntax
-# 
-#   alias = CROSS alias, alias [, alias …] [PARALLEL n];
-#   
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# PARALLEL n::
-#   Increase the parallelism of a job by specifying the number of reduce tasks,
-#   n. The optimal number of parallel tasks depends on the amount of memory on
-#   each node and the memory required by each of the tasks. To determine n, use
-#   the following as a general guideline:
-#     n = (nr_nodes - 1) * 0.45 * nr_GB
-#   where nr_nodes is the number of nodes used and nr_GB is the amount of physical
-#   memory on each node.
-# 
-#   Note the following:
-#   * Parallel only affects the number of reduce tasks. Map parallelism is
-#     determined by the input file, one map for each HDFS block.
-#   * If you don’t specify parallel, you still get the same map parallelism but
-#     only one reduce task.
-# 
-# == Usage
-# 
-# Use the CROSS operator to compute the cross product (Cartesian product) of two
-# or more relations.
-# 
-# CROSS is an expensive operation and should be used sparingly.
-# 
-# == Example
-# 
-# Suppose we have relations A and B.
-# 
-#   (A)           (B)
-#   ----------- --------   
-#   (1, 2, 3)    (2, 4)    
-#   (4, 2, 1)    (8, 9)                
-#   	     (1, 3)
-# 
-# In this example the cross product of relation A and B is computed.
-# 
-#    	X = CROSS A, B;
-# 
-# Relation X looks like this.
-# 
-#   (1, 2, 3, 2, 4)
-#   (1, 2, 3, 8, 9)
-#   (1, 2, 3, 1, 3)
-#   (4, 2, 1, 2, 4)
-#   (4, 2, 1, 8, 9)
-#   (4, 2, 1, 1, 3)
-#
-
-
-# ---------------------------------------------------------------------------
-# 
-# DISTINCT
-# 
-# Removes duplicate tuples in a relation.
-# 
-# == Syntax
-# 
-# alias = DISTINCT alias [PARALLEL n];             
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# PARALLEL n::
-#   Increase the parallelism of a job by specifying the number of reduce tasks,
-#   n. The optimal number of parallel tasks depends on the amount of memory on
-#   each node and the memory required by each of the tasks. To determine n, use
-#   the following as a general guideline:
-#     n = (nr_nodes - 1) * 0.45 * nr_GB
-#   where nr_nodes is the number of nodes used and nr_GB is the amount of physical
-#   memory on each node.
-# 
-#   Note the following:
-#   * Parallel only affects the number of reduce tasks. Map parallelism is
-#     determined by the input file, one map for each HDFS block.
-#   * If you don’t specify parallel, you still get the same map parallelism but
-#     only one reduce task.
-# 
-# == Usage
-# 
-# Use the DISTINCT operator to remove duplicate tuples in a relation. DISTINCT
-# does not preserve the original order of the contents (to eliminate duplicates,
-# Pig must first sort the data). You cannot use DISTINCT on a subset of fields. To
-# do this, use FOREACH … GENERATE to select the fields, and then use DISTINCT.
-# 
-# == Example
-# 
-# Suppose we have relation A.
-# 
-#      (A)       
-#   ---------
-#   (8, 3, 4)
-#   (1, 2, 3)                        
-#   (4, 3, 3)                                                                                    
-#   (4, 3, 3)
-#   (1, 2, 3)
-# 
-# In this example all duplicate tuples are removed.
-# 
-#    	X = DISTINCT A;
-# 
-# Relation X looks like this.
-# 
-#   (1, 2, 3)
-#   (4, 3, 3)
-#   (8, 3, 4)
-#
-
-# ---------------------------------------------------------------------------
-# 
-# FILTER
-# 
-# Selects tuples (rows) from a relation based on some condition.
-# 
-# == Syntax
-# 
-# alias = FILTER alias  BY expression;
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# BY::
-#   Required keyword.
-# 
-# expression::
-#   An expression.
-# 
-# == Usage
-# 
-# Use the FILTER operator to work with tuples (rows) of data. FILTER is commonly
-# used to select the data that you want; or, conversely, to filter out (remove)
-# the data you don’t want.
-# 
-# Note: If you want to work with specific fields (columns) of data, use the
-# FOREACH …GENERATE operation.
-# 
-# == Examples
-# 
-# Suppose we have relation A.
-# 
-#   (A: f1:int, f2:int, f3:int)
-#   ----------------
-#   (1, 2, 3)
-#   (4, 2, 1)
-#   (8, 3, 4)
-#   (4, 3, 3)
-#   (7, 2, 5)
-#   (8, 4, 3)
-# 
-# In this example the condition states that if the third field equals 3, then add the tuple to relation X.
-# 
-#    	X = FILTER A BY f3 == 3;
-# 
-# Relation X looks like this.
-# 
-#   (1, 2, 3)
-#   (4, 3, 3)
-#   (8, 4, 3)
-# 
-# In this example the condition states that if the first field equals 8 or if the sum of fields f2 and f3 is not greater than first field, then add the tuple to relation X.
-# 
-#    	X = FILTER A BY (f1 == 8) OR (NOT (f2+f3 > f1));
-# 
-# Relation X looks like this.
-# 
-#   (4, 2, 1)
-#   (8, 3, 4)
-#   (7, 2, 5)
-#   (8, 4, 3)
-#
-
-# ---------------------------------------------------------------------------
-# 
-# FOREACH … GENERATE
-# 
-# Generates data transformations based on fields (columns) of data.
-# 
-# == Syntax
-# 
-# alias  = FOREACH { gen_blk | nested_gen_blk } [AS schema];
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# gen_blk::
-#   FOREACH … GENERATE used with a non-nested relation. Use this syntax:
-#   
-#     alias = FOREACH alias GENERATE expression [expression ….]
-# 
-# nested_gen_blk::
-#   FOREACH … GENERATE used with a nested relation. Use this syntax:
-# 
-#     alias = FOREACH nested_alias {
-#        alias = nested_op; [alias = nested_op; …]
-#        GENERATE expression [expression ….]
-#     };
-# 
-#   where:
-#   * The nested block is enclosed in opening and closing brackets { … }.
-#   * The GENERATE keyword must be the last statement within the nested block.
-# 
-# expression::
-#   An expression.
-# 
-# nested_alias::
-#   If one of the fields (columns) in a relation is a bag, the bag can be treated
-#   as an inner or a nested relation.
-# 
-# nested_op::
-#   Allowable operations include FILTER, ORDER, and DISTINCT.
-# 
-#   The FOREACH … GENERATE operation itself is not allowed since this could lead
-#   to an arbitrary number of nesting levels.
-# 
-# AS::
-#   Keyword.
-# 
-# schema::
-#   A schema using the AS keyword (see Schemas).
-# 
-# * If the FLATTEN keyword is used, enclose the schema in parentheses.
-# 
-# * If the FLATTEN keyword is not used, don't enclose the schema in parentheses.
-# 
-# == Usage
-# 
-# Use the FOREACH …GENERATE operation to work with individual fields (columns) of data. The FOREACH …GENERATE operation works with non-nested and nested relations.
-# 
-# A statement with a non-nested relation A could look like this.
-# 
-# X = FOREACH A GENERATE f1;
-# 
-# A statement with a nested relation A could look like this.
-# 
-# X = FOREACH B {
-# 
-#             S = FILTER A by 'xyz';
-# 
-#             GENERATE COUNT (S.$0);
-# 
-# }
-# 
-# Note: FOREACH … GENERATE works with fields (columns) of data. If you want to work with entire tuples (rows) of data, use the FILTER operation.
-# 
-# == Examples
-# 
-# Suppose we have relations A and B, and derived relation C (where C = COGROUP A BY a1 INNER, B BY b1 INNER;).
-# 
-# (A: a1:int, a2:int, a3:int) 
-# -----------------                  
-# (1, 2, 3)                        
-# (4, 2, 1)            
-# (8, 3, 4)            
-# (4, 3, 3)                        
-# (7, 2, 5)                        
-# (8, 4, 3)                        
-#                        
-# 	
-# (B: b1:int, b2:int)
-# ---------------        
-# (2, 4)                
-# (8, 9)                
-# (1, 3)                
-# (2, 7)
-# (2, 9)
-# (4, 6)
-# (4, 9)
-# 	
-#  (C: c1, c2, c3)
-# ---------------------
-#  (1, {(1, 2, 3)}, {(1, 3)})
-#  (4, {(4, 2, 1), (4, 3, 3)}, {(4, 6), (4, 9)})
-#  (8, {(8, 3, 4), (8, 4, 3)}, {(8, 9)})
-#  
-# 
-# == Example: Projection
-# 
-# In this example the asterisk (*) is used to project all fields from relation A to relation X (this is similar to SQL Select *). Relation A and X are identical.
-# 
-# X = FOREACH A GENERATE *;
-# 
-# In this example two fields from relation A are projected to form relation X.
-# 
-# X = FOREACH A GENERATE a1, a2;
-# 
-# Relation X looks this.
-# 
-# (1, 2)
-# (4, 2)
-# (8, 3)
-# (4, 3)
-# (7, 2)
-# (8, 4)
-# == Example: Nested Projection
-# 
-# Note: See GROUP for information about the "group" field in relation C.
-# 
-# In this example if one of the fields in the input relation is a tuple, bag or map, we can perform projection on that field.
-# 
-# X = FOREACH C GENERATE group, B.b2;
-# 
-# Relation X looks like this.
-# 
-# (1, {(3)})
-# (4, {(6), (9)})
-# (8, {(9)})
-# 
-# In this example multiple nested columns are retained.
-# 
-# X = FOREACH C GENERATE group, A.(a1, a2);
-# 
-# Relation X looks like this.
-# 
-# (1, {(1, 2)})
-# (4, {(4, 2), (4, 3)})
-# (8, {(8, 3), (8, 4)})
-# == Example: Schema
-# 
-# In this example two fields in relation A are summed to form relation X. A schema is defined for the projected field.
-# 
-# X = FOREACH A GENERATE a1+a2 AS f1:int;
-# 
-# Y = FILTER X by f1 > 10;
-# 
-# Relations X and Y look this.
-# 
-# (X)        (Y)
-# -----       ------
-# (3)        (11)
-# (6)        (12)
-# (11)
-# (7)
-# (9)
-# (12)
-# 
-# == Example: Applying Functions
-# 
-# Note: See GROUP for information about the "group" field in relation C.
-# 
-# In this example the built-in function SUM() is used to sum a set of numbers in a bag.
-# 
-# X = FOREACH C GENERATE group, SUM (A.a1);
-# 
-# Relation X looks like this.
-# 
-# (1, 1)
-# (4, 8)
-# (8, 16)
-# == Example: Flattening
-# 
-# Note: See GROUP for information about the "group" field in relation C.
-# 
-# In this example the FLATTEN keyword is used to eliminate nesting.
-# 
-# X = FOREACH C GENERATE group, FLATTEN(A);
-# 
-# Relation X looks like this.
-# 
-# (1, 1, 2, 3)
-# (4, 4, 2, 1)
-# (4, 4, 3, 3)
-# (8, 8, 3, 4)
-# (8, 8, 4, 3)
-# 
-# Another FLATTEN example.
-# 
-# X = FOREACH C GENERATE GROUP, FLATTEN(A.a3);
-# 
-# Relation X looks like this.
-# 
-# (1, 3)
-# (4, 1)
-# (4, 3)
-# (8, 4)
-# (8, 3)
-# 
-# Another FLATTEN example.
-# 
-# X = FOREACH C GENERATE FLATTEN(A.(f1, f2)), FLATTEN(B.$1);
-# 
-# Relation X looks like this. Note that for the group '4' in C, there are two tuples in each bag. Thus, when both bags are flattened, the cross product of these tuples is returned; that is, tuples (4, 2, 6), (4, 3, 6), (4, 2, 9), and (4, 3, 9).
-# 
-# (1, 2, 3)
-# (4, 2, 6)
-# (4, 3, 6)
-# (4, 2, 9)
-# (4, 3, 9)
-# (8, 3, 9)
-# (8, 4, 9)
-# 
-# == Example: Nested Block
-# 
-# Suppose we have relation A and derived relation B (where B = GROUP A BY url;). Since relation B contains tuples with bags it can be treated as a nested relation.
-# 
-# A (url:chararray, outlink:chararray)          
-# ---------------------------------------------  
-# (www.ccc.com,www.hjk.com)     
-# (www.ddd.com,www.xyz.org)     
-# (www.aaa.com,www.cvn.org)                     
-# (www.www.com,www.kpt.net)
-# (www.www.com,www.xyz.org)
-# (www.ddd.com,www.xyz.org)
-#  
-# 
-# B
-# ---------------------------------------------  
-#  (www.aaa.com,{(www.aaa.com,www.cvn.org)})
-#  (www.ccc.com,{(www.ccc.com,www.hjk.com)})
-#  (www.ddd.com,{(www.ddd.com,www.xyz.org),(www.ddd.com,www.xyz.org)})
-#  (www.www.com,{(www.www.com,www.kpt.net),(www.www.com,www.xyz.org)})
-# 
-# In this example we perform two of the allowed Pig operations, FILTER (FA) and DISTINCT (DA), as well as projection (PA). Note that the last statement in the nested block must be GENERATE.
-# 
-# X = foreach B {
-#         FA= FILTER A BY outlink == 'www.xyz.org';
-#         PA = FA.outlink;
-#         DA = DISTINCT PA;
-#         GENERATE GROUP, COUNT(DA);
-# }
-# 
-# Relation X looks like this.
-# 
-# (www.ddd.com,1L)
-# (www.www.com,1L)
-
-
-# ---------------------------------------------------------------------------
-# 
-# GROUP
-# 
-# Groups the data in a single relation.
-# 
-# == Syntax
-# 
-#   alias = GROUP alias
-#   	    [BY {[field_alias [, field_alias]] | * | [expression] } ]
-#             [ALL] [PARALLEL n];
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# BY::
-#   Keyword. Use this clause to group the relation by fields or by expression.
-# 
-# field_alias::
-#   The name of a field in a relation. This is the group key or key field.
-# 
-#   A relation can be grouped by a single field (f1) or by the composite value of
-#   multiple fields (f1,f2).
-# 
-# *::
-#   The asterisk. A designator for all fields in the relation.
-# 
-# expression::
-#   An expression.
-# 
-# ALL::
-#   Keyword. Use ALL if you want all tuples to go to a single group; for example, when doing aggregates across entire relations.
-# 
-# PARALLEL n::
-#   Increase the parallelism of a job by specifying the number of reduce tasks,
-#   n. The optimal number of parallel tasks depends on the amount of memory on
-#   each node and the memory required by each of the tasks. To determine n, use
-#   the following as a general guideline:
-#     n = (nr_nodes - 1) * 0.45 * nr_GB
-#   where nr_nodes is the number of nodes used and nr_GB is the amount of physical
-#   memory on each node.
-# 
-#   Note the following:
-#   * Parallel only affects the number of reduce tasks. Map parallelism is
-#     determined by the input file, one map for each HDFS block.
-#   * If you don’t specify parallel, you still get the same map parallelism but
-#     only one reduce task.
-# 
-# == Usage
-# 
-# The GROUP operator groups together tuples that have the same group key (key
-# field). The result of a GROUP operation is a relation that includes one tuple
-# per group. This tuple contains two fields:
-# 
-# * The first field is named "group" (do not confuse this with the GROUP operator)
-#   and is the same type of the group key.
-# 
-# * The second field takes the name of the original relation and is type bag.
-# 
-# Suppose we have the following data:
-# 
-#   john      25      3.6
-#   george  25      2.9
-#   anne     27      3.9
-#   julia      28      3.6
-# 
-# And, suppose we perform the LOAD and GROUP statements shown below. We can use
-# the DESCRIBE operator to view the schemas for relation Y. We can use DUMP to
-# view the contents of Y.
-# 
-# Note that relation Y has two fields. The first field is named "group" and is
-# type int (the same as age). The second field takes the name of the original
-# relation "X" and is type bag (that can contain tuples with three elements of
-# type chararray, int, and float).
-# 
-# Statements
-# 
-#   X = LOAD 'data AS (name:chararray, age:int, gpa:float);
-#   Y = GROUP X BY age;
-#   DESCRIBE Y;
-#   Y: {group: int,X: {name: chararray,age: int,gpa: float}}
-#   DUMP Y;
-# 
-#   (25,{(john,25,3.6F),(george,25,2.9F)})
-#   (27,{(anne,27,3.9F)})
-#   (28,{(julia,28,3.6F)})
-#  
-# As shown in this FOREACH statement, we can refer to the fields in relation Y by their names "group" and "X".
-#   
-#   Z = FOREACH Y GENERATE group, COUNT(X);
-# 
-# Relation Z looks like this.
-# 
-#   (25,2L)
-#   (27,1L)
-#   (28,1L)
-# 
-# == Examples
-# 
-# Suppose we have relation A.
-#   
-#   A: (owner:chararray, pet:chararray)
-#   -----------------
-#   (Alice, turtle)
-#   (Alice, goldfish)
-#   (Alice, cat)
-#   (Bob, dog)
-#   (Bob, cat)
-# 
-# In this example tuples are grouped using the field "owner."
-# 
-#   X = GROUP A BY owner;
-# 
-# Relation X looks like this. "group" is the name of the first field. "A" is the
-# name of the second field.
-#   
-#   (Alice, {(Alice, turtle), (Alice, goldfish)})
-#   (Bob, {(Bob, dog), (Bob, cat)})
-# 
-# In this example tuples are grouped using the ALL keyword. Field "A" is then
-# counted and projected to from relation Y.
-#     
-#   X = GROUP A ALL;
-#   Y = FOREACH X GENERATE COUNT(A);
-# 
-# Relation X looks like this. "group" is the name of the first field. "A" is the
-# name of the second field.
-# 
-#   (all,{(Alice,turtle),(Alice,goldfish),(Alice,cat),(Bob,dog),(Bob,cat)})
-# 
-# Relation Y looks like this.
-# 
-#   (5L)
-# 
-# Suppose we have relation S.
-#   
-#   S: (f1:chararay, f2:int, f3:int)
-#   -----------------
-#   (r1, 1, 2)
-#   (r2, 2, 1)
-#   (r3, 2, 8)
-#   (r4, 4, 4)
-# 
-# In this example tuples are grouped using an expression, f2*f3.
-# 
-#   X = GROUP S BY f2*f3;
-# 
-# Relation Y looks like this. The first field is named "group". The second field is named "S".
-# 
-#   (2, {(r1, 1, 2), (r2, 2, 1)})
-#   (16, {(r3, 2, 8), (r4, 4, 4)})
-
-
-# ---------------------------------------------------------------------------
-# 
-# JOIN
-# 
-# Joins two or more relations based on common field values.
-# 
-# == Syntax
-# 
-# alias = JOIN alias BY field_alias,
-#              alias BY field_alias [, alias BY field_alias …]
-# 	     [PARALLEL n];                   
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# BY::
-#   Keyword.
-# 
-# field_alias::
-#   The name of a field in a relation. The alias and field_alias specified in the
-#   BY clause must correspond.
-# 
-# == Example:
-#   X = JOIN relationA BY fieldA, relationB by fieldB, relationC by fieldC;
-# 
-# PARALLEL n::
-#   Increase the parallelism of a job by specifying the number of reduce tasks,
-#   n. The optimal number of parallel tasks depends on the amount of memory on
-#   each node and the memory required by each of the tasks. To determine n, use
-#   the following as a general guideline:
-#     n = (nr_nodes - 1) * 0.45 * nr_GB
-#   where nr_nodes is the number of nodes used and nr_GB is the amount of physical
-#   memory on each node.
-# 
-#   Note the following:
-#   * Parallel only affects the number of reduce tasks. Map parallelism is
-#     determined by the input file, one map for each HDFS block.
-#   * If you don’t specify parallel, you still get the same map parallelism but
-#     only one reduce task.
-# 
-# == Usage
-# 
-# Use the JOIN operator to join two or more relations based on common field
-# values. The JOIN operator always performs an inner join.
-# 
-# Note: The JOIN and COGROUP operators perform similar functions. JOIN creates a
-# flat set of output records while COGROUP creates a nested set of output records.
-# 
-# == Example
-# 
-# Suppose we have relations A and B.
-# 
-# (A: a1, a2, a3)   (B: b1, b2)
-# -----------------       ---------------        
-# (1, 2, 3)             (2, 4)    
-# (4, 2, 1)             (8, 9)                
-# (8, 3, 4)             (1, 3)                
-# (4, 3, 3)             (2, 7)
-# (7, 2, 5)             (2, 9)
-# (8, 4, 3)             (4, 6)
-#                         (4, 9)
-# 
-# In this example relations A and B are joined on their first fields.
-# 
-# X = JOIN A BY a1, B BY b1;
-# 
-# Relation X looks like this.
-# 
-# (1, 2, 3, 1, 3)
-# (4, 2, 1, 4, 6)
-# (4, 3, 3, 4, 6)
-# (4, 2, 1, 4, 9)
-# (4, 3, 3, 4, 9)
-# (8, 3, 4, 8, 9)
-# (8, 4, 3, 8, 9)
-#
-
-
-# ---------------------------------------------------------------------------
-# 
-# LIMIT
-# 
-# Limits the number of output tuples.
-# 
-# == Syntax
-# 
-# alias = LIMIT alias  n;
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# n::
-#   The number of tuples.
-# 
-# == Usage
-# 
-# Use the LIMIT operator to limit the number of output tuples (rows). If the
-# specified number of output tuples is equal to or exceeds the number of tuples in
-# the relation, the output will include all tuples in the relation.
-# 
-# There is no guarantee which tuples will be returned, and the tuples that are
-# returned can change from one run to the next. A particular set of tuples can be
-# requested using the ORDER operator followed by LIMIT.
-# 
-# Note: The LIMIT operator allows Pig to avoid processing all tuples in a
-# relation. In most cases a query that uses LIMIT will run more efficiently than
-# an identical query that does not use LIMIT. It is always a good idea to use
-# limit if you can.
-# 
-# == Examples
-# 
-# Suppose we have relation A.
-#   
-#   (A: f1:int, f2:int, f3:int)    
-#   -----------------                  
-#   (1, 2, 3)
-#   (4, 2, 1)
-#   (8, 3, 4)
-#   (4, 3, 3)
-#   (7, 2, 5)
-#   (8, 4, 3)
-# 
-# In this example output is limited to 3 tuples.
-# 
-#   X = LIMIT A 3;
-# 
-# Relation X could look like this (there is no guarantee which three tuples will be output).
-#   
-#   (1, 2, 3)
-#   (4, 3, 3)
-#   (7, 2, 5)
-# 
-# In this example the ORDER operator is used to order the tuples and the LIMIT operator is used to output the first three tuples.
-# 
-#   B = ORDER A BY f1 DESC, f2 ASC;
-#   X = LIMIT B 3;
-# 
-# Relation B and relation X look like this.
-#   
-#   (B)                                     (X)  
-#   -----------             -----------
-#   (8, 3, 4)                         (8, 3, 4)
-#   (8, 4, 3)                         (8, 4, 3)
-#   (7, 2, 5)                         (7, 2, 5)
-#   (4, 2, 1)
-#   (4, 3, 3)
-#   (1, 2, 3)
-
-
-# ---------------------------------------------------------------------------
-# 
-# LOAD
-# 
-# Loads data from the file system.
-# 
-# == Syntax
-# 
-# LOAD 'data' [USING function] [AS schema];              
-# 
-# == Terms
-# 
-# 'data'::
-#   The name of the file or directory, in single quotes.
-# 
-#   If you specify a directory name, all the files in the directory are loaded.
-# 
-#   You can use hadoop-supported globing to specify files at the file system or
-#   directory levels (see [WWW]hadoop glob documentation for details on globing
-#   syntax).
-# 
-# USING::
-#   Keyword.
-# 
-# function::
-#   The load function.
-# 
-#   PigStorage is the default load/store function and does not need to be
-#   specified. This function reads/writes simple newline-separated records with
-#   delimiter-separated fields. The function has one parameter, the field
-#   delimiter (tab (‘\t’) if the default delimiter).
-# 
-#   If the data is stored in a special format that the Pig load functions cannot
-#   parse, you can write your own load function.
-# 
-# AS::
-#   Keyword.
-# 
-# schema::
-#   A schema using the AS keyword, enclosed in parentheses (see Schemas).
-# 
-# == Usage
-# 
-# Use the LOAD operator to load data from the file system.
-# 
-# == Examples
-# 
-# Suppose we have a data file called myfile.txt. The fields are tab-delimited. The
-# records are newline-separated.
-#   
-#   1          2          3
-#   4          2          1
-#   8          3          4
-# 
-# In this example the default load function, PigStorage, loads data from
-# myfile.txt into relation A. Note that, because no schema is specified, the
-# fields are not named and all fields default to type bytearray. The two
-# statements are equivalent.
-#   
-#   A = LOAD 'myfile.txt';
-#   A = LOAD 'myfile.txt' USING PigStorage('\t');
-# 
-# Relation A looks like this.
-#   
-#   (1, 2, 3)
-#   (4, 2, 1)
-#   (8, 3, 4)
-# 
-# In this example a schema is specified using the AS keyword. The two statements
-# are equivalent.
-#   
-#   A = LOAD 'myfile.txt' AS (f1:int, f2:int, f3:int);
-#   A = LOAD 'myfile.txt' USING PigStorage(‘\t’) AS (f1:int, f2:int, f3:int);
-
-
-# ---------------------------------------------------------------------------
-# 
-# ORDER
-# 
-# Sorts a relation based on one or more fields.
-# 
-# == Syntax
-# 
-# alias = ORDER alias BY { * [ASC|DESC] | field_alias [ASC|DESC]
-#           [, field_alias [ASC|DESC] …] } [PARALLEL n];
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# BY::
-#   Required keyword.
-# 
-# *::
-#   Represents all fields in the relation.
-# 
-# ASC::
-#   Sort in ascending order.
-# 
-# DESC::
-#   Sort in descending order.
-# 
-# field_alias::
-#   A field in the relation.
-# 
-# PARALLEL n::
-#   Increase the parallelism of a job by specifying the number of reduce tasks,
-#   n. The optimal number of parallel tasks depends on the amount of memory on
-#   each node and the memory required by each of the tasks. To determine n, use
-#   the following as a general guideline:
-#     n = (nr_nodes - 1) * 0.45 * nr_GB
-#   where nr_nodes is the number of nodes used and nr_GB is the amount of physical
-#   memory on each node.
-# 
-#   Note the following:
-#   * Parallel only affects the number of reduce tasks. Map parallelism is
-#     determined by the input file, one map for each HDFS block.
-#   * If you don’t specify parallel, you still get the same map parallelism but
-#     only one reduce task.
-# 
-# == Usage
-# 
-# In Pig, relations are logically unordered.
-# 
-# * If you order relation A to produce relation X (X = ORDER A BY * DESC;),
-#   relations A and X still contain the same thing.
-# 
-# * If you retrieve the contents of relation X, they are guaranteed to be in the
-#   order you specified (descending).
-# 
-# * However, if you further process relation X, there is no guarantee that the
-#   contents will be processed in the order you specified.
-# 
-# == Examples
-# 
-# Suppose we have relation A.
-# 
-#   (A: f1, f2, f3)     
-#   -----------------                  
-#   (1, 2, 3)                        
-#   (4, 2, 1)                                    
-#   (8, 3, 4)                                    
-#   (4, 3, 3)
-#   (7, 2, 5)            
-#   (8, 4, 3)
-# 
-# In this example relation A is sorted by the third field, f3 in descending order.
-# 
-#   X = ORDER A BY f3 DESC;
-# 
-# Relation X could look like this (note that the order of the three tuples ending
-# in 3 can vary).
-# 
-#   (7, 2, 5)
-#   (8, 3, 4)
-#   (1, 2, 3)
-#   (4, 3, 3)
-#   (8, 4, 3)
-#   (4, 2, 1)
-
-
-# ---------------------------------------------------------------------------
-# 
-# SPLIT
-# 
-# Partitions a relation into two or more relations.
-# 
-# == Syntax
-# 
-# SPLIT alias INTO alias IF expression, alias IF expression [, alias IF expression …];
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# INTO::
-#   Required keyword.
-# 
-# IF::
-#   Required keyword.
-# 
-# expression::
-#   An expression.
-# 
-# == Usage
-# 
-# Use the SPLIT operator to partition a relation into two or more relations based
-# on some expression. Depending on the expression:
-# 
-# * A tuple may be assigned to more than one relation.
-# 
-# * A tuple may not be assigned to any relation.
-# 
-# == Example
-# 
-# Suppose we have relation A.
-# 
-# (A: f1, f2, f3)     
-# -----------------                              
-# (1, 2, 3)
-# (4, 5, 6)
-# (7, 8, 9)
-# 
-# In this example relation A is split into three relations, X, Y, and Z.
-# 
-# SPLIT A INTO X IF f1< 7, Y IF f2==5, Z IF (f3<6 OR f3>6);
-# 
-# Relations X, Y, and Z look like this.
-# 
-# (X)                    (Y)                    (Z)
-# ----------              ----------- -----------
-# (1, 2, 3)             (4, 5, 6)             (1, 2, 3)
-# (4, 5, 6)                                     (7, 8, 9)
-
-
-# ---------------------------------------------------------------------------
-# 
-# STORE
-# 
-# Stores data to the file system.
-# 
-# == Syntax
-# 
-# STORE alias INTO 'directory' [USING function];
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# INTO::
-#   Required keyword.
-# 
-# 'directory'::
-#   The name of the storage directory, in quotes. If the directory already exists, the STORE operation will fail.
-# 
-#  
-# 
-# The output data files, named part-nnnnn, are written to this directory.
-# 
-# USING::
-#   Keyword. Use this clause to name the store function.
-# 
-# function::
-#   The load function.
-# 
-# PigStorage is the default load/store function and does not need to be specified. This function reads/writes simple newline-separated records with delimiter-separated fields. The function has one parameter, the field delimiter (tab ‘\t’ if the default delimiter)
-# 
-# If you want to store the data in a special format that the Pig Load/Store functions cannot handle, you can write your own store function.
-# 
-# == Usage
-# 
-# Use the STORE operator to store data on the file system.
-# 
-# == Example
-# 
-# Suppose we have relation A.
-# 
-# (A)
-# 
-# ----------------
-# (1, 2, 3)
-# (4, 2, 1)
-# (8, 3, 4)
-# (4, 3, 3)
-# (7, 2, 5)
-# (8, 4, 3)
-# 
-# In this example the contents of relation A are written to file part-00000 located in directory myoutput.
-# 
-# STORE relationA INTO ‘myoutput’ USING PigStorage (‘*’);
-# 
-# The part-00000 file looks like this. Fields are delimited with the asterisk * characters and records are separated by newlines.
-# 
-# 1*2*3
-# 4*2*1
-# 8*3*4
-# 4*3*3
-# 7*2*5
-# 8*4*3
-#
-
-
-# ---------------------------------------------------------------------------
-# 
-# STREAM
-# 
-# Sends data to an external script or program.
-# 
-# == Syntax
-# 
-# alias = STREAM alias [, alias …] THROUGH {`command` | cmd_alias } [AS schema] ;
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# THROUGH::
-#   Keyword.
-# 
-# `command`::
-#   A command, including the arguments, enclosed in back tics (where a command is anything that can be executed).
-# 
-# cmd_alias::
-#   The name of a command created using the DEFINE operator.
-# 
-# AS::
-#   Keyword.
-# 
-# schema::
-#   A schema using the AS keyword, enclosed in parentheses (see Schemas).
-# 
-# == Usage
-# 
-# Use the STREAM operator to send data through an external script or program. Multiple stream operators can appear in the same Pig script. The stream operators can be adjacent to each other or have other operations in between.
-# 
-# When used with a command, a stream statement could look like this:
-# 
-# A = LOAD 'data';
-# 
-# B = STREAM A THROUGH `stream.pl -n 5`;
-# 
-# When used with a cmd_alias, a stream statement could look like this, where cmd is the defined alias.
-# 
-# A = LOAD 'data';
-# 
-# DEFINE cmd `stream.pl –n 5`;
-# 
-# B = STREAM A THROUGH cmd;
-# About Data Guarantees
-# 
-# Data guarantees are determined based on the position of the streaming operator in the Pig script.
-# 
-# * Unordered data – No guarantee for the order in which the data is delivered to
-#   the streaming application.
-# 
-# * Grouped data – The data for the same grouped key is guaranteed to be provided
-#   to the streaming application contiguously
-# 
-# * Grouped and ordered data – The data for the same grouped key is guaranteed to
-#   be provided to the streaming application contiguously. Additionally, the data
-#   within the group is guaranteed to be sorted by the provided secondary key.
-# 
-# In addition to position, data grouping and ordering can be determined by the
-# data itself. However, you need to know the property of the data to be able to
-# take advantage of its structure.
-# 
-# == Example: Data Guarantees
-# 
-# In this example the data is unordered.
-# 
-# A = LOAD 'data';
-# B = STREAM A THROUGH `stream.pl`;
-# 
-# In this example the data is grouped.
-# 
-# A = LOAD 'data';
-# B = GROUP A BY $1;
-# C = FOREACH B FLATTEN(A);
-# D = STREAM C THROUGH `stream.pl`
-# 
-# In this example the data is grouped and ordered.
-# 
-# A = LOAD 'data';
-# B = GROUP A BY $1;
-# C = FOREACH B {
-#       D = ORDER A BY ($3, $4);
-#       GENERATE D;
-# }
-# E = STREAM C THROUGH `stream.pl`;
-# 
-# == Example: Schemas
-# 
-# In this example a schema is specified as part of the STREAM statement.
-# 
-#   X = STREAM A THROUGH `stream.pl` as (f1:int, f2;int, f3:int);
-# 
-# Additional Examples
-# 
-#   See DEFINE for additional examples.
-
-
-# ---------------------------------------------------------------------------
-# 
-# UNION
-# 
-# Computes the union of two or more relations.
-# 
-# == Syntax
-# 
-# alias = UNION alias, alias [, alias …];
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# == Usage
-# 
-# Use the UNION operator to compute the union of two or more relations. The UNION operator:
-# 
-# * Does not preserve the order of tuples. Both the input and output relations are
-#   interpreted as unordered bags of tuples.
-# 
-# * Does not ensure (as databases do) that all tuples adhere to the same schema or
-#   that they have the same number of fields. In a typical scenario, however, this
-#   should be the case; therefore, it is the user's responsibility to either (1)
-#   ensure that the tuples in the input relations have the same schema or (2) be
-#   able to process varying tuples in the output relation.
-# 
-# * Does not eliminate duplicate tuples.
-# 
-# == Example
-# 
-# Suppose we have relations A and B.
-# 
-# (A)                    (B)
-# ----------- --------   
-# (1, 2, 3)             (2, 4)    
-# (4, 2, 1)             (8, 9)                
-#                         (1, 3)
-# 
-# In this example the union of relation A and B is computed.
-# 
-# X = UNION A, B;
-# 
-# Relation X looks like this.
-# 
-# (1, 2, 3)
-# (4, 2, 1)
-# (2, 4)
-# (8, 9)
-# (1, 3)
-# Diagnostic Operators
-# DESCRIBE
-# 
-# Returns the schema of an alias.
-# 
-# == Syntax
-# 
-# DESCRIBE alias;                                                                      
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# == Usage
-# 
-# Use the DESCRIBE operator to review the schema of a particular alias.
-# 
-# == Example
-# 
-# In this example a schema is specified using the AS clause.
-#   
-#   A = LOAD 'students' AS (name:chararray, age:int, gpa:float);
-#   B = FILTER A BY name matches 'John%';
-#   C = GROUP B BY name;
-#   D = FOREACH B GENERATE COUNT(B.age);
-#   DESCRIBE A;
-#   A: {group, B: (name: chararray,age: int,gpa: float}
-#   DESCRIBE B;
-#   B: {group, B: (name: chararray,age: int,gpa: float}
-#   DESCRIBE C;
-#   C: {group, chararry,B: (name: chararray,age: int,gpa: float}
-#   DESCRIBE D;
-#   D: {long}
-# 
-# In this example no schema is specified. All data items default to type bytearray.
-# 
-#   grunt> a = LOAD '/data/students';
-#   grunt> b = FILTER a BY $0 matches 'John%';
-#   grunt> c = GROUP b BY $0;
-#   grunt> d = FOREACH c GENERATE COUNT(b.$1);
-#   grunt> DESCRIBE a;
-# 
-# Schema for a unknown.
-# 
-#   grunt> DESCRIBE b;
-#   2008-12-05 01:17:15,316 [main] WARN  org.apache.pig.PigServer - bytearray is implicitly cast to chararray under LORegexp Operator
-# 
-# Schema for b unknown.
-# 
-#   grunt> DESCRIBE c;
-#   2008-12-05 01:17:23,343 [main] WARN  org.apache.pig.PigServer - bytearray is implicitly caste to chararray under LORegexp Operator
-# 
-# c: {group: bytearray,b: {null}}
-# 
-#   grunt> DESCRIBE d;
-#   2008-12-05 03:04:30,076 [main] WARN  org.apache.pig.PigServer - bytearray is implicitly caste to chararray under LORegexp Operator
-# 
-# d: {long}
-# 
-# DUMP
-# 
-# Displays the contents of an alias.
-# 
-# == Syntax
-# 
-# DUMP alias;                                                           
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# == Usage
-# 
-# Use the DUMP operator to display the contents of an alias. You can use DUMP as a
-# debugging device to make sure the correct results are being generated.
-# 
-# == Example
-# 
-# In this example a dump is performed after each statement.
-# 
-#   A = LOAD 'students' AS (name:chararray, age:int, gpa:float);
-#   DUMP A;
-#   B = FILTER A BY name matches 'John%';
-#   DUMP B;
-#   B = GROUP B BY name;
-#   DUMP C;
-#   D = FOREACH C GENERATE COUNT(B.age);
-#   DUMP D;
-# 
-# EXPLAIN
-# 
-# Displays execution plans.
-# 
-# == Syntax
-# 
-# EXPLAIN alias;                            
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# == Usage
-# 
-# Use the EXPLAIN operator to review the logical, physical, and map reduce
-# execution plans that are used to compute the specified relationship.
-# 
-# * The logical plan shows a pipeline of operators to be executed to build the
-#   relation. Type checking and backend-independent optimizations (such as
-#   applying filters early on) also apply.
-# 
-# * The physical plan shows how the logical operators are translated to
-#   backend-specific physical operators. Some backend optimizations also apply.
-# 
-# * The map reduce plan shows how the physical operators are grouped into map
-#   reduce jobs.
-# 
-# == Example
-# 
-# In this example the EXPLAIN operator produces all three plans. (Note that only a
-# portion of the output is shown in this example.)
-# 
-# A = LOAD 'students' AS (name:chararray, age:int, gpa:float);
-# B = GROUP A BY name;
-# C = FOREACH B GENERATE COUNT(A.age);
-# EXPLAIN C;
-# 
-#  
-# Logical Plan:
-# 
-# Store xxx-Fri Dec 05 19:42:29 UTC 2008-23 Schema: {long} Type: Unknown
-# |
-# |---ForEach xxx-Fri Dec 05 19:42:29 UTC 2008-15 Schema: {long} Type: bag
-# etc …
-# 
-# -----------------------------------------------
-# Physical Plan:
-# -----------------------------------------------
-# Store(fakefile:org.apache.pig.builtin.PigStorage) - xxx-Fri Dec 05 19:42:29 UTC 2008-40
-# |
-# |---New For Each(false)[bag] - xxx-Fri Dec 05 19:42:29 UTC 2008-39
-#     |   |
-#     |   POUserFunc(org.apache.pig.builtin.COUNT)[long] - xxx-Fri Dec 05
-# etc …
-#  
-# --------------------------------------------------
-# | Map Reduce Plan                                |
-# --------------------------------------------------
-# MapReduce node xxx-Fri Dec 05 19:42:29 UTC 2008-41
-# Map Plan
-# Local Rearrange[tuple]{chararray}(false) - xxx-Fri Dec 05 19:42:29 UTC 2008-34
-# |   |
-# |   Project[chararray][0] - xxx-Fri Dec 05 19:42:29 UTC 2008-35
-# etc …  
-# ILLUSTRATE
-# 
-# Displays a step-by-step execution of a sequence of statements.
-# 
-# == Syntax
-# 
-# ILLUSTRATE alias;                                                 
-# 
-# == Terms
-# 
-# alias::
-#   The name of a relation.
-# 
-# == Usage
-# 
-# Use the ILLUSTRATE operator to review how data items are transformed through a
-# sequence of Pig Latin statements.
-# 
-# ILLUSTRATE accesses the ExampleGenerator algorithm which can select an
-# appropriate and concise set of example data items automatically. It does a
-# better job than random sampling would do; for example, random sampling suffers
-# from the drawback that selective operations such as filters or joins can
-# eliminate all the sampled data items, giving you empty results which is of no
-# help with debugging.
-# 
-# With the ILLUSTRATE operator you can test your programs on small datasets and
-# get faster turnaround times. The ExampleGenerator algorithm uses Pig's Local
-# mode (rather than Hadoop mode) which means that illustrative example data is
-# generated in near real-time.
-# 
-# == Example
-# 
-# Suppose we have a data file called 'visits.txt'.
-# Amy     cnn.com            20080218
-# Fred    harvard.edu         20081204
-# Amy     bbc.com            20081205
-# Fred    stanford.edu        20081206
-# 
-# In this example we count the number of sites a user has visited since
-# 12/1/08. The ILLUSTRATE statement will show how the results for num_user_visits
-# are derived.
-# 
-# visits = LOAD 'visits.txt' AS (user:chararray, url:chararray, timestamp:chararray);
-# 
-# recent_visits = FILTER visits BY timestamp >= '20081201';
-# 
-# user_visits = GROUP recent_visits BY user;
-# 
-# num_user_visits = FOREACH user_visits GENERATE COUNT(recent_visits);
-# 
-# ILLUSTRATE num_user_visits
-# 
-# The output from the ILLUSTRATE statement looks like this.
-# 
-# ------------------------------------------------------------------------
-# 
-# | visits     | user: bytearray | url: bytearray | timestamp: bytearray |
-# ------------------------------------------------------------------------
-# |            | Amy             | cnn.com        | 20080218             |
-# |            | Fred            | harvard.edu    | 20081204            |
-# |            | Amy             | bbc.com        | 20081205             |
-# |            | Fred            | stanford.edu   | 20081206            |
-# ------------------------------------------------------------------------
-# 
-# -------------------------------------------------------------------------------
-# | recent_visits     | user: chararray | url: chararray | timestamp: chararray |
-# -------------------------------------------------------------------------------
-# |                   | Fred            | harvard.edu    | 20081204             |
-# |                   | Amy             | bbc.com        | 20081205             |
-# |                   | Fred            | stanford.edu   | 20081206             |
-# -------------------------------------------------------------------------------
-# 
-# ------------------------------------------------------------------------------------------------------------------
-# | user_visits     | group: chararray | recent_visits: bag({user: chararray,url: chararray,timestamp: chararray}) |
-# ------------------------------------------------------------------------------------------------------------------
-# |                 | Amy              | {(Amy, bbc.com, 20081205)}                                                |
-# |                 | Fred             | {(Fred, harvard.edu, 20081204), (Fred, stanford.edu, 20081206)}           |
-# ------------------------------------------------------------------------------------------------------------------
-# 
-# -------------------------------
-# | num_user_visits     | long  |
-# -------------------------------
-# |                     | 1     |
-# |                     | 2     |
-# -------------------------------
-#
-
-# ---------------------------------------------------------------------------
-# 
-# DEFINE
-# 
-# Assigns an alias to a function or command.
-# 
-# == Syntax
-# 
-# DEFINE alias {function | [`command` [input] [output] [ship] [cache]] };
-# 
-# == Terms
-# 
-# alias::
-#   The name for the function or command.
-# 
-# function::
-#   The name of a function.
-# 
-# Use this option to define functions for use with the FOREACH and FILTER operators.
-# 
-# `command `::
-#   A command, including the arguments, enclosed in back tics (where a command is anything that can be executed).
-# 
-# Use this option to define commands for use with the STREAM operator.
-# 
-# input::
-#   INPUT ( {stdin | 'path'} [USING serializer] [, {stdin | 'path'} [USING serializer] …] )
-# 
-# Where:
-# * INPUT – Keyword.
-# * 'path' – A file path, enclosed in single quotes.
-# * USING – Keyword.
-# * serializer – A function that converts data from tuples to stream format. PigStorage is the default serializer. You can also write your own UDF.
-# 
-# output::
-#   OUTPUT ( {stdout | stderr | 'path'} [USING deserializer] [, {stdout | stderr | 'path'} [USING deserializer] …] )
-# 
-#   Where:
-#   
-#   * OUTPUT – Keyword.
-#   * 'path' – A file path, enclosed in single quotes.
-#   * USING – Keyword.
-#   * deserializer – A function that converts data from stream format to tuples. PigStorage is the default deserializer. You can also write your own UDF.
-# 
-# ship::
-#   SHIP('path' [, 'path' …])
-# 
-#   Where:
-#   
-#   * SHIP – Keyword.
-#   * 'path' – A file path, enclosed in single quotes.
-# 
-# cache::
-#   CACHE('dfs_path#dfs_file' [, 'dfs_path#dfs_file' …])
-#   
-#   Where:
-#   
-#   * CACHE – Keyword.
-#   * 'dfs_path#dfs_file' – A file path/file name on the distributed file system,
-#     enclosed in single quotes. Example: '/mydir/mydata.txt#mydata.txt'
-# 
-# 
-# == Usage
-# 
-# Use the DEFINE statement to assign a name (alias) to a function or to a command.
-# 
-# Use DEFINE to specify a function when:
-# 
-# * The function has a log package name that you don't want to include in a
-#   script, especially if you call the function several times in that script.
-# 
-# * The constructor for the function takes parameters (see the first example
-#   below). If you need to use different constructor parameters for different
-#   calls to the function you will need to create multiple defines – one for each
-#   parameter set.
-# 
-# Use DEFINE to specify a command when the streaming command specification is
-# complex or requires additional parameters (input, output, and so on).
-# 
-# === About Input and Output
-# 
-# Serialization is needed to convert data from tuples to a format that can be
-# processed by the streaming application. Deserialization is needed to convert the
-# output from the streaming application back into tuples.
-# 
-# PigStorage, the default serialization/deserialization function, converts tuples
-# to tab-delimited lines. Pig's BinarySerializer and BinaryDeserializer functions
-# treat the entire file as a byte stream (no formatting or interpretation takes
-# place). You can also write your own serialization/deserialization functions.
-# 
-# === About Ship
-# 
-# Use the ship option to send streaming binary and supporting files, if any, from
-# the client node to the compute nodes. Pig does not automatically ship
-# dependencies; it is your responsibility to explicitly specify all the
-# dependencies and to make sure that the software the processing relies on (for
-# instance, perl or python) is installed on the cluster. Supporting files are
-# shipped to the task's current working directory and only relative paths should
-# be specified. Any pre-installed binaries should be specified in the path.
-# 
-# Only files, not directories, can be specified with the ship option. One way to
-# work around this limitation is to tar all the dependencies into a tar file that
-# accurately reflects the structure needed on the compute nodes, then have a
-# wrapper for your script that un-tars the dependencies prior to execution.
-# 
-# Note that the ship option has two components: the source specification, provided
-# in the ship clause, is the view of your machine; the command specification is
-# the view of the cluster.The only guarantee is that the shipped files are
-# available is the current working directory of the launched job and that your
-# current working directory is also on the PATH environment variable.
-# 
-# Shipping files to relative paths or absolute paths is not supported since you
-# might not have permission to read/write/execute from arbitrary paths on the
-# clusters.
-# 
-# === About Cache
-# 
-# The ship option works with binaries, jars, and small datasets. However, loading
-# larger datasets at run time for every execution can severely impact
-# performance. Instead, use the cache option to access large files already moved
-# to and available on the compute nodes. Only files, not directories, can be
-# specified with the cache option.
-# 
-# == Example: Input/Output
-# 
-# In this example PigStorage is the default serialization/deserialization
-# function. The tuples from relation A are converted to tab-delimited lines that
-# are passed to the script.
-# 
-#   X = STREAM A THROUGH `stream.pl`;
-# 
-# In this example PigStorage is used as the serialization/deserialization
-# function, but a comma is used as the delimiter.
-# 
-#   DEFINE Y `stream.pl` INPUT(stdin USING PigStorage(',')) OUTPUT (stdout USING PigStorage(','));
-#   X = STREAM A THROUGH Y;
-# 
-# In this example user-defined serialization/deserialization functions are used
-# with the script.
-# 
-#   DEFINE Y `stream.pl` INPUT(stdin USING MySerializer) OUTPUT (stdout USING MyDeserializer);
-#   X = STREAM A THROUGH Y;
-# 
-# == Example: Ship/Cache
-# 
-# In this example ship is used to send the script to the cluster compute nodes.
-# 
-#   DEFINE Y `stream.pl` SHIP('/work/stream.pl');
-#   X = STREAM A THROUGH Y;
-# 
-# In this example cache is used to specify a file located on the cluster compute
-# nodes.
-# 
-#   DEFINE Y `stream.pl data.gz` SHIP('/work/stream.pl') CACHE('/input/data.gz#data.gz');
-#   X = STREAM A THROUGH Y;
-# 
-# == Example: Logging
-# 
-# In this example the streaming stderr is stored in the _logs/<dir> directory of
-# the job's output directory. Because the job can have multiple streaming
-# applications associated with it, you need to ensure that different directory
-# names are used to avoid conflicts. Pig stores up to 100 tasks per streaming job.
-# 
-#   DEFINE Y `stream.pl` stderr('<dir>' limit 100);
-#   X = STREAM A THROUGH Y;
-# 
-# In this example a function is defined for use with the FOREACH …GENERATE operator.
-#   grunt> REGISTER /src/myfunc.jar
-#   grunt> define myFunc myfunc.MyEvalfunc('foo');
-#   grunt> A = LOAD 'students';
-#   grunt> B = FOREACH A GENERATE myFunc($0);
-# 
-# In this example a command is defined for use with the STREAM operator.
-#   grunt> A = LOAD 'data';
-#   grunt> DEFINE cmd `stream_cmd –input file.dat`
-#   grunt> B = STREAM A through cmd.
-# 
-
-
-# ---------------------------------------------------------------------------
-# 
-# = REGISTER
-# 
-# Registers a JAR file so that the UDFs in the file can be used.
-# 
-# == Syntax
-# 
-# REGISTER alias;
-# 
-# == Terms
-# 
-# [alias]		The path of a Java JAR file. Do not place the name in quotes.
-# 
-# == Usage
-# 
-# Use the REGISTER statement to specify the path of a Java JAR file containing UDFs.
-# 
-# For more information about UDFs, see the User Defined Function Guide. Note that
-# Pig currently only supports functions written in Java.
-# 
-# == Example
-# 
-# In this example REGISTER states that myfunc.jar is located in the /src
-# directory.
-# 
-#   grunt> REGISTER /src/myfunc.jar;
-#   grunt> A = LOAD 'students';
-#   grunt> B = FOREACH A GENERATE myfunc.MyEvalFunc($0);
-#