wukong 3.0.0.pre → 3.0.0.pre2

data/docpages/INSTALL.textile

@@ -1,92 +0,0 @@
----
-layout: default
-title:  Install Notes
-collapse: false
----
-h1(gemheader). {{ site.gemname }} %(small):: install%
-
-** "Get the code":#getcode
-** "Setup":#setup
-** "Installing and Running Wukong with Hadoop":#gethadoop
-** "Installing and Running Wukong with Datamapper, ActiveRecord, the command-line and more":#others
-
-
-<notextile><div class="toggle"></notextile>
-
-h2(#getcode). Get the code
-
-We're still actively developing {{ site.gemname }}.  The newest version is available via "Git":http://git-scm.com on "github:":http://github.com/mrflip/{{ site.gemname }}
-
-pre. $ git clone git://github.com/mrflip/{{ site.gemname }}
-
-A gem is available from "gemcutter:":http://gemcutter.org/gems/{{ site.gemname }}
-
-pre. $ sudo gem install {{ site.gemname }} --source=http://gemcutter.org
-
-(don't use the gems.github.com version -- it's way out of date.)
-
-You can instead download this project in either "zip":http://github.com/mrflip/{{ site.gemname }}/zipball/master or "tar":http://github.com/mrflip/{{ site.gemname }}/tarball/master formats.
-
-h3. Get the Dependencies
-
-* Hadoop
-* Pig (optional)
-* Parts of {{ site.gemname }} require these gems:
-** addressable/uri
-** htmlentities 
-** extlib
-** YAML
-** JSON
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#setup). Setup
-
-1. Allow Wukong to discover where his elephant friend lives by setting a $HADOOP_HOME environment variable:  @export HADOOP_HOME="/usr/local/share/hadoop"@
-2. Add wukong's @bin/@ directory to your $PATH if you'd like to use the "wutils":wutils.html
-
-<i>(see also: "Ruby Hadoop Quickstart":http://blog.pdatasolutions.com/post/191978092/ruby-on-hadoop-quickstart)</i>
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#gethadoop). Installing and Running Wukong with Hadoop
-
-Wukong was primarily developed for Hadoop, and we think it's the best way to use Hadoop (it's certainly the most fun!).
-
-h3. Run Wukong on the Amazon AWS EC2 Cloud
-
-h3. Hadoop Infrastructure
-
-Even if you have a bunch of machines with spare cycles, lots of RAM, and a shared filesystem... do yourself a favor and start out using the "Cloudera AMIs on Amazon's EC2 cloud.":http://www.cloudera.com/hadoop-ec2 There are an overwhelming number of fiddly little parameters and you'll be glad for the user experience before you get into server setup. If it's still mid-late 2009 when you read this, ignore prudence and jump straight to using Hadoop 0.20.  It will be a) more fun, b) much more robust (trust me, at "v0.20" you want to live on the bleeding edge), and c) you won't have to suffer through migrating your HDFS two weeks after setup.
-
-To set up hadoop, your best bet are the Cloudera AMIs on Amazon's EC2 compute cloud:
-
-* http://www.cloudera.com/hadoop-ec2
-* http://www.cloudera.com/hadoop-ec2-ebs-beta
-
-EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.
-
-h3. Run Wukong using Amazon AWS Elastic MapReduce
-
-AWS Elastic MapReduce saves the trouble of even setting up a cluster: click, bam, there it is.
-
-Phil Ripperger has prepared a "Ruby Hadoop Quickstart":http://blog.pdatasolutions.com/post/191978092/ruby-on-hadoop-quickstart explaining how to get started with Wukong, Hadoop and the Amazon Elastic MapReduce cloud -- it's better than anything we could put here. Thanks Phil!
-
-h3. Set up a Hadoop cluster
-
-If you have a local cluster, or just want to experiment with a single-machine install, check out the Cloudera packages for both Debian/Ubuntu-based and Redhat/RPM-based Linux systems.
-
-h3. More Hadoop Notes
-
-I've braindumped some random notes on configuring and using hadoop "over here":hadoop-tips.html
-
-<notextile></div><div class="toggle"></notextile>
-
-h2(#others). Wukong isn't just Hadoop: Datamapper, ActiveRecord, command-line usage and more
-
-Wukong is used by many in an non-Hadoop environment -- anywhere you can stream data records, you can unleash its monkey power.
-
-Please see the "usage notes":usage.html#playnice for more!
-
-
-<notextile></div></notextile>

data/docpages/LICENSE.textile

@@ -1,107 +0,0 @@
----
-layout: default
-title:  Apache License
----
-
-
-h1(gemheader). {{ site.gemname }} %(small):: license%
-
-
-The wukong code is __Copyright (c) 2009 Philip (flip) Kromer__
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
-
-h1. Apache License
-
-                              Apache License
-                        Version 2.0, January 2004
-                     http://www.apache.org/licenses/
-
-TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-<notextile><div class="toggle"></notextile>
-
-h2. 1. Definitions.
-
-* **License** shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
-* **Licensor** shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
-* **Legal Entity** shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, **control** means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
-* **You** (or **Your**) shall mean an individual or Legal Entity exercising permissions granted by this License.
-* **Source** form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
-* **Object** form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
-* **Work** shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
-* **Derivative Works** shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
-* **Contribution** shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, **submitted** means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
-* **Contributor** shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 2. Grant of Copyright License.
-
-Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
-
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 3. Grant of Patent License.
-
-Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 4. Redistribution.
-
-You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
-
-# You must give any other recipients of the Work or Derivative Works a copy of this License; and
-# You must cause any modified files to carry prominent notices stating that You changed the files; and
-# You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
-# If the Work includes a __NOTICE__ text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
-
-You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 5. Submission of Contributions.
-
-Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 6. Trademarks.
-
-This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 7. Disclaimer of Warranty.
-
-Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 8. Limitation of Liability.
-
-In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
-
-<notextile></div><div class="toggle"></notextile>
-
-h2. 9. Accepting Warranty or Additional Liability.
-
-While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
-
-END OF TERMS AND CONDITIONS
-
-<notextile></div><div class="toggle"></notextile>
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
-
-<notextile></div></notextile>

data/docpages/README-elastic_map_reduce.textile

@@ -1,377 +0,0 @@
-h2. Questions
-
-* can I access an EC2 resource (eg cassandra cluster)
-
-
-h2. Setup
-
-* download from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2264&categoryID=273
-* wget http://elasticmapreduce.s3.amazonaws.com/elastic-mapreduce-ruby.zip
-* unzip elastic-mapreduce-ruby.zip
-* cd elastic-mapreduce-ruby
-* ln -nfs ~/.wukong/credentials.json
-* put your keypair in ~/.wukong/keypairs/WHATEVER.pem
-
-  {
-    "access-id":     "<insert your aws access id here>",
-    "private-key":   "<insert your aws secret access key here>",
-    "key-pair":      "WHATEVER",
-    "key-pair-file": "~/.wukong/keypairs/WHATEVER.pem",
-    "log-uri":       "s3n://yourmom/emr/logs"
-  }
-
-h4. Paths
-
-
-Paths:
-  LogUri		s3	s3n://yourmom/emr/logs
-  step log files 	s3	{log_uri}/Steps/{step}/{syslog,stdout,controller,stderr}
-  Script                s3      s3://yourmom/emr/scripts/path/to/script
-  Wukong                s3      s3://s3scripts.infochimps.org/wukong/current/....
-  Input                 s3      s3n://yourmom/data/wordcount/input 
-  Output                s3      s3n://yourmom/data/wordcount/output
-  Bootstrap Scripts     s3      s3://elasticmapreduce/bootstrap-actions/{configure-hadoop,configure-daemons,run-if}
-  
-  
-  Credentials   	desk	elastic-mapreduce-ruby/credentials.json
-
-  hadoop.tmp.dir	inst	/mnt/var/lib/hadoop/tmp
-  local hdfs		inst	/mnt/var/lib/hadoop/dfs
-  your home dir		inst	/home/hadoop			(small space)
-  Job Settings		inst	/mnt/var/lib/info/job-flow.json
-  Instance Settings	inst	/mnt/var/lib/info/instance.json
-
-
-h4. Launching emr tasks in wukong
-
-* Uses configliere to get your credentials, log_uri, emr_root, script_path
-* Uploads script phases.
-    s3://emr_root/scripts/:script_path/script_name-datetime-mapper.rb
-    s3://emr_root/scripts/:script_path/script_name-datetime-reducer.rb
-** You can use the following symbols to assemble the path:
-    :emr_root, :script_name, :script_path, :username, :date, :datetime, :phase, :rand, :pid, :hostname, :keypair
-  The values for :emr_root and :script_path are taken from configliere.
-  if :script_path is missing, scripts/:username is used.
-  The same timestamp and random number will be used for each phase
-
-* uses elastic-mapreduce-ruby to launch the job
-
-** specify --emr.{option}
-** eg --emr.alive, --emr.num-instances
-
-reads ~/.wukong/emr.yaml
-
-  common
-  jobs / jobname 
-
-  name                  same as for hadoop name
-  alive
-  
-  num_instances         .
-  instance_type         .
-  master_instance_type  .
-  availability_zone     us-east-1b
-  key_pair              job_handle
-  key_pair_file         ~/.wukong/keypairs/{key_pair}.pem
-
-  hadoop_version        0.20
-  plain_output          Return the job flow id from create step as simple text
-  info                  JSON hash
-  emr_root              
-  log_uri               emr_root/logs/:script_path/:script_name-:datetime
-
-  --hadoop-version=0.20 --stream --enable_debugging --verbose --debug --alive
-  --availability-zone AZ --key_pair KP --key_pair_file KPF --access_id EC2ID --private_key EC2PK
-  --slave_instance_type m2.xlarge --master_instance_type m2.xlarge --num_instances NUM
-  #
-  --step_name           
-  --step_action         CANCEL_AND_WAIT, TERMINATE_JOB_FLOW or CONTINUE
-  --jobflow             JOBFLOWID
-  #
-  --info                Settings.emr.info.to_json
-  #
-  --input               INPUT
-  --output              OUTPUT
-  --mapper               s3://emr_root/jobs/:script_path/script_name-datetime-mapper.rb  (or class)
-  --reducer              s3://emr_root/jobs/:script_path/script_name-datetime-reducer.rb (or class)
-  --cache               s3n://emr_root/jobs/:script_path/cache/sample.py#sample.py
-  --cache-archive        s3://s3scripts.infochimps.org/wukong/current/wukong.zip
-  --cache-archive       s3n://emr_root/jobs/:script_path/cache/sample.jar
-  --jobconf             whatever
-  
-...
-
-also:
-
-  --ssh
-  --scp SRC --to DEST
-  --terminate
-  --logs
-  --list
-  --all   
-
-h4. Aggregate
-
-http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapred/lib/aggregate/package-summary.html
-
-  DoubleValueSum 	sums up a sequence of double values.
-  LongValueMax  	maintain the maximum of a sequence of long values.
-  LongValueMin  	maintain the minimum of a sequence of long values.
-  LongValueSum  	sums up a sequence of long values.
-  StringValueMax 	maintain the biggest of a sequence of strings.
-  StringValueMin 	maintain the smallest of a sequence of strings.
-  UniqValueCount 	dedupes a sequence of objects.
-  ValueHistogram 	computes the histogram of a sequence of strings.
-  
-h2. Commands
-
-    # create a job and run a mapper written in python and stored in Amazon S3
-    elastic-mapreduce --create --enable_debugging \
-      --stream 
-      --mapper s3://elasticmapreduce/samples/wordcount/wordSplitter.py \
-      --input  s3n://elasticmapreduce/samples/wordcount/input \
-      --output s3n://mybucket/output_path
-      --log_uri 
-
-    elastic-mapreduce --list           # list recently created job flows
-    elastic-mapreduce --list --active  # list all running or starting job flows
-    elastic-mapreduce --list --all     # list all job flows      
-
-h4. Bootstrap actions
-
-    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop
-    --args "--site-config-file,s3://bucket/config.xml,-s,mapred.tasktracker.map.tasks.maximum=2" 
-
-    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-daemons
-    --args "--namenode-heap-size=2048,--namenode-opts=\"-XX:GCTimeRatio=19\""
-
-You should recompile cascading applications with the Hadoop 0.20 version specified so they can take advantage of the new features available in this version.
-Hadoop 0.20 fully supports Pig scripts.
-All Amazon Elastic MapReduce sample apps are compatible with Hadoop 0.20. The AWS Management Console supports only Hadoop 0.20, so samples will default to 0.20 once launched. 
-
-For Hadoop version 0.20, Hive version 0.5 and version Pig 0.6 is used. The version can be selected by setting HadoopVersion in JobFlowInstancesConfig. 
-
-h3. Pig
-
-  REGISTER s3:///my-bucket/piggybank.jar
-
-  Additional functions:
-  
-  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2730
-  
-  
-h2. Hadoop and Cluster setup
-  
-h3. Data Compression
-
-  Output Compression:           -jobconf mapred.output.compress=true        FileOutputFormat.setCompressOutput(conf, true);
-  Intermediate Compression:     -jobconf mapred.compress.map.output=true    conf.setCompressMapOutput(true);    
-   
-You can also use a bootstrap action to automatically compress all job outputs. Here is how to do that with the Ruby client.
-  
-   --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args "-s,mapred.output.compress=true"
-
-Compressed Input data Hadoop automatically detects the .gz extension on file names and extracts the contents. You do not need to take any action to extract gzipped files.
-
-
-===========================================================================
-
-
-$LOAD_PATH << File.dirname(__FILE__)
-  require 'amazon/coral/elasticmapreduceclient'
-  require 'amazon/retry_delegator'
-  
-  config = {
-    :endpoint            => "https://elasticmapreduce.amazonaws.com",
-    :ca_file             => File.join(File.dirname(__FILE__), "cacert.pem"),
-    :aws_access_key      => my_access_id,
-    :aws_secret_key      => my_secret_key,
-    :signature_algorithm => :V2
-  }
-  client = Amazon::Coral::ElasticMapReduceClient.new_aws_query(config)
-  
-  is_retryable_error_response = Proc.new do |response| 
-  if response == nil then
-    false
-    else
-      ret = false
-    if response['Error'] then 
-      ret ||= ['InternalFailure', 'Throttling', 'ServiceUnavailable', 'Timeout'].include?(response['Error']['Code'])
-    end
-    ret 
-   end
-  end
-  
-  client = Amazon::RetryDelegator.new(client, :retry_if => is_retryable_error_response)
-  
-  puts client.DescribeJobFlows.inspect
-  puts client.DescribeJobFlows('JobFlowId' => 'j-ABAYAS1019012').inspect
-
-h3. Example job-flow.json and instance.json
-
-job-flow.json {"jobFlowId":"j-1UVPY9PQ3XAXE","jobFlowCreationInstant":1271711181000,
-  "instanceCount":4,"masterInstanceId":"i-f987ee92","masterPrivateDnsName":
-  "localhost","masterInstanceType":"m1.small","slaveInstanceType":
-  "m1.small","hadoopVersion":"0.18"}
-
-instance.json {"isMaster":true,"isRunningNameNode":true,"isRunningDataNode":true,
-  "isRunningJobTracker":false,"isRunningTaskTracker":false}        
-
-h3. Configuraion
-
-h4. Configure Hadoop
-
-  Location: s3://elasticmapreduce/bootstrap-actions/configure-hadoop
-  
-  -<f>, --<file>-key-value
-    Key/value pair that will be merged into the specified config file.
-  
-  -<F>, --<file>-config-file
-    Config file in Amazon S3 or locally that will be merged with the specified config file.
-  
-  Acceptable config files:
-  s/S  site     hadoop-site.xml
-  d/D  default  hadoop-default.xml
-  c/C  core     core-site.xml
-  h/H  hdfs     hdfs-site.xml
-  m/M  mapred   mapred-site.xml
-  
-  
-  Example Usage:
-  
-  elastic-mapreduce --create \
-    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop
-    --args "--site-config-file,s3://bucket/config.xml,-s,mapred.tasktracker.map.tasks.maximum=2" 
-    
-
-  Specify no reducers:
-    --mapred-key-value mapred.reduce.tasks=0
-
-
-  -cacheFile	-files   	Comma separated URIs
-  -cacheArchive	-archives	Comma separated URIs
-  -jobconf	-D      	key=value
-
-h4. Run If  
-
-Location: s3://elasticmapreduce/bootstrap-actions/run-if <JSON path>[!]=<value> <command> [args...]
-
-  JSON path       A path in the instance config or job flow config for the key we should look up.
-  Value           The value we expect to find.
-  Command         The command to run if the value is what we expect (or not what we expect in the case of !=). This can be a path in S3 or a local command.
-  Args            Arguments to pass to the command as it runs.
-
-  elastic-mapreduce --create --alive \
-    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/run-if 
-    --args "instance.isMaster=true,echo,Running,on,master,node"
-
-  
-h4. Configure Daemons
-
-  --<daemon>-heap-size  	Set the heap size in megabytes for the specified daemon.
-  --<daemon>-opts       	Set additional Java options for the specified daemon.
-  --replace               	Replace the existing hadoop-user-env.sh file if it exists.
-  
-  <daemon> is one of: namenode, datanode, jobtracker, tasktracker, client
-
-  elastic-mapreduce --create --alive
-    --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-daemons
-    --args "--namenode-heap-size=2048,--namenode-opts=\"-XX:GCTimeRatio=19\""
-
-
-h2. Command Line
-
-
-  Creating Job Flows
-        --create                     Create a new job flow
-        --name NAME                  Name of the job flow
-        --alive                      Create a job flow that stays running even though it has executed all its steps
-        --num-instances NUM          Number of instances in the job flow
-        --instance-type TYPE         The type of the instances to launch
-        --slave-instance-type TYPE   The type of the slave instances to launch
-        --master-instance-type TYPE  The type of the master instance to launch
-        --key-pair KEY_PAIR          The name of your Amazon EC2 Keypair
-        --key-pair-file FILE_PATH    Path to your local pem file for your EC2 key pair
-        --log-uri LOG_URI            Location in S3 to store logs from the job flow, e.g. s3n://mybucket/logs
-        --availability-zone A_Z      Specify the Availability Zone in which to launch the jobflow
-        --info INFO                  Specify additional info in JSON
-        --hadoop-version INFO        Specify the Hadoop Version to install
-        --plain-output               Return the job flow id from create step as simple text
-
-  Adding Jar Steps to Job Flows
-        --jar JAR                    Add a step that executes a jar
-        --wait-for-step              Wait for the step to finish
-        --main-class MAIN_CLASS      Specify main class for the JAR
-
-  Adding Streaming Steps to Job Flows
-        --stream                     Add a step that performs hadoop streaming
-        --input INPUT                Input to the steps, e.g. s3n://mybucket/input
-        --output OUTPUT              The output to the steps, e.g. s3n://mybucket/output
-        --mapper MAPPER              The mapper program or class
-        --cache CACHE_FILE           A file to load into the cache, e.g. s3n://mybucket/sample.py#sample.py
-        --cache-archive CACHE_FILE   A file to unpack into the cache, e.g. s3n://mybucket/sample.jar
-        --jobconf KEY=VALUE          Specify jobconf arguments to pass to streaming, e.g. mapred.task.timeout=800000
-        --reducer REDUCER            The reducer program or class
-
-  Job Flow Deugging Options
-        --enable-debugging           Enable job flow debugging (you must be signed up to SimpleDB for this to work)
-
-  Adding Pig steps to job flows
-        --pig-script                 Add a step that runs a Pig script
-        --pig-interactive            Add a step that sets up the job flow for an interactive (via SSH) pig session
-
-  Configuring a Hive on a JobFlow
-        --hive-site HIVE_SITE        Override Hive configuration with configuration from HIVE_SITE
-        --hive-script                Add a step that runs a Hive script
-        --hive-interactive           Add a step that sets up the job flow for an interactive (via SSH) hive session
-
-  Adding Steps from a Json File to Job Flows
-        --json FILE                  Add a sequence of steps stored in a json file
-        --param VARIABLE=VALUE       subsitute <variable> with value in the json file
-
-  Contacting the Master Node
-        --no-wait                    Don't wait for the Master node to start before executing scp or ssh
-        --ssh [COMMAND]              SSH to the master node and optionally run a command
-        --logs                       Display the step logs for the last executed step
-        --scp SRC                    Copy a file to the master node
-        --to DEST                    the destination to scp a file to
-
-  Settings common to all step types
-        --step-name STEP_NAME        Set name for the step
-        --step-action STEP_NAME      Action to take when step finishes. One of CANCEL_AND_WAIT, TERMINATE_JOB_FLOW or CONTINUE
-        --arg ARG                    Specify an argument to a bootstrap action, jar, streaming, pig-script or hive-script step
-        --args ARGS                  Specify a comma seperated list of arguments, e.g --args 1,2,3 would three arguments
-
-  Specifying Bootstrap Actions
-        --bootstrap-action SCRIPT    Run a bootstrap action script on all instances
-        --bootstrap-name NAME        Set the name of the bootstrap action
-Note --arg and --args are used to pass arguments to bootstrap actions
-
-  Listing and Describing Job Flows
-        --list                       List all job flows created in the last 2 days
-        --describe                   Dump a JSON description of the supplied job flows
-        --active                     List running, starting or shutting down job flows
-        --all                        List all job flows in the last 2 months
-        --nosteps                    Do not list steps when listing jobs
-        --state STATE                List job flows in STATE
-    -n, --max-results MAX_RESULTS    Maximum number of results to list
-
-  Terminating Job Flows
-        --terminate                  Terminate the job flow
-
-  Common Options
-    -j, --jobflow JOB_FLOW_ID
-        --job-flow-id
-    -c, --credentials CRED_FILE      File containing access-id and private-key
-    -a, --access-id ACCESS-ID        AWS Access Id
-    -k, --private-key PRIVATE-KEY    AWS Private Key
-    -v, --verbose                    Turn on verbose logging of program interaction
-
-  Uncommon Options
-        --debug                      Print stack traces when exceptions occur
-        --endpoint ENDPOINT          Specify the webservice endpoint to talk to
-        --region REGION              The region to use for the endpoint
-        --apps-path APPS_PATH        Specify s3:// path to the base of the emr public bucket to use. e.g s3://us-east-1.elasticmapreduce
-        --beta-path BETA_PATH        Specify s3:// path to the base of the emr public bucket to use for beta apps. e.g s3://beta.elasticmapreduce
-        --version                    Print a version string
-    -h, --help                       Show help message