elasticity 1.5 → 2.0

data/.rspec

@@ -1,2 +1,3 @@
 --color
---format progress
+--format progress
+--require spec_helper

data/.rvmrc

@@ -1 +1 @@
-rvm use ruby-1.9.2-p180@elasticity --create
+rvm use ruby-1.9.3-p194@elasticity --create

data/HISTORY.md

@@ -1,56 +1,79 @@
-### 1.5
+## 2.0 - June 26, 2012
+
+2.0 is a rewrite of the simplified API after a year's worth of daily use at [Sharethrough](http://www.sharethrough.com/).  We're investing heavily in our data processing infrastucture and many Elasticity feature ideas have come from those efforts.
+
+In order to move more quickly and support interesting features like a command-line interface, configuration-file-based launching, keep-alive clusters and more - a remodeling of the simplified API was done.  This is going to result in breaking changes to the API, hence the bump to 2.0.  I hope that most of you were using ```gem 'elasticity', '~> 1.5'``` in your Gemfile :)
+
+#### API Changes
+
++ The ```SimpleJob```-based API has been removed in favour of a more modular step-based approach using the "job flow" and "step" vernacular, in line with Amazon's own language.  If you're familiar with the AWS UI, using Elasticity will be a bit more straightforward.
++ The functionality provided by ```JobFlow``` and ```JobFlowStep``` has been transitioned to ```JobFlowStatus``` and ```JobFlowStatusStep``` respectively, clearing the path for use of ```JobFlow``` and ```JobFlowStep``` in job submission.
+
+#### New Features!
+
++ When submitting jobs via ```JobFlow``` API, it is now possible to specify the version of the AMI, whether or not the cluster is keep-alive, and the subnet ID (for launching into a VPC).  Keep in mind that AWS will error out if you specify an unsupported combination of AMI and Hadoop version.
++ The default version of Hadoop in ```JobFlow``` is now 0.20.205.  The previous default was 0.20 in case you'd like to set it yourself.
++ It is now possible to name Hadoop bootstrap actions, making it easier to understand the actions when looking in the AWS UI after a job is submitted.
+
+#### Under The Hood
+
++ AWS requests are now POSTs (thanks to [Menno van der Sman](https://github.com/menno)) in order to avoid server-imposed GET request size limits.  Rather than maintain two separate code paths for GET and POST, we decided to only support POST as there is no reason to support both.
++ Drastic simplification of the testing around EMR submission, reducing LoC (however important that metric is you :) and complexity by ~50%.
++ Development dependency updates: updated to ruby-1.9.3-p194 and rspec-2.10.  Removed dependency on VCR and WebMock (no longer using either of these).
+
+## 1.5
 
 + Added support for Hadoop bootstrap actions to all job types (Pig, Hive and Custom Jar).
 + Added support for REE 1.8.7-2011.12, Ruby 1.9.2 and 1.9.3.
 + Updated to the latest versions of all development dependencies (notably VCR 2).
 
-### 1.4.1
+## 1.4.1
 
-+ Added Elasticity::EMR#describe_jobflow("jobflow_id") for describing a specific job.  If you happen to run hundreds of EMR jobs, this makes retrieving jobflow status much faster than using Elasticity::EMR#describe_jobflowS which pulls down and parses XML status for hundreds of jobs.
++ Added ```Elasticity::EMR#describe_jobflow("jobflow_id")``` for describing a specific job.  If you happen to run hundreds of EMR jobs, this makes retrieving jobflow status much faster than using ```Elasticity::EMR#describe_jobflowS``` which pulls down and parses XML status for hundreds of jobs.
 
-### 1.4
+## 1.4
 
-+ Added Elasticity::CustomJarJob for launching "Custom Jar" jobs.
++ Added ```Elasticity::CustomJarJob``` for launching "Custom Jar" jobs.
 
-### 1.3.1
+## 1.3.1
 
 + Explicitly requiring 'time' (only a problem if you aren't running from within a Rails environment).
-+ Elasticity::JobFlow now exposes last_state_change_reason.
++ ```Elasticity::JobFlow``` now exposes ```last_state_change_reason```.
 
-### 1.3 (Contributions from Wouter Broekhof)
+## 1.3 (Contributions from Wouter Broekhof)
 
 + The default mode of communication is now via HTTPS.
-+ Elasticity::AwsRequest new option :secure => true|false (whether to use HTTPS).
-+ Elasticity::AwsRequest new option :region => eu-west-1|... (which region to run the EMR job).
-+ Elasticity::EMR#describe_jobflows now accepts additional params for filtering the jobflow query (see docs).
++ ```Elasticity::AwsRequest``` new option ```:secure => true|false``` (whether to use HTTPS).
++ ```Elasticity::AwsRequest``` new option ```:region => eu-west-1|...``` (which region to run the EMR job).
++ ```Elasticity::EMR#describe_jobflows``` now accepts additional params for filtering the jobflow query (see docs).
 
-### 1.2.2
+## 1.2.2
 
-+ HiveJob and PigJob now support configuring Hadoop options via .add_hadoop_bootstrap_action().
++ ```HiveJob``` and ```PigJob``` now support configuring Hadoop options via ```#add_hadoop_bootstrap_action()```.
 
-### 1.2.1
+## 1.2.1
 
 + Shipping up E_PARALLELS Pig variable with each invocation; reasonable default value for PARALLEL based on the number and type of instances configured.
 
-### 1.2
+## 1.2
 
-+ Added PigJob!
++ Added ```PigJob```!
 
-### 1.1.1
+## 1.1.1
 
-+ HiveJob critical bug fixed, now it works :)
-+ Added log_uri and action_on_failure as options to HiveJob.
-+ Added integration tests to HiveJob.
++ ```HiveJob``` critical bug fixed, now it works :)
++ Added ```log_uri``` and ```action_on_failure``` as options to ```HiveJob```.
++ Added integration tests to ```HiveJob```.
 
-### 1.1
+## 1.1
 
-+ Added HiveJob, a simplified way to launch basic Hive job flows.
++ Added ```HiveJob```, a simplified way to launch basic Hive job flows.
 + Added HISTORY.
 
-### 1.0.1
+## 1.0.1
 
 + Added LICENSE.
 
-### 1.0
+## 1.0
 
 + Released!

data/LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2011 Robert Slifka
+   Copyright 2011-2012 Robert Slifka
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

data/README.md

@@ -1,78 +1,131 @@
-Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently wrap the API operations in a manner that makes working with EMR job flows from Ruby more productive and more enjoyable, without having to understand the nuts and bolts of the EMR REST API.  At the very least, using Elasticity allows you to easily experiment with the EMR API :)
+Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently map the EMR REST API calls to higher level operations that make working with job flows more productive and more enjoyable.
 
-[![Build Status](https://secure.travis-ci.org/rslifka/elasticity.png)](http://travis-ci.org/rslifka/elasticity)
+[![Build Status](https://secure.travis-ci.org/rslifka/elasticity.png)](http://travis-ci.org/rslifka/elasticity) REE, 1.8.7, 1.9.2, 1.9.3
 
-# Installation and Usage
+Elasticity provides two ways to access EMR:
 
-<pre>
+* **Indirectly through a JobFlow-based API**. This README discusses the Elasticity API.
+* **Directly through access to the EMR REST API**. The less-discussed hidden darkside... I use this to enable the Elasticity API though it is not documented save for RubyDoc available at the the RubyGems [auto-generated documentation site](http://rubydoc.info/gems/elasticity/frames).  Be forewarned: Making the calls directly requires that you understand how to structure EMR requests at the Amazon API level and from experience I can tell you there are more fun things you could be doing :)  Scroll to the end for more information on the Amazon API. 
+
+# Installation
+
+```
   gem install elasticity
-</pre>
+```
 
-All you have to do is <code>require 'elasticity'</code> and you're all set!
+or in your Gemfile
 
-# Simplified API Reference
+```
+  gem 'elasticity', '~> 2.0'
+```
 
-Elasticity currently provides simplified access to launching Hive, Pig and Custom Jar job flows, specifying several default values that you may optionally override:
+This will ensure that you protect yourself from API changes, which will only be made in major revisions.
 
-<pre>
-  @action_on_failure = "TERMINATE_JOB_FLOW"
-  @ec2_key_name = "default"
-  @hadoop_version = "0.20"
-  @instance_count = 2
-  @master_instance_type = "m1.small"
-  @name = "Elasticity Job"
-  @slave_instance_type = "m1.small"
-</pre>
+# Kicking Off a Job
 
-These are all accessible from the simplified jobs.  See the PigJob description for an example.
+When using the EMR UI, there are several sample jobs that Amazon supplies.  The assets for these sample jobs are hosted on S3 and publicly available meaning you can run this code as-is (supplying your AWS credentials appropriately) and ```JobFlow#run``` will return the ID of the job flow.
 
-### Bootstrap Actions
+```
+require 'elasticity'
 
-You can also configure Hadoop options with add_hadoop_bootstrap_action().
+# Create a job flow with your AWS credentials
+jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
 
-<pre>
-  pig = Elasticity::PigJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  pig.add_hadoop_bootstrap_action("-m", "mapred.job.reuse.jvm.num.tasks=120")
-  ...
-</pre>
+# This is the first step in the jobflow - running a custom jar
+step = Elasticity::CustomJarStep.new('s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar')
+
+# Here are the arguments to pass to the jar
+c.arguments = %w(s3n://elasticmapreduce/samples/cloudburst/input/s_suis.br s3n://elasticmapreduce/samples/cloudburst/input/100k.br s3n://slif-output/cloudburst/output/2012-06-22 36 3 0 1 240 48 24 24 128 16)
+
+# Add the step to the jobflow
+jobflow.add_step(step)
+
+# Let's go!
+jobflow.run
+```
+
+Note that this example is only for ```CustomJarStep```.  ```PigStep``` and ```HiveStep``` will have different means of passing parameters.
+
+# Working with Job Flows
+
+Job flows are the center of the EMR universe.  The general order of operations is:
+
+  1. Create a job flow.
+  1. Specify options.
+  1. Add bootstrap actions.
+  1. Create steps.
+  1. Run the job flow.
+  1. (optional) Add additional steps.
+  1. (optional) Shutdown the job flow.
+
+## 1 - Creating Job Flows
+
+Only your AWS credentials are needed.
+
+```
+jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
+```
+
+## 2 - Specifying Job Flow Options
+
+Configuration job flow options, shown below with default values.  Note that these defaults are subject to change - they are reasonable defaults at the time(s) I work on them (e.g. the latest version of Hadoop).
+
+These options are sent up as part of job flow submission (i.e. ```JobFlow#run```), so be sure to configure these before running the job.
+
+```
+jobflow.action_on_failure                 = 'TERMINATE_JOB_FLOW'
+jobflow.ami_version                       = 'latest'
+jobflow.ec2_key_name                      = 'default'
+jobflow.ec2_subnet_id                     = nil
+jobflow.hadoop_version                    = '0.20.205'
+jobflow.instance_count                    = 2
+jobflow.keep_job_flow_alive_when_no_steps = true
+jobflow.log_uri                           = nil
+jobflow.master_instance_type              = 'm1.small'
+jobflow.name                              = 'Elasticity Job Flow'
+jobflow.slave_instance_type               = 'm1.small'
+```
+
+## 3 - Adding Bootstrap Actions
 
-## Hive
+Bootstrap actions are run as part of setting up the job flow, so be sure to configure these before running the job.
 
-HiveJob allows you to quickly launch Hive jobs without having to understand the ins and outs of the EMR API.  Specify only the Hive script location and (optionally) variables to make available to the Hive script.
+```
+[
+  Elasticity::HadoopBootstrapAction.new('-m', 'mapred.map.tasks=101'),
+  Elasticity::HadoopBootstrapAction.new('-m', 'mapred.reduce.child.java.opts=-Xmx200m')
+  Elasticity::HadoopBootstrapAction.new('-m', 'mapred.tasktracker.map.tasks.maximum=14')
+].each do |action|
+  jobflow.add_bootstrap_action(action)
+end
+```
 
-<pre>
-  hive = Elasticity::HiveJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  hive.run("s3n://slif-hive/test.q", {
-    "LIB"    => "s3n://slif-test/lib",
-    "OUTPUT" => "s3n://slif-test/output"
-  })
-  
-  > "j-129V5AQFMKO1C"
-</pre>
+## 4 - Adding Steps
 
-## Pig
+Each type of step has a default name that can be overridden (the :name field).  Apart from that, steps are configured differently - exhaustively described below.
 
-Like HiveJob, PigJob allows you to quickly launch Pig jobs :)
+### Adding a Pig Step
 
-<pre>
-  pig = Elasticity::PigJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  pig.log_uri = "s3n://slif-elasticity/pig-apache/logs"
-  pig.ec2_key_name = "slif_dev"
-  pig.run("s3n://elasticmapreduce/samples/pig-apache/do-reports.pig", {
-    "INPUT"  => "s3n://elasticmapreduce/samples/pig-apache/input",
-    "OUTPUT" => "s3n://slif-elasticity/pig-apache/output/2011-05-04"
-  })
-  
-  > "j-16PZ24OED71C6"
-</pre>
+```
+# Path to the Pig script
+pig_step = Elasticity::PigStep.new('s3n://mybucket/script.pig')
 
-### PARALLEL
+# (optional) These variables are available during the execution of your script
+pig_step.variables = {
+  'VAR1' => 'VALUE1',
+  'VAR2' => 'VALUE2'
+}
+
+jobflow.add_step(pig_step)
+```
+
+#### PARALLEL
 
 Given the importance of specifying a reasonable value for [the number of parallel reducers](http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features PARALLEL), Elasticity calculates and passes through a reasonable default up with every invocation in the form of a script variable called E_PARALLELS.  This default value is based off of the formula in the Pig Cookbook and the number of reducers AWS configures per instance.
 
 For example, if you had 8 instances in total and your slaves were m1.xlarge, the value is 26 (as shown below).
 
-<pre>
+```
   s3://elasticmapreduce/libs/pig/pig-script
     --run-pig-script
       --args
@@ -80,309 +133,104 @@ For example, if you had 8 instances in total and your slaves were m1.xlarge, the
         -p OUTPUT=s3n://slif-elasticity/pig-apache/output/2011-05-04
         -p E_PARALLELS=26
     s3n://elasticmapreduce/samples/pig-apache/do-reports.pig
-</pre>
+```
 
 Use this as you would any other Pig variable.
 
-<pre>
+```
   A = LOAD 'myfile' AS (t, u, v);
   B = GROUP A BY t PARALLEL $E_PARALLELS;
   ...
-</pre>
+```
+
+### Adding a Hive Step
+
+```
+# Path to the Hive Script
+hive_step = Elasticity::HiveStep.new('s3n://mybucket/script.hql')
+
+# (optional) These variables are available during the execution of your script
+hive_step.variables = {
+  'VAR1' => 'VALUE1',
+  'VAR2' => 'VALUE2'
+}
+
+jobflow.add_step(hive_step)
+```
+
+### Adding a Custom Jar Step
+
+```
+# Path to your jar
+jar_step = Elasticity::CustomJarStep.new('s3n://mybucket/my.jar')
+
+# (optional) Arguments passed to the jar
+jar_step.arguments = ['arg1', 'arg2']
 
-## Custom Jar
+jobflow.add_step(jar_step)
+```
 
-Custom jar jobs are also available.  To kick off a custom job, specify the path to the jar and any arguments you'd like passed to the jar.
+## 5 - Running the Job Flow
 
-<pre>
-  custom_jar = Elasticity::CustomJarJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  custom_jar.log_uri = "s3n://slif-test/output/logs"
-  custom_jar.action_on_failure = "TERMINATE_JOB_FLOW"
-  jobflow_id = custom_jar.run('s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar', [
-      "s3n://elasticmapreduce/samples/cloudburst/input/s_suis.br",
-      "s3n://elasticmapreduce/samples/cloudburst/input/100k.br",
-      "s3n://slif_hadoop_test/cloudburst/output/2011-12-09",
-  ])
-  
-  > "j-1IU6NM8OUPS9I"
-</pre>
+Submit the job flow to Amazon, storing the ID of the running job flow.
 
-Custom jar jobs support arbitrary entry points.  Specify the class on which to call main() either via the JAR manifest or as the first argument to the job:
+```
+jobflow_id = jobflow.run
+```
 
-<pre>
-  Elasticity::CustomJarJob.new(key, secret).run(s3_jar_path, ['MyCustomClass', 'arg1', 'arg2'])
-</pre>
+## 6 - Adding Additional Steps (optional)
 
-# Amazon API Reference
+Steps can be added to a running jobflow just by calling ```#add_step``` on the job flow exactly how you add them prior to submitting the job.
+
+## 7 - Shutting Down the Job Flow (optional)
+
+By default, job flows are set to terminate when there are no more running steps.  You can tell the job flow to stay alive when it has nothing left to do:
+
+```
+jobflow.keep_job_flow_alive_when_no_steps = true
+```
+
+If that's the case, or if you'd just like to terminate a running jobflow before waiting for it to finish:
+
+```
+jobflow.shutdown
+```
+
+# Amazon EMR Documentation
 
 Elasticity wraps all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).
 
-You may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.  Direct access is described below the API catalog.
+You may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.
 
-In addition to the [AWS EMR subsite](http://aws.amazon.com/elasticmapreduce/), there are three primary resources of reference information for EMR:
+In addition to the [AWS EMR site](http://aws.amazon.com/elasticmapreduce/), there are three primary resources of reference information for EMR:
 
 * [Amazon EMR Getting Started Guide](http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/)
 * [Amazon EMR Developer Guide](http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/)
 * [Amazon EMR API Reference](http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/)
 
-Unfortunately, the documentation is sometimes incorrect and sometimes missing.  E.g. the allowable values for AddInstanceGroups are present in the [PDF](http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf) version of the API reference but not in the [HTML](http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/) version.  Elasticity implements the API as specified in the PDF reference as that is the most complete description I could find.
-
-## AddInstanceGroups
-
-AddInstanceGroups adds a group of instances to an existing job flow.  The available instance configuration options are listed in the EMR API reference.  They've been converted to be more Ruby-like in the wrappers, as shown in the example below.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  instance_group_config = {
-    :instance_count => 1,
-    :instance_role => "TASK",
-    :instance_type => "m1.small",
-    :market => "ON_DEMAND",
-    :name => "Go Canucks Go!"
-  }
-  emr.add_instance_groups("j-26LIXPUNSC0M3", [instance_group_config])
-  
-  > ["ig-E7C8MGA2ULQ1"]
-</pre>
-
-Some combinations of the options will be rejected by Amazon and some once-valid options will sometimes be rejected if they not relevant to the current state of the job flow (e.g. duplicate addition of TASK groups to the same job flow).
-
-<pre>
-  emr.add_instance_groups("j-26LIXPUNSC0M3", [instance_group_config])
-  
-  > Task instance group already exists in the job flow, cannot add more task groups
-</pre>
-
-## AddJobFlowSteps
-
-AddJobFlowSteps adds the specified steps to the specified job flow.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  jobflow_id = emr.run_job_flow(...)
-  emr.add_jobflow_steps(jobflow_id, {
-    :steps => [
-      {
-        :action_on_failure => "TERMINATE_JOB_FLOW",
-        :hadoop_jar_step => {
-          :args => [
-            "s3://elasticmapreduce/libs/pig/pig-script",
-            "--base-path",
-            "s3://elasticmapreduce/libs/pig/",
-            "--install-pig"
-          ],
-          :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
-        },
-        :name => "Setup Pig"
-      }
-    ]
-  })
-</pre>
-
-## describe_jobflow (Elasticity Convenience Method)
-
-This is a convenience methods that wraps DescribeJobFlow to return the status of a single job.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  jobflow = emr.describe_jobflow("j-129V5AQFMKO1C")
-  p jobflow.jobflow_id
-  > "j-129V5AQFMKO1C"
-  p jobflow.name
-  > "Elasticity Test Job"
-</pre>
-
-## DescribeJobFlows
-
-DescribeJobFlows returns detailed information as to the state of all jobs.  Currently this is wrapped in an <code>Elasticity::JobFlow</code> that contains the <code>name</code>, <code>jobflow_id</code> and <code>state</code>.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  jobflows = emr.describe_jobflows
-  p jobflows.map(&:name)
-
-  > ["Hive Test", "Pig Test", "Interactive Hadoop", "Interactive Hive"]
-</pre>
-
-## ModifyInstanceGroups
-
-A job flow contains several "instance groups" of various types.  These instances are where the work for your EMR task occurs.  After a job flow has been created, you can find these instance groups in the AWS web UI by clicking on a job flow and then clicking on the "Instance Groups" tab.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.modify_instance_groups({"ig-2T1HNUO61BG3O" => 3})
-</pre>
-
-If there's an error, you'll receive an ArgumentError containing the message from Amazon.  For example if you attempt to modify an instance group that's part of a terminated job flow:
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.modify_instance_groups({"ig-some_terminated_group" => 3})
-  
-  > ArgumentError: An instance group may only be modified when the job flow is running or waiting
-</pre>
-
-Or if you attempt to increase the instance count of the MASTER instance group:
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.modify_instance_groups({"ig-some_terminated_group" => 3})
-  
-  > ArgumentError: A master instance group may not be modified
-</pre>
-
-## RunJobFlow
-
-RunJobFlow creates and starts a new job flow.  Specifying the arguments to RunJobFlow is a bit of a hot mess at the moment, requiring you to understand the EMR syntax as well as the data structure for specifying jobs.  Here's a beefy example:
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  jobflow_id = emr.run_job_flow({
-    :name => "Elasticity Test Flow (EMR Pig Script)",
-    :instances => {
-      :ec2_key_name => "sharethrough-dev",
-      :hadoop_version => "0.20",
-      :instance_count => 2,
-      :master_instance_type => "m1.small",
-      :placement => {
-        :availability_zone => "us-east-1a"
-      },
-      :slave_instance_type => "m1.small",
-    },
-    :steps => [
-      {
-        :action_on_failure => "TERMINATE_JOB_FLOW",
-        :hadoop_jar_step => {
-          :args => [
-            "s3://elasticmapreduce/libs/pig/pig-script",
-            "--base-path",
-            "s3://elasticmapreduce/libs/pig/",
-            "--install-pig"
-          ],
-          :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
-        },
-        :name => "Setup Pig"
-      },
-        {
-          :action_on_failure => "TERMINATE_JOB_FLOW",
-          :hadoop_jar_step => {
-            :args => [
-              "s3://elasticmapreduce/libs/pig/pig-script",
-              "--run-pig-script",
-              "--args",
-              "-p",
-              "INPUT=s3n://elasticmapreduce/samples/pig-apache/input",
-              "-p",
-              "OUTPUT=s3n://slif-elasticity/pig-apache/output/2011-04-19",
-              "s3n://elasticmapreduce/samples/pig-apache/do-reports.pig"
-            ],
-            :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
-          },
-          :name => "Run Pig Script"
-        }
-    ]
-  })
-
-  > "j-129V5AQFMKO1C"
-</pre>
-
-Currently Elasticity doesn't do much to ease this pain although this is what I would like to focus on in coming releases.  Feel free to ship ideas my way.  In the meantime, have a look at the EMR API [PDF](http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf) under the RunJobFlow action and riff off of the example here.
-
-## SetTerminationProtection
-
-Enable or disable "termination protection" on the specified job flows.  Termination protection prevents a job flow from from being terminated by any user-initiated action.  
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5"])
-</pre>
-
-To disable termination protection, specify false as the second parameter.
-
-<pre>
-  emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5"], false)
-</pre>
-
-## TerminateJobFlows
-
-Terminate the specified job flow.  When the job flow '''exists''', you will receive no output.  This is because Amazon does not return anything other than a 200 when you terminate a job flow :)  You'll want to continuously poll with DescribeJobFlows to see when the job was actually terminated.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.terminate_jobflows("j-BOWBV7884XD0")
-</pre>
-
-When the job flow '''doesn't exist''':
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.terminate_jobflows("no-flow")
-  
-  > ArgumentError: Job flow 'no-flow' does not exist.
-</pre>
-
-# Direct Response Access
-
-If you're fine with Elasticity's invocation wrapping and would prefer to get at the resulting XML rather than the wrapped response, throw a block our way and we'll yield the result.  This still saves you the trouble of having to create the params and sign the request yet gives you direct access to the response XML for your parsing pleasure.
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  emr.describe_jobflows{|xml| puts xml[0..77]}
-  
-  > &lt;DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/200...
-</pre>
-
-# Direct Request/Response Access
-
-If you're chomping at the bit to initiate some EMR functionality that isn't wrapped (or isn't wrapped in a way you prefer :) feel free to access the AWS EMR API directly by using <code>EMR.direct()</code>.  You can find the allowed values in Amazon's EMR API [developer documentation](http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html).
-
-<pre>
-  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
-  params = {"Operation" => "DescribeJobFlows"}
-  result_xml = emr.direct(params)
-  result_xml[0..78]
-  
-  > &lt;DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009...
-</pre>
-
-# Something Borrowed...
-
-AWS signing was used from [RightScale's](http://www.rightscale.com/) amazing [right_aws gem](https://github.com/rightscale/right_aws) which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.
-
-Used camelize from ActiveSupport as well, thank you \Rails :)
+Unfortunately, the documentation is sometimes incorrect and sometimes missing.  E.g. the allowable values for ```AddInstanceGroups``` are present in the [PDF](http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf) version of the API reference but not in the [HTML](http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/) version.  Elasticity implements the API as specified in the PDF reference as that is the most complete description I could find.
 
 # Thanks!
 
-Thanks to the following people who have contributed patches or helpful suggestions:
+* AWS signing was used from [RightScale's](http://www.rightscale.com/) amazing [right_aws gem](https://github.com/rightscale/right_aws) which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.
+* <code>camelize</code> was used from ActiveSupport to assist in converting parmeters to AWS request format.
+* Thanks to the following people who have contributed patches or helpful suggestions: [Ryan Weald](https://github.com/rweald), [Aram Price](https://github.com/aramprice/), [Wouter Broekhof](https://github.com/wouter/) and [Menno van der Sman](https://github.com/menno)
 
-+ [Aram Price](https://github.com/aramprice/)
-+ [Wouter Broekhof](https://github.com/wouter/)
 
 # License
 
-<pre>
+```
   Copyright 2011-2012 Robert Slifka
 
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at
 
-     http://www.apache.org/licenses/LICENSE-2.0
+    http://www.apache.org/licenses/LICENSE-2.0
 
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-</pre>
-
-### Development Notes for Slif
-
-[Versioning Guide](http://docs.rubygems.org/read/chapter/7#page27), c/o [@brokenladder](https://twitter.com/#!/brokenladder)
-
-<pre>
-  rake build    # Build lorem-0.0.2.gem into the pkg directory
-  rake install  # Build and install lorem-0.0.2.gem into system gems
-  rake release  # Create tag v0.0.2 and build
-                # and push lorem-0.0.2.gem to Rubygems
-</pre>
+```