elasticity 0.5 → 0.6

data/README.mediawiki

@@ -1,8 +1,8 @@
 Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently wrap the API operations in a manner that makes working with EMR job flows from Ruby more enjoyable.  At the very least, using Elasticity allows you to easily experiment with the EMR API :)
 
-'''BACKLOG''': Have a look at the backlog  [https://www.pivotaltracker.com/projects/272429 here] (Pivotal Tracker).
+'''BACKLOG''': Have a look at the [https://www.pivotaltracker.com/projects/272429 backlog] to see where this is headed.
 
-'''CREDITS''': AWS signing was used from [http://www.rightscale.com/ RightScale's] amazing [https://github.com/rightscale/right_aws right_aws gem] which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.
+'''CREDITS''': AWS signing was used from [http://www.rightscale.com/ RightScale's] amazing [https://github.com/rightscale/right_aws right_aws gem] which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.  Used camelize from ActiveSupport as well, thank you \Rails :)
 
 = Installation and Usage =
 
@@ -14,9 +14,9 @@ All you have to do is <code>require 'elasticity'</code> and you're all set!
 
 = API Reference =
 
-Elasticity wraps (well, eventually!) all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).
+Elasticity wraps all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).
 
-Alternatively, you may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.  Direct access is described below the API catalog.
+You may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.  Direct access is described below the API catalog.
 
 In addition to the [http://aws.amazon.com/elasticmapreduce/ AWS EMR subsite], there are three primary resources of reference information for EMR:
 
@@ -31,12 +31,13 @@ Unfortunately, the documentation is sometimes incorrect and sometimes missing.
 AddInstanceGroups adds a group of instances to an existing job flow.  The available instance configuration options are listed in the EMR API reference.  They've been converted to be more Ruby-like in the wrappers, as shown in the example below.
 
 <pre>
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   instance_group_config = {
-    :count  => 1,
-    :role   => "TASK",
-    :type   => "m1.small",
+    :instance_count => 1,
+    :instance_role => "TASK",
+    :instance_type => "m1.small",
     :market => "ON_DEMAND",
-    :name   => "Go Canucks!"
+    :name => "Go Canucks Go!"
   }
   emr.add_instance_groups("j-26LIXPUNSC0M3", [instance_group_config])
   
@@ -60,7 +61,7 @@ Some combinations of the options will be rejected by Amazon and some once-valid
 DescribeJobFlows returns detailed information as to the state of all jobs.  Currently this is wrapped in an <code>Elasticity::JobFlow</code> that contains the <code>name</code>, <code>jobflow_id</code> and <code>state</code>.  
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   jobflows = emr.describe_jobflows
   p jobflows.map(&:name)
   
@@ -72,14 +73,14 @@ DescribeJobFlows returns detailed information as to the state of all jobs.  Curr
 A job flow contains several "instance groups" of various types.  These instances are where the work for your EMR task occurs.  After a job flow has been created, you can find these instance groups in the AWS web UI by clicking on a job flow and then clicking on the "Instance Groups" tab.
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.modify_instance_groups({"ig-2T1HNUO61BG3O" => 3})
 </pre>
 
 If there's an error, you'll receive an ArgumentError containing the message from Amazon.  For example if you attempt to modify an instance group that's part of a terminated job flow:
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.modify_instance_groups({"ig-some_terminated_group" => 3})
   
   > ArgumentError: An instance group may only be modified when the job flow is running or waiting
@@ -88,7 +89,7 @@ If there's an error, you'll receive an ArgumentError containing the message from
 Or if you attempt to increase the instance count of the MASTER instance group:
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.modify_instance_groups({"ig-some_terminated_group" => 3})
   
   > ArgumentError: A master instance group may not be modified
@@ -96,21 +97,89 @@ Or if you attempt to increase the instance count of the MASTER instance group:
 
 == RunJobFlow ==
 
-''(not yet supported)''
+RunJobFlow creates and starts a new job flow.  Specifying the arguments to RunJobFlow is a bit of a hot mess at the moment, requiring you to understand the EMR syntax as well as the data structure for specifying jobs.  Here's a beefy example:
+
+<pre>
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
+  jobflow_id = emr.run_job_flow({
+    :name => "Elasticity Test Flow (EMR Pig Script)",
+    :instances => {
+      :ec2_key_name => "sharethrough-dev",
+      :hadoop_version => "0.20",
+      :instance_count => 2,
+      :master_instance_type => "m1.small",
+      :placement => {
+        :availability_zone => "us-east-1a"
+      },
+      :slave_instance_type => "m1.small",
+    },
+    :steps => [
+      {
+        :action_on_failure => "TERMINATE_JOB_FLOW",
+        :hadoop_jar_step => {
+          :args => [
+            "s3://elasticmapreduce/libs/pig/pig-script",
+              "--base-path",
+              "s3://elasticmapreduce/libs/pig/",
+              "--install-pig"
+          ],
+          :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
+        },
+        :name => "Setup Pig"
+      },
+        {
+          :action_on_failure => "TERMINATE_JOB_FLOW",
+          :hadoop_jar_step => {
+            :args => [
+              "s3://elasticmapreduce/libs/pig/pig-script",
+                "--run-pig-script",
+                "--args",
+                "-p",
+                "INPUT=s3n://elasticmapreduce/samples/pig-apache/input",
+                "-p",
+                "OUTPUT=s3n://slif-elasticity/pig-apache/output/2011-04-19",
+                "s3n://elasticmapreduce/samples/pig-apache/do-reports.pig"
+            ],
+            :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
+          },
+          :name => "Run Pig Script"
+        }
+    ]
+  })
+
+  > "j-129V5AQFMKO1C"
+</pre>
+
+Currently Elasticity doesn't do much to ease this pain although this is what I would like to focus on in coming releases.  Feel free to ship ideas my way.  In the meantime, have a look at the EMR API [http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf PDF] under the RunJobFlow action and riff off of the example here.
+
+== SetTerminationProtection ==
+
+Enable or disable "termination protection" on the specified job flows.  Termination protection prevents a job flow from from being terminated by any user-initiated action.  
+
+<pre>
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
+  emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5"])
+</pre>
+
+To disable termination protection, specify false as the second parameter.
+
+<pre>
+  emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5"], false)
+</pre>
 
 == TerminateJobFlows ==
 
-When the job flow '''exists''', you will receive no output.  This is because Amazon does not return anything other than a 200 when you terminate a job flow :)  You'll want to continuously poll with DescribeJobFlows to see when the job was actually terminated.
+Terminate the specified job flow.  When the job flow '''exists''', you will receive no output.  This is because Amazon does not return anything other than a 200 when you terminate a job flow :)  You'll want to continuously poll with DescribeJobFlows to see when the job was actually terminated.
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.terminate_jobflows("j-BOWBV7884XD0")
 </pre>
 
 When the job flow '''doesn't exist''':
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.terminate_jobflows("no-flow")
   
   > ArgumentError: Job flow 'no-flow' does not exist.
@@ -121,7 +190,7 @@ When the job flow '''doesn't exist''':
 If you're fine with Elasticity's invocation wrapping and would prefer to get at the resulting XML rather than the wrapped response, throw a block our way and we'll yield the result.  This still saves you the trouble of having to create the params and sign the request yet gives you direct access to the response XML for your parsing pleasure.
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.describe_jobflows{|xml| puts xml[0..77]}
   
   > <DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/200
@@ -132,7 +201,7 @@ If you're fine with Elasticity's invocation wrapping and would prefer to get at
 If you're chomping at the bit to initiate some EMR functionality that isn't wrapped (or isn't wrapped in a way you prefer :) feel free to access the AWS EMR API directly by using <code>EMR.direct()</code>.  You can find the allowed values in Amazon's EMR API [http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html developer documentation].
 
 <pre>
-  emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+  emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   params = {"Operation" => "DescribeJobFlows"}
   result_xml = emr.direct(params)
   result_xml[0..78]

data/lib/elasticity/emr.rb

@@ -8,7 +8,7 @@ module Elasticity
 
     # Lists all jobflows in all states.
     def describe_jobflows
-      aws_result = @aws_request.aws_emr_request({"Operation" => "DescribeJobFlows"})
+      aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(:operation => "DescribeJobFlows"))
       xml_doc = Nokogiri::XML(aws_result)
       xml_doc.remove_namespaces!
       yield aws_result if block_given?
@@ -22,11 +22,11 @@ module Elasticity
     #
     # instance_group_config = {
     #   :bid_price => 5,
-    #   :count => 1,
-    #   :role => "TASK",
-    #   :type => "m1.small",
+    #   :instance_count => 1,
+    #   :instance_role => "TASK",
     #   :market => "SPOT",
     #   :name => "Go Canucks Go!"
+    #   :type => "m1.small",
     # }
     #
     # add_instance_groups takes an array of {}.
@@ -36,19 +36,12 @@ module Elasticity
     #  ["ig-2GOVEN6HVJZID", "ig-1DU9M2UQMM051", "ig-3DZRW4Y2X4S", ...]
     def add_instance_groups(jobflow_id, instance_group_configs)
       params = {
-        "Operation" => "AddInstanceGroups",
-        "JobFlowId" => jobflow_id
+        :operation => "AddInstanceGroups",
+        :job_flow_id => jobflow_id,
+        :instance_groups => instance_group_configs
       }
-      instance_group_configs.each_with_index do |ig_config, index|
-        params.merge!("InstanceGroups.member.#{index+1}.BidPrice" => ig_config[:bid_price]) if ig_config[:bid_price]
-        params.merge!("InstanceGroups.member.#{index+1}.InstanceCount" => ig_config[:count]) if ig_config[:count]
-        params.merge!("InstanceGroups.member.#{index+1}.InstanceRole" => ig_config[:role]) if ig_config[:role]
-        params.merge!("InstanceGroups.member.#{index+1}.InstanceType" => ig_config[:type]) if ig_config[:type]
-        params.merge!("InstanceGroups.member.#{index+1}.Market" => ig_config[:market]) if ig_config[:market]
-        params.merge!("InstanceGroups.member.#{index+1}.Name" => ig_config[:name]) if ig_config[:name]
-      end
       begin
-        aws_result = @aws_request.aws_emr_request(params)
+        aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(params))
         xml_doc = Nokogiri::XML(aws_result)
         xml_doc.remove_namespaces!
         instance_group_ids = []
@@ -58,7 +51,7 @@ module Elasticity
         yield aws_result if block_given?
         instance_group_ids
       rescue RestClient::BadRequest => e
-        raise ArgumentError, parse_error_response(e.http_body)
+        raise ArgumentError, EMR.parse_error_response(e.http_body)
       end
     end
 
@@ -71,18 +64,53 @@ module Elasticity
     #
     # {"ig-1" => 40, "ig-2" => 5, ...}
     def modify_instance_groups(instance_group_config)
-      params = {"Operation" => "ModifyInstanceGroups"}
-      instance_group_config.keys.each_with_index do |instance_group, index|
-        params.merge!(
-          "InstanceGroups.member.#{index+1}.InstanceGroupId" => instance_group,
-          "InstanceGroups.member.#{index+1}.InstanceCount" => instance_group_config[instance_group]
-        )
+      params = {
+        :operation => "ModifyInstanceGroups",
+        :instance_groups => instance_group_config.map { |k, v| {:instance_group_id => k, :instance_count => v} }
+      }
+      begin
+        aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(params))
+        yield aws_result if block_given?
+      rescue RestClient::BadRequest => e
+        raise ArgumentError, EMR.parse_error_response(e.http_body)
       end
+    end
+
+    # TODO DOCUMENT ME
+    def run_job_flow(job_flow_config)
+      params = {
+        :operation => "RunJobFlow",
+      }.merge!(job_flow_config)
       begin
-        aws_result = @aws_request.aws_emr_request(params)
+        aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(params))
         yield aws_result if block_given?
+        xml_doc = Nokogiri::XML(aws_result)
+        xml_doc.remove_namespaces!
+        xml_doc.xpath("/RunJobFlowResponse/RunJobFlowResult/JobFlowId").text
       rescue RestClient::BadRequest => e
-        raise ArgumentError, parse_error_response(e.http_body)
+        raise ArgumentError, EMR.parse_error_response(e.http_body)
+      end
+    end
+
+    # Enabled or disable "termination protection" on the specified job flows.
+    # Termination protection prevents a job flow from being terminated by a
+    # user initiated action, although the job flow will still terminate
+    # naturally.
+    #
+    # Takes an [] of job flow IDs.
+    #
+    # ["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5", ...]
+    def set_termination_protection(jobflow_ids, protection_enabled=true)
+      params = {
+        :operation => "SetTerminationProtection",
+        :termination_protected => protection_enabled,
+        :job_flow_ids => jobflow_ids
+      }
+      begin
+        aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(params))
+        yield aws_result if block_given?
+      rescue RestClient::BadRequest => e
+        raise ArgumentError, EMR.parse_error_response(e.http_body)
       end
     end
 
@@ -91,32 +119,77 @@ module Elasticity
     # the state of the jobflow.  Raises ArgumentError if the specified job
     # flow does not exist.
     def terminate_jobflows(jobflow_id)
+      params = {
+        :operation => "TerminateJobFlows",
+        :job_flow_ids => [jobflow_id]
+      }
       begin
-        aws_result = @aws_request.aws_emr_request({
-          "Operation" => "TerminateJobFlows",
-          "JobFlowIds.member.1" => jobflow_id
-        })
+        aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(params))
         yield aws_result if block_given?
       rescue RestClient::BadRequest
         raise ArgumentError, "Job flow '#{jobflow_id}' does not exist."
       end
     end
 
-    # Pass the specified params hash directly through to the AWS request
-    # URL.  Use this if you want to perform an operation that hasn't yet
-    # been wrapped by Elasticity or you just want to see the response
-    # XML for yourself :)
+    # Pass the specified params hash directly through to the AWS request URL.
+    # Use this if you want to perform an operation that hasn't yet been wrapped
+    # by Elasticity or you just want to see the response XML for yourself :)
     def direct(params)
       @aws_request.aws_emr_request(params)
     end
 
     private
 
-    def parse_error_response(error_xml)
-      xml_doc = Nokogiri::XML(error_xml)
-      xml_doc.remove_namespaces!
-      xml_doc.xpath("/ErrorResponse/Error/Message").text
+    class << self
+
+      # AWS error responses all follow the same form.  Extract the message from
+      # the error document.
+      def parse_error_response(error_xml)
+        xml_doc = Nokogiri::XML(error_xml)
+        xml_doc.remove_namespaces!
+        xml_doc.xpath("/ErrorResponse/Error/Message").text
+      end
+
+      # Since we use the same structure as AWS, we can generate AWS param names
+      # from the Ruby versions of those names (and the param nesting).
+      def convert_ruby_to_aws(params)
+        result = {}
+        params.each do |key, value|
+          case value
+            when Array
+              prefix = "#{camelize(key.to_s)}.member"
+              value.each_with_index do |item, index|
+                if item.is_a?(String)
+                  result["#{prefix}.#{index+1}"] = item
+                else
+                  convert_ruby_to_aws(item).each do |nested_key, nested_value|
+                    result["#{prefix}.#{index+1}.#{nested_key}"] = nested_value
+                  end
+                end
+              end
+            when Hash
+              prefix = "#{camelize(key.to_s)}"
+              convert_ruby_to_aws(value).each do |nested_key, nested_value|
+                result["#{prefix}.#{nested_key}"] = nested_value
+              end
+            else
+              result[camelize(key.to_s)] = value
+          end
+        end
+        result
+      end
+
+      # (Used from Rails' ActiveSupport)
+      def camelize(lower_case_and_underscored_word, first_letter_in_uppercase = true)
+        if first_letter_in_uppercase
+          lower_case_and_underscored_word.to_s.gsub(/\/(.?)/) { "::" + $1.upcase }.gsub(/(^|_)(.)/) { $2.upcase }
+        else
+          lower_case_and_underscored_word.first + camelize(lower_case_and_underscored_word)[1..-1]
+        end
+      end
+
     end
 
   end
+
 end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = "0.5"
+  VERSION = "0.6"
 end

data/spec/fixtures/vcr_cassettes/run_jobflow/word_count.yml

@@ -0,0 +1,35 @@
+--- 
+- !ruby/struct:VCR::HTTPInteraction 
+  request: !ruby/struct:VCR::Request 
+    method: :get
+    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&Instances\.Ec2KeyName=sharethrough-dev&Instances\.HadoopVersion=0\.20&Instances\.InstanceCount=2&Instances\.MasterInstanceType=m1\.small&Instances\.Placement\.AvailabilityZone=us-east-1a&Instances\.SlaveInstanceType=m1\.small&Name=Elasticity%20Test%20Flow%20\(EMR%20Pig%20Script\)&Operation=RunJobFlow&.*&Steps\.member\.1\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.1\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps\.member\.1\.HadoopJarStep\.Args\.member\.2=--base-path&Steps\.member\.1\.HadoopJarStep\.Args\.member\.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps\.member\.1\.HadoopJarStep\.Args\.member\.4=--install-pig&Steps\.member\.1\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.1\.Name=Setup%20Pig&Steps\.member\.2\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.2\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.2=--run-pig-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.3=--args&Steps\.member\.2\.HadoopJarStep\.Args\.member\.4=-p&Steps\.member\.2\.HadoopJarStep\.Args\.member\.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps\.member\.2\.HadoopJarStep\.Args\.member\.6=-p&Steps\.member\.2\.HadoopJarStep\.Args\.member\.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-04-19&Steps\.member\.2\.HadoopJarStep\.Args\.member\.8=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports\.pig&Steps\.member\.2\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.2\.Name=Run%20Pig%20Script/
+    body:
+    headers: 
+      accept: 
+      - "*/*; q=0.5, application/xml"
+      accept-encoding: 
+      - gzip, deflate
+  response: !ruby/struct:VCR::Response 
+    status: !ruby/struct:VCR::ResponseStatus 
+      code: 200
+      message: OK
+    headers: 
+      x-amzn-requestid: 
+      - b22f4aea-6a4b-11e0-9ddc-a168e244afdb
+      content-type: 
+      - text/xml
+      date: 
+      - Tue, 19 Apr 2011 06:10:16 GMT
+      content-length: 
+      - "296"
+    body: |
+      <RunJobFlowResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+        <RunJobFlowResult>
+          <JobFlowId>j-G6N5HA528AD4</JobFlowId>
+        </RunJobFlowResult>
+        <ResponseMetadata>
+          <RequestId>b22f4aea-6a4b-11e0-9ddc-a168e244afdb</RequestId>
+        </ResponseMetadata>
+      </RunJobFlowResponse>
+
+    http_version: "1.1"

data/spec/fixtures/vcr_cassettes/set_termination_protection/nonexistent_job_flows.yml

@@ -0,0 +1,35 @@
+--- 
+- !ruby/struct:VCR::HTTPInteraction 
+  request: !ruby/struct:VCR::Request 
+    method: :get
+    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&JobFlowIds.member.1=j-1B4D1XP0C0A35&JobFlowIds.member.2=j-2&Operation=SetTerminationProtection.*&TerminationProtected=true/
+    body: 
+    headers: 
+      accept: 
+      - "*/*; q=0.5, application/xml"
+      accept-encoding: 
+      - gzip, deflate
+  response: !ruby/struct:VCR::Response 
+    status: !ruby/struct:VCR::ResponseStatus 
+      code: 400
+      message: Bad Request
+    headers: 
+      x-amzn-requestid: 
+      - dc0ce847-6923-11e0-9da4-8d278034dea6
+      content-type: 
+      - text/xml
+      date: 
+      - Sun, 17 Apr 2011 18:52:35 GMT
+      content-length: 
+      - "289"
+    body: |
+      <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+        <Error>
+          <Type>Sender</Type>
+          <Code>ValidationError</Code>
+          <Message>Specified job flow ID not valid</Message>
+        </Error>
+        <RequestId>dc0ce847-6923-11e0-9da4-8d278034dea6</RequestId>
+      </ErrorResponse>
+
+    http_version: "1.1"

data/spec/fixtures/vcr_cassettes/set_termination_protection/protect_multiple_job_flows.yml

@@ -0,0 +1,32 @@
+--- 
+- !ruby/struct:VCR::HTTPInteraction 
+  request: !ruby/struct:VCR::Request 
+    method: :get
+    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&JobFlowIds.member.1=j-1B4D1XP0C0A35&JobFlowIds.member.2=j-1YG2MYL0HVYS5&Operation=SetTerminationProtection&.*TerminationProtected=true/
+    body: 
+    headers: 
+      accept: 
+      - "*/*; q=0.5, application/xml"
+      accept-encoding: 
+      - gzip, deflate
+  response: !ruby/struct:VCR::Response 
+    status: !ruby/struct:VCR::ResponseStatus 
+      code: 200
+      message: OK
+    headers: 
+      x-amzn-requestid: 
+      - 755ebe8a-6923-11e0-a9c2-c126f1bb4493
+      content-type: 
+      - text/xml
+      date: 
+      - Sun, 17 Apr 2011 18:49:43 GMT
+      content-length: 
+      - "239"
+    body: |
+      <SetTerminationProtectionResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+        <ResponseMetadata>
+          <RequestId>755ebe8a-6923-11e0-a9c2-c126f1bb4493</RequestId>
+        </ResponseMetadata>
+      </SetTerminationProtectionResponse>
+
+    http_version: "1.1"

data/spec/lib/elasticity/emr_spec.rb

@@ -2,6 +2,9 @@ require 'spec_helper'
 
 describe Elasticity::EMR do
 
+  AWS_ACCESS_KEY_ID = ENV["AWS_ACCESS_KEY_ID"]
+  AWS_SECRET_KEY = ENV["AWS_SECRET_KEY"]
+
   describe "#add_instance_groups" do
 
     describe "integration happy path" do
@@ -9,11 +12,11 @@ describe Elasticity::EMR do
       context "when properly specified" do
         use_vcr_cassette "add_instance_groups/one_group_successful", :record => :none
         it "should add the instance groups" do
-          emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
           instance_group_config = {
-            :count => 1,
-            :role => "TASK",
-            :type => "m1.small",
+            :instance_count => 1,
+            :instance_role => "TASK",
+            :instance_type => "m1.small",
             :market => "ON_DEMAND",
             :name => "Go Canucks Go!"
           }
@@ -25,12 +28,12 @@ describe Elasticity::EMR do
       context "when improperly specified" do
         use_vcr_cassette "add_instance_groups/one_group_unsuccessful", :record => :none
         it "should add the instance groups" do
-          emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
           instance_group_config = {
             :bid_price => 0,
-            :count => 1,
-            :role => "TASK",
-            :type => "m1.small",
+            :instance_count => 1,
+            :instance_role => "TASK",
+            :instance_type => "m1.small",
             :market => "ON_DEMAND",
             :name => "Go Canucks Go!"
           }
@@ -62,8 +65,8 @@ describe Elasticity::EMR do
 
         it "should iterate over them and send the correct params to AWS" do
           instance_group_configs = [
-            {:type=>"m1.small", :role=>"CORE", :market=>"ON_DEMAND", :count=>1, :name=>"Go Canucks Go!", :bid_price=>0},
-            {:type=>"m1.small", :role=>"CORE", :market=>"ON_DEMAND", :count=>1, :name=>"Go Canucks Go!", :bid_price=>0},
+            {:instance_type=>"m1.small", :instance_role=>"CORE", :market=>"ON_DEMAND", :instance_count=>1, :name=>"Go Canucks Go!", :bid_price=>0},
+              {:instance_type=>"m1.small", :instance_role=>"CORE", :market=>"ON_DEMAND", :instance_count=>1, :name=>"Go Canucks Go!", :bid_price=>0},
           ]
           aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
           aws_request.should_receive(:aws_emr_request).with({
@@ -119,7 +122,7 @@ describe Elasticity::EMR do
     describe "integration happy path" do
       use_vcr_cassette "describe_jobflows/all_jobflows", :record => :none
       it "should return the names of all running job flows" do
-        emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
         jobflows = emr.describe_jobflows
         jobflows.map(&:name).should == ["WM+RS", "Interactive Audience Hive Test", "Audience (Hive)", "Audience Reporting"]
         jobflows.map(&:jobflow_id).should == ["j-1MZ5TVWFJRSKN", "j-38EU2XZQP9KJ4", "j-2TDCVGEEHOFI9", "j-NKKQ429D858I"]
@@ -179,13 +182,244 @@ describe Elasticity::EMR do
 
   end
 
+  describe "#modify_instance_groups" do
+
+    describe "integration happy path" do
+      context "when the instance group exists" do
+        use_vcr_cassette "modify_instance_groups/set_instances_to_3", :record => :none
+        it "should terminate the specified jobflow" do
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+          instance_group_config = {"ig-2T1HNUO61BG3O" => 2}
+          emr.modify_instance_groups(instance_group_config)
+        end
+      end
+    end
+
+    describe "unit tests" do
+
+      context "when the instance group exists" do
+        it "should modify the specified instance group" do
+          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
+          aws_request.should_receive(:aws_emr_request).with({
+            "Operation" => "ModifyInstanceGroups",
+            "InstanceGroups.member.1.InstanceGroupId" => "ig-1",
+            "InstanceGroups.member.1.InstanceCount" => 2
+          })
+          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
+          emr.modify_instance_groups({"ig-1" => 2})
+        end
+      end
+
+      context "when a block is given" do
+        it "should yield the XML result" do
+          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
+          aws_request.should_receive(:aws_emr_request).and_return("xml result!")
+          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
+          xml_result = nil
+          emr.modify_instance_groups({"ig-1" => 2}) do |xml|
+            xml_result = xml
+          end
+          xml_result.should == "xml result!"
+        end
+      end
+
+
+      context "when there is an error" do
+
+        before do
+          @error_message = "1 validation error detected: Value null at 'instanceGroups.1.member.instanceCount' failed to satisfy constraint: Member must not be null"
+          @error_xml = <<-ERROR
+            <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+              <Error>
+                <Message>#{@error_message}</Message>
+              </Error>
+            </ErrorResponse>
+          ERROR
+        end
+
+        it "should raise an ArgumentError with the error message" do
+          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
+          @exception = RestClient::BadRequest.new
+          @exception.should_receive(:http_body).and_return(@error_xml)
+          aws_request.should_receive(:aws_emr_request).and_raise(@exception)
+          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
+          lambda {
+            emr.modify_instance_groups({"ig-1" => 2})
+          }.should raise_error(ArgumentError, @error_message)
+        end
+
+      end
+
+    end
+
+  end
+
+  describe "#run_jobflow" do
+
+    describe "integration happy path" do
+
+      context "when the job flow is properly specified" do
+        use_vcr_cassette "run_jobflow/word_count", :record => :none
+        it "should start the specified job flow" do
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+          jobflow_id = emr.run_job_flow({
+            :name => "Elasticity Test Flow (EMR Pig Script)",
+            :instances => {
+              :ec2_key_name => "sharethrough-dev",
+              :hadoop_version => "0.20",
+              :instance_count => 2,
+              :master_instance_type => "m1.small",
+              :placement => {
+                :availability_zone => "us-east-1a"
+              },
+              :slave_instance_type => "m1.small",
+            },
+            :steps => [
+              {
+                :action_on_failure => "TERMINATE_JOB_FLOW",
+                :hadoop_jar_step => {
+                  :args => [
+                    "s3://elasticmapreduce/libs/pig/pig-script",
+                      "--base-path",
+                      "s3://elasticmapreduce/libs/pig/",
+                      "--install-pig"
+                  ],
+                  :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
+                },
+                :name => "Setup Pig"
+              },
+                {
+                  :action_on_failure => "TERMINATE_JOB_FLOW",
+                  :hadoop_jar_step => {
+                    :args => [
+                      "s3://elasticmapreduce/libs/pig/pig-script",
+                        "--run-pig-script",
+                        "--args",
+                        "-p",
+                        "INPUT=s3n://elasticmapreduce/samples/pig-apache/input",
+                        "-p",
+                        "OUTPUT=s3n://slif-elasticity/pig-apache/output/2011-04-19",
+                        "s3n://elasticmapreduce/samples/pig-apache/do-reports.pig"
+                    ],
+                    :jar => "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
+                  },
+                  :name => "Run Pig Script"
+                }
+            ]
+          })
+          jobflow_id.should == "j-G6N5HA528AD4"
+        end
+      end
+    end
+
+    describe "unit tests" do
+      it "should return the job flow ID of the new job" do
+        run_jobflow_response = <<-RESPONSE
+          <RunJobFlowResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+            <RunJobFlowResult>
+              <JobFlowId>j-N500G8Y8U7ZQ</JobFlowId>
+            </RunJobFlowResult>
+            <ResponseMetadata>
+              <RequestId>a6dddf4c-6a49-11e0-b6c0-e9580d1f7304</RequestId>
+            </ResponseMetadata>
+          </RunJobFlowResponse>
+        RESPONSE
+        aws_request = Elasticity::AwsRequest.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        aws_request.should_receive(:aws_emr_request).and_return(run_jobflow_response)
+        Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        jobflow_id = emr.run_job_flow({})
+        jobflow_id.should == "j-N500G8Y8U7ZQ"
+      end
+
+      it "should run the specified job flow" do
+        aws_request = Elasticity::AwsRequest.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        aws_request.should_receive(:aws_emr_request).with({
+          "Operation" => "RunJobFlow",
+          "Name" => "Job flow name",
+          "Instances.MasterInstanceType" => "m1.small",
+          "Instances.Placement.AvailabilityZone" => "us-east-1a",
+          "Steps.member.1.Name" => "Streaming Job",
+          "Steps.member.1.ActionOnFailure" => "TERMINATE_JOB_FLOW",
+          "Steps.member.1.HadoopJarStep.Jar" => "/home/hadoop/contrib/streaming/hadoop-streaming.jar",
+          "Steps.member.1.HadoopJarStep.Args.member.1" => "-input",
+          "Steps.member.1.HadoopJarStep.Args.member.2" => "s3n://elasticmapreduce/samples/wordcount/input"
+        })
+        Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        emr.run_job_flow({
+          :name => "Job flow name",
+          :instances => {
+            :master_instance_type => "m1.small",
+            :placement => {
+              :availability_zone => "us-east-1a"
+            }
+          },
+          :steps => [
+            {
+              :action_on_failure => "TERMINATE_JOB_FLOW",
+              :name => "Streaming Job",
+              :hadoop_jar_step => {
+                :args => ["-input", "s3n://elasticmapreduce/samples/wordcount/input"],
+                :jar => "/home/hadoop/contrib/streaming/hadoop-streaming.jar",
+              }
+            }
+          ]
+        })
+      end
+
+      context "when there is an error" do
+        before do
+          @error_message = "1 validation error detected: Value null at 'instanceGroups.1.member.instanceCount' failed to satisfy constraint: Member must not be null"
+          @error_xml = <<-ERROR
+            <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+              <Error>
+                <Message>#{@error_message}</Message>
+              </Error>
+            </ErrorResponse>
+          ERROR
+        end
+
+        it "should raise an ArgumentError with the error message" do
+          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
+          @exception = RestClient::BadRequest.new
+          @exception.should_receive(:http_body).and_return(@error_xml)
+          aws_request.should_receive(:aws_emr_request).and_raise(@exception)
+          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
+          lambda {
+            emr.run_job_flow({})
+          }.should raise_error(ArgumentError, @error_message)
+        end
+      end
+
+      context "when a block is given" do
+        it "should yield the XML result" do
+          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
+          aws_request.should_receive(:aws_emr_request).and_return("jobflow_id!")
+          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
+          xml_result = nil
+          emr.run_job_flow({}) do |xml|
+            xml_result = xml
+          end
+          xml_result.should == "jobflow_id!"
+        end
+      end
+    end
+
+  end
+
   describe "#terminate_jobflows" do
 
     describe "integration happy path" do
       context "when the job flow exists" do
         use_vcr_cassette "terminate_jobflows/one_jobflow", :record => :none
         it "should terminate the specified jobflow" do
-          emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
           emr.terminate_jobflows("j-1MZ5TVWFJRSKN")
         end
       end
@@ -244,77 +478,92 @@ describe Elasticity::EMR do
     end
   end
 
-  describe "#modify_instance_groups" do
+  describe "#set_termination_protection" do
 
     describe "integration happy path" do
-      context "when the instance group exists" do
-        use_vcr_cassette "modify_instance_groups/set_instances_to_3", :record => :none
-        it "should terminate the specified jobflow" do
-          emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
-          instance_group_config = {"ig-2T1HNUO61BG3O" => 2}
-          emr.modify_instance_groups(instance_group_config)
+
+      context "when protecting multiple job flows" do
+        use_vcr_cassette "set_termination_protection/protect_multiple_job_flows", :record => :none
+        it "should protect the specified job flows" do
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+          emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5"], true)
         end
       end
-    end
-
-    describe "unit tests" do
 
-      context "when the instance group exists" do
-        it "should modify the specified instance group" do
-          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
-          aws_request.should_receive(:aws_emr_request).with({
-            "Operation" => "ModifyInstanceGroups",
-            "InstanceGroups.member.1.InstanceGroupId" => "ig-1",
-            "InstanceGroups.member.1.InstanceCount" => 2
-          })
-          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
-          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
-          emr.modify_instance_groups({"ig-1" => 2})
+      context "when specifying a job flow that doesn't exist" do
+        use_vcr_cassette "set_termination_protection/nonexistent_job_flows", :record => :none
+        it "should have an error" do
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+          lambda {
+            emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-2"], true)
+          }.should raise_error(ArgumentError, "Specified job flow ID not valid")
         end
       end
 
-      context "when a block is given" do
-        it "should yield the XML result" do
-          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
-          aws_request.should_receive(:aws_emr_request).and_return("xml result!")
-          Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
-          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
-          xml_result = nil
-          emr.modify_instance_groups({"ig-1" => 2}) do |xml|
-            xml_result = xml
-          end
-          xml_result.should == "xml result!"
-        end
+    end
+
+    describe "unit tests" do
+      it "should enable protection on the specified job flows" do
+        aws_request = Elasticity::AwsRequest.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+        aws_request.should_receive(:aws_emr_request).with({
+          "Operation" => "SetTerminationProtection",
+          "JobFlowIds.member.1" => "jobflow1",
+          "JobFlowIds.member.2" => "jobflow2",
+          "TerminationProtected" => true
+        })
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        emr.set_termination_protection(["jobflow1", "jobflow2"], true)
       end
 
+      it "should disable protection on the specified job flows" do
+        aws_request = Elasticity::AwsRequest.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+        aws_request.should_receive(:aws_emr_request).with({
+          "Operation" => "SetTerminationProtection",
+          "JobFlowIds.member.1" => "jobflow1",
+          "JobFlowIds.member.2" => "jobflow2",
+          "TerminationProtected" => false
+        })
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        emr.set_termination_protection(["jobflow1", "jobflow2"], false)
+      end
 
-      context "when there is an error" do
+      it "should enable protection when not specified" do
+        aws_request = Elasticity::AwsRequest.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+        aws_request.should_receive(:aws_emr_request).with({
+          "Operation" => "SetTerminationProtection",
+          "JobFlowIds.member.1" => "jobflow1",
+          "JobFlowIds.member.2" => "jobflow2",
+          "TerminationProtected" => true
+        })
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+        emr.set_termination_protection(["jobflow1", "jobflow2"])
+      end
 
+      context "when a block is given" do
         before do
-          @error_message = "1 validation error detected: Value null at 'instanceGroups.1.member.instanceCount' failed to satisfy constraint: Member must not be null"
-          @error_xml = <<-ERROR
-            <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
-              <Error>
-                <Message>#{@error_message}</Message>
-              </Error>
-            </ErrorResponse>
-          ERROR
+          @xml_response = <<-RESPONSE
+            <SetTerminationProtectionResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+              <ResponseMetadata>
+                <RequestId>755ebe8a-6923-11e0-a9c2-c126f1bb4493</RequestId>
+              </ResponseMetadata>
+            </SetTerminationProtectionResponse>
+          RESPONSE
         end
-
-        it "should raise an ArgumentError with the error message" do
-          aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
-          @exception = RestClient::BadRequest.new
-          @exception.should_receive(:http_body).and_return(@error_xml)
-          aws_request.should_receive(:aws_emr_request).and_raise(@exception)
+        it "should yield the XML result" do
+          aws_request = Elasticity::AwsRequest.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
           Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
-          emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
-          lambda {
-            emr.modify_instance_groups({"ig-1" => 2})
-          }.should raise_error(ArgumentError, @error_message)
+          aws_request.should_receive(:aws_emr_request).and_return(@xml_response)
+          emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
+          xml = nil
+          emr.set_termination_protection([]) do |aws_response|
+            xml = aws_response
+          end
+          xml.should == @xml_response
         end
-
       end
-
     end
 
   end
@@ -324,7 +573,7 @@ describe Elasticity::EMR do
     describe "integration happy path" do
       use_vcr_cassette "direct/terminate_jobflow", :record => :none
       it "should terminate the specified jobflow" do
-        emr = Elasticity::EMR.new(ENV["aws_access_key_id"], ENV["aws_secret_key"])
+        emr = Elasticity::EMR.new(AWS_ACCESS_KEY_ID, AWS_SECRET_KEY)
         params = {
           "Operation" => "TerminateJobFlows",
           "JobFlowIds.member.1" => "j-1MZ5TVWFJRSKN"
@@ -360,4 +609,50 @@ describe Elasticity::EMR do
     end
   end
 
+  describe ".convert_ruby_to_aws" do
+    it "should convert the params" do
+      add_jobflow_steps_params = {
+        :job_flow_id => "j-1",
+        :steps => [
+          {
+            :action_on_failure => "CONTINUE",
+            :name => "First New Job Step",
+            :hadoop_jar_step => {
+              :args => ["arg1", "arg2", "arg3",],
+              :jar => "first_step.jar",
+              :main_class => "first_class.jar"
+            }
+          },
+            {
+              :action_on_failure => "CANCEL_AND_WAIT",
+              :name => "Second New Job Step",
+              :hadoop_jar_step => {
+                :args => ["arg4", "arg5", "arg6",],
+                :jar => "second_step.jar",
+                :main_class => "second_class.jar"
+              }
+            }
+        ]
+      }
+      expected_result = {
+        "JobFlowId" => "j-1",
+        "Steps.member.1.Name" => "First New Job Step",
+        "Steps.member.1.ActionOnFailure" => "CONTINUE",
+        "Steps.member.1.HadoopJarStep.Jar" => "first_step.jar",
+        "Steps.member.1.HadoopJarStep.MainClass" => "first_class.jar",
+        "Steps.member.1.HadoopJarStep.Args.member.1" => "arg1",
+        "Steps.member.1.HadoopJarStep.Args.member.2" => "arg2",
+        "Steps.member.1.HadoopJarStep.Args.member.3" => "arg3",
+        "Steps.member.2.Name" => "Second New Job Step",
+        "Steps.member.2.ActionOnFailure" => "CANCEL_AND_WAIT",
+        "Steps.member.2.HadoopJarStep.Jar" => "second_step.jar",
+        "Steps.member.2.HadoopJarStep.MainClass" => "second_class.jar",
+        "Steps.member.2.HadoopJarStep.Args.member.1" => "arg4",
+        "Steps.member.2.HadoopJarStep.Args.member.2" => "arg5",
+        "Steps.member.2.HadoopJarStep.Args.member.3" => "arg6"
+      }
+      Elasticity::EMR.send(:convert_ruby_to_aws, add_jobflow_steps_params).should == expected_result
+    end
+  end
+
 end

data/spec/spec_helper.rb

@@ -1,3 +1,25 @@
+if !ENV["AWS_ACCESS_KEY_ID"] || !ENV["AWS_SECRET_KEY"]
+  puts "\n\n\e[32m**********************************************************************************************"
+  puts "Please set \e[0;1mAWS_ACCESS_KEY_ID\e[32m and \e[0;1mAWS_SECRET_KEY\e[32m in your environment to run the tests."
+  puts ""
+  puts "These keys can be found on your AWS Account > Security Credentials page, at the following URL:"
+  puts ""
+  puts "  \e[0;1mhttps://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key"
+  puts ""
+  bourne = !ENV['SHELL'].nil? && ENV['SHELL'] =~ /(bash|ksh|zsh)/
+  puts "\e[0mbash, zsh, ksh: #{"\e[35;1mThis is you!\e[0m" if bourne}"
+  puts "  export AWS_ACCESS_KEY_ID=01234"
+  puts "  export AWS_SECRET_KEY=56789"
+  puts ""
+  puts "\e[0mcsh, tcsh: #{"\e[35;1mThis is you!\e[0m" unless bourne}"
+  puts "  setenv AWS_ACCESS_KEY_ID 01234"
+  puts "  setenv AWS_SECRET_KEY 56789"
+  puts ""
+  puts "\e[32m**********************************************************************************************\n\n\n"
+  exit
+end
+
+
 require 'rubygems'
 require 'bundler/setup'
 

metadata

@@ -1,12 +1,12 @@
 --- !ruby/object:Gem::Specification 
 name: elasticity
 version: !ruby/object:Gem::Version 
-  hash: 1
+  hash: 7
   prerelease: 
   segments: 
   - 0
-  - 5
-  version: "0.5"
+  - 6
+  version: "0.6"
 platform: ruby
 authors: 
 - Robert Slifka
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-04-16 00:00:00 -07:00
+date: 2011-04-18 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -133,6 +133,9 @@ files:
 - spec/fixtures/vcr_cassettes/describe_jobflows/all_jobflows.yml
 - spec/fixtures/vcr_cassettes/direct/terminate_jobflow.yml
 - spec/fixtures/vcr_cassettes/modify_instance_groups/set_instances_to_3.yml
+- spec/fixtures/vcr_cassettes/run_jobflow/word_count.yml
+- spec/fixtures/vcr_cassettes/set_termination_protection/nonexistent_job_flows.yml
+- spec/fixtures/vcr_cassettes/set_termination_protection/protect_multiple_job_flows.yml
 - spec/fixtures/vcr_cassettes/terminate_jobflows/one_jobflow.yml
 - spec/lib/elasticity/aws_request_spec.rb
 - spec/lib/elasticity/emr_spec.rb
@@ -178,6 +181,9 @@ test_files:
 - spec/fixtures/vcr_cassettes/describe_jobflows/all_jobflows.yml
 - spec/fixtures/vcr_cassettes/direct/terminate_jobflow.yml
 - spec/fixtures/vcr_cassettes/modify_instance_groups/set_instances_to_3.yml
+- spec/fixtures/vcr_cassettes/run_jobflow/word_count.yml
+- spec/fixtures/vcr_cassettes/set_termination_protection/nonexistent_job_flows.yml
+- spec/fixtures/vcr_cassettes/set_termination_protection/protect_multiple_job_flows.yml
 - spec/fixtures/vcr_cassettes/terminate_jobflows/one_jobflow.yml
 - spec/lib/elasticity/aws_request_spec.rb
 - spec/lib/elasticity/emr_spec.rb