elasticity 1.2.2 → 1.2.3

data/{README.mediawiki → README.md}

@@ -1,10 +1,14 @@
 Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently wrap the API operations in a manner that makes working with EMR job flows from Ruby more productive and more enjoyable, without having to understand the nuts and bolts of the EMR REST API.  At the very least, using Elasticity allows you to easily experiment with the EMR API :)
 
-'''BACKLOG''': Have a look at the [https://www.pivotaltracker.com/projects/272429 backlog] to see where this is headed.
+**BACKLOG**: Have a look at the [backlog](https://www.pivotaltracker.com/projects/272429) to see where this is headed.
 
-'''CREDITS''': AWS signing was used from [http://www.rightscale.com/ RightScale's] amazing [https://github.com/rightscale/right_aws right_aws gem] which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.  Used camelize from ActiveSupport as well, thank you \Rails :)
+**CREDITS**: AWS signing was used from [RightScale's](http://www.rightscale.com/) amazing [right_aws gem](https://github.com/rightscale/right_aws) which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.  Used camelize from ActiveSupport as well, thank you \Rails :)
 
-= Installation and Usage =
+**CONTRIBUTIONS**:
+
++ [Wouter Broekhof](https://github.com/wouter/) - HTTPS and AWS region support, additional params to describe_jobflows.
+
+# Installation and Usage
 
 <pre>
   gem install elasticity
@@ -12,7 +16,7 @@ Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.
 
 All you have to do is <code>require 'elasticity'</code> and you're all set!
 
-= Simplified API Reference =
+# Simplified API Reference
 
 Elasticity currently provides simplified access to launching Hive and Pig job flows, specifying several default values that you may optionally override:
 
@@ -28,7 +32,7 @@ Elasticity currently provides simplified access to launching Hive and Pig job fl
 
 These are all accessible from HiveJob and PigJob.  See the PigJob description for an example.
 
-=== Bootstrap Actions ===
+### Bootstrap Actions
 
 You can also configure Hadoop options with add_hadoop_bootstrap_action().
 
@@ -38,7 +42,7 @@ You can also configure Hadoop options with add_hadoop_bootstrap_action().
   ...
 </pre>
 
-== Hive ==
+## Hive
 
 HiveJob allows you to quickly launch Hive jobs without having to understand the ins and outs of the EMR API.  Specify only the Hive script location and (optionally) variables to make available to the Hive script.
 
@@ -52,7 +56,7 @@ HiveJob allows you to quickly launch Hive jobs without having to understand the
   > "j-129V5AQFMKO1C"
 </pre>
 
-== Pig ==
+## Pig
 
 Like HiveJob, PigJob allows you to quickly launch Pig jobs :)
 
@@ -68,9 +72,9 @@ Like HiveJob, PigJob allows you to quickly launch Pig jobs :)
   > "j-16PZ24OED71C6"
 </pre>
 
-=== PARALLEL ===
+### PARALLEL
 
-Given the importance of specifying a reasonable value for [http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features PARALLEL] (the number of parallel reducers), Elasticity calculates and passes through a reasonable default up with every invocation in the form of a script variable called E_PARALLELS.  This default value is based off of the formula in the Pig Cookbook and the number of reducers AWS configures per instance.
+Given the importance of specifying a reasonable value for [the number of parallel reducers](http://pig.apache.org/docs/r0.8.1/cookbook.html#Use+the+Parallel+Features PARALLEL), Elasticity calculates and passes through a reasonable default up with every invocation in the form of a script variable called E_PARALLELS.  This default value is based off of the formula in the Pig Cookbook and the number of reducers AWS configures per instance.
 
 For example, if you had 8 instances in total and your slaves were m1.xlarge, the value is 26 (as shown below).
 
@@ -92,21 +96,21 @@ Use this as you would any other Pig variable.
   ...
 </pre>
 
-= Amazon API Reference =
+# Amazon API Reference
 
 Elasticity wraps all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).
 
 You may opt for "direct" access to the API where you specify the params and Elasticity takes care of the signing for you, responding with the XML from Amazon.  Direct access is described below the API catalog.
 
-In addition to the [http://aws.amazon.com/elasticmapreduce/ AWS EMR subsite], there are three primary resources of reference information for EMR:
+In addition to the [AWS EMR subsite](http://aws.amazon.com/elasticmapreduce/), there are three primary resources of reference information for EMR:
 
-* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/ Amazon EMR Getting Started Guide]
-* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/ Amazon EMR Developer Guide]
-* [http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/ Amazon EMR API Reference]
+* [Amazon EMR Getting Started Guide](http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/)
+* [Amazon EMR Developer Guide](http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/)
+* [Amazon EMR API Reference](http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/)
 
-Unfortunately, the documentation is sometimes incorrect and sometimes missing.  E.g. the allowable values for AddInstanceGroups are present in the [http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf PDF] version of the API reference but not in the [http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/ HTML] version.  Elasticity implements the API as specified in the PDF reference as that is the most complete description I could find.
+Unfortunately, the documentation is sometimes incorrect and sometimes missing.  E.g. the allowable values for AddInstanceGroups are present in the [PDF](http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf) version of the API reference but not in the [HTML](http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/) version.  Elasticity implements the API as specified in the PDF reference as that is the most complete description I could find.
 
-== AddInstanceGroups ==
+## AddInstanceGroups
 
 AddInstanceGroups adds a group of instances to an existing job flow.  The available instance configuration options are listed in the EMR API reference.  They've been converted to be more Ruby-like in the wrappers, as shown in the example below.
 
@@ -132,7 +136,7 @@ Some combinations of the options will be rejected by Amazon and some once-valid
   > Task instance group already exists in the job flow, cannot add more task groups
 </pre>
 
-== AddJobFlowSteps ==
+## AddJobFlowSteps
 
 AddJobFlowSteps adds the specified steps to the specified job flow.
 
@@ -158,7 +162,7 @@ AddJobFlowSteps adds the specified steps to the specified job flow.
   })
 </pre>
 
-== DescribeJobFlows ==
+## DescribeJobFlows
 
 DescribeJobFlows returns detailed information as to the state of all jobs.  Currently this is wrapped in an <code>Elasticity::JobFlow</code> that contains the <code>name</code>, <code>jobflow_id</code> and <code>state</code>.  
 
@@ -170,7 +174,7 @@ DescribeJobFlows returns detailed information as to the state of all jobs.  Curr
   > ["Hive Test", "Pig Test", "Interactive Hadoop", "Interactive Hive"]
 </pre>
 
-== ModifyInstanceGroups ==
+## ModifyInstanceGroups
 
 A job flow contains several "instance groups" of various types.  These instances are where the work for your EMR task occurs.  After a job flow has been created, you can find these instance groups in the AWS web UI by clicking on a job flow and then clicking on the "Instance Groups" tab.
 
@@ -197,7 +201,7 @@ Or if you attempt to increase the instance count of the MASTER instance group:
   > ArgumentError: A master instance group may not be modified
 </pre>
 
-== RunJobFlow ==
+## RunJobFlow
 
 RunJobFlow creates and starts a new job flow.  Specifying the arguments to RunJobFlow is a bit of a hot mess at the moment, requiring you to understand the EMR syntax as well as the data structure for specifying jobs.  Here's a beefy example:
 
@@ -252,9 +256,9 @@ RunJobFlow creates and starts a new job flow.  Specifying the arguments to RunJo
   > "j-129V5AQFMKO1C"
 </pre>
 
-Currently Elasticity doesn't do much to ease this pain although this is what I would like to focus on in coming releases.  Feel free to ship ideas my way.  In the meantime, have a look at the EMR API [http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf PDF] under the RunJobFlow action and riff off of the example here.
+Currently Elasticity doesn't do much to ease this pain although this is what I would like to focus on in coming releases.  Feel free to ship ideas my way.  In the meantime, have a look at the EMR API [PDF](http://awsdocs.s3.amazonaws.com/ElasticMapReduce/20090331/emr-api-20090331.pdf) under the RunJobFlow action and riff off of the example here.
 
-== SetTerminationProtection ==
+## SetTerminationProtection
 
 Enable or disable "termination protection" on the specified job flows.  Termination protection prevents a job flow from from being terminated by any user-initiated action.  
 
@@ -269,7 +273,7 @@ To disable termination protection, specify false as the second parameter.
   emr.set_termination_protection(["j-1B4D1XP0C0A35", "j-1YG2MYL0HVYS5"], false)
 </pre>
 
-== TerminateJobFlows ==
+## TerminateJobFlows
 
 Terminate the specified job flow.  When the job flow '''exists''', you will receive no output.  This is because Amazon does not return anything other than a 200 when you terminate a job flow :)  You'll want to continuously poll with DescribeJobFlows to see when the job was actually terminated.
 
@@ -287,7 +291,7 @@ When the job flow '''doesn't exist''':
   > ArgumentError: Job flow 'no-flow' does not exist.
 </pre>
 
-= Direct Response Access =
+# Direct Response Access
 
 If you're fine with Elasticity's invocation wrapping and would prefer to get at the resulting XML rather than the wrapped response, throw a block our way and we'll yield the result.  This still saves you the trouble of having to create the params and sign the request yet gives you direct access to the response XML for your parsing pleasure.
 
@@ -295,12 +299,12 @@ If you're fine with Elasticity's invocation wrapping and would prefer to get at
   emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   emr.describe_jobflows{|xml| puts xml[0..77]}
   
-  > <DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/200
+  > &lt;DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/200...
 </pre>
 
-= Direct Request/Response Access =
+# Direct Request/Response Access
 
-If you're chomping at the bit to initiate some EMR functionality that isn't wrapped (or isn't wrapped in a way you prefer :) feel free to access the AWS EMR API directly by using <code>EMR.direct()</code>.  You can find the allowed values in Amazon's EMR API [http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html developer documentation].
+If you're chomping at the bit to initiate some EMR functionality that isn't wrapped (or isn't wrapped in a way you prefer :) feel free to access the AWS EMR API directly by using <code>EMR.direct()</code>.  You can find the allowed values in Amazon's EMR API [developer documentation](http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html).
 
 <pre>
   emr = Elasticity::EMR.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
@@ -308,10 +312,10 @@ If you're chomping at the bit to initiate some EMR functionality that isn't wrap
   result_xml = emr.direct(params)
   result_xml[0..78]
   
-  > <DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009
+  > &lt;DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009...
 </pre>
 
-= License =
+# License
 
 <pre>
   Copyright 2011 Robert Slifka
@@ -327,6 +331,4 @@ If you're chomping at the bit to initiate some EMR functionality that isn't wrap
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-</pre>
-
-<!-- __NOTOC__ -->
+</pre>

data/elasticity.gemspec

@@ -11,8 +11,6 @@ Gem::Specification.new do |s|
   s.summary     = %q{Programmatic access to Amazon's Elastic Map Reduce service.}
   s.description = %q{Programmatic access to Amazon's Elastic Map Reduce service.}
 
-  s.rubyforge_project = "elasticity"
-
   s.add_dependency("rest-client")
   s.add_dependency("nokogiri")
 

data/lib/elasticity/aws_request.rb

@@ -2,14 +2,21 @@ module Elasticity
 
   class AwsRequest
 
-    def initialize(aws_access_key_id, aws_secret_access_key)
+    # Supported values for options:
+    #  :region - AWS region (e.g. us-west-1)
+    #  :secure - true or false, default true.
+    def initialize(aws_access_key_id, aws_secret_access_key, options = {})
       @access_key = aws_access_key_id
       @secret_key = aws_secret_access_key
+      @options = {:secure => true}.merge(options)
     end
 
     def aws_emr_request(params)
-      signed_params = sign_params(params, "GET", "elasticmapreduce.amazonaws.com", "/")
-      signed_request = "http://elasticmapreduce.amazonaws.com?#{signed_params}"
+      host = @options[:region] ? "elasticmapreduce.#{@options[:region]}.amazonaws.com" : "elasticmapreduce.amazonaws.com"
+      protocol = @options[:secure] ? "https" : "http"
+
+      signed_params = sign_params(params, "GET", host, "/")
+      signed_request = "#{protocol}://#{host}?#{signed_params}"
       RestClient.get signed_request
     end
 
@@ -45,4 +52,4 @@ module Elasticity
 
   end
 
-end
+end

data/lib/elasticity/emr.rb

@@ -2,13 +2,20 @@ module Elasticity
 
   class EMR
 
-    def initialize(aws_access_key_id, aws_secret_access_key)
-      @aws_request = Elasticity::AwsRequest.new(aws_access_key_id, aws_secret_access_key)
+    def initialize(aws_access_key_id, aws_secret_access_key, options = {})
+      @aws_request = Elasticity::AwsRequest.new(aws_access_key_id, aws_secret_access_key, options)
     end
 
     # Lists all jobflows in all states.
-    def describe_jobflows
-      aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(:operation => "DescribeJobFlows"))
+    #
+    # To override this behaviour, pass additional filters as specified in the AWS
+    # documentation - http://docs.amazonwebservices.com/ElasticMapReduce/latest/API/index.html?API_DescribeJobFlows.html.
+    #
+    #   describe_jobflows(:CreatedBefore => "2011-10-04")
+    def describe_jobflows(params = {})
+      aws_result = @aws_request.aws_emr_request(EMR.convert_ruby_to_aws(
+        params.merge({:operation => "DescribeJobFlows"}))
+      )
       xml_doc = Nokogiri::XML(aws_result)
       xml_doc.remove_namespaces!
       yield aws_result if block_given?
@@ -31,7 +38,7 @@ module Elasticity
     #
     # add_instance_groups takes an array of {}.  Returns an array of the instance IDs
     # that were created by the specified configs.
-    # 
+    #
     #   ["ig-2GOVEN6HVJZID", "ig-1DU9M2UQMM051", "ig-3DZRW4Y2X4S", ...]
     def add_instance_groups(jobflow_id, instance_group_configs)
       params = {

data/lib/elasticity/job_flow.rb

@@ -6,34 +6,44 @@ module Elasticity
     attr_accessor :jobflow_id
     attr_accessor :state
     attr_accessor :steps
+    attr_accessor :created_at
+    attr_accessor :started_at
+    attr_accessor :ready_at
+    attr_accessor :instance_count
+    attr_accessor :master_instance_type
+    attr_accessor :slave_instance_type
 
     def initialize
       @steps = []
     end
 
-    class << self
-
-      # Create a jobflow from an AWS <member> (Nokogiri::XML::Element):
-      #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows/member
-      def from_member_element(xml_element)
-        jobflow = JobFlow.new
-        jobflow.name = xml_element.xpath("./Name").text
-        jobflow.jobflow_id = xml_element.xpath("./JobFlowId").text
-        jobflow.state = xml_element.xpath("./ExecutionStatusDetail/State").text
-        jobflow.steps = JobFlowStep.from_members_nodeset(xml_element.xpath("./Steps/member"))
-        jobflow
-      end
+    # Create a jobflow from an AWS <member> (Nokogiri::XML::Element):
+    #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows/member
+    def self.from_member_element(xml_element)
+      jobflow = JobFlow.new
+      jobflow.name = xml_element.xpath("./Name").text.strip
+      jobflow.jobflow_id = xml_element.xpath("./JobFlowId").text.strip
+      jobflow.state = xml_element.xpath("./ExecutionStatusDetail/State").text.strip
+      jobflow.steps = JobFlowStep.from_members_nodeset(xml_element.xpath("./Steps/member"))
+      jobflow.created_at = Time.parse(xml_element.xpath("./ExecutionStatusDetail/CreationDateTime").text.strip)
+      started_at = xml_element.xpath("./ExecutionStatusDetail/StartDateTime").text.strip
+      jobflow.started_at = (started_at == "") ? (nil) : (Time.parse(started_at))
+      ready_at = xml_element.xpath("./ExecutionStatusDetail/ReadyDateTime").text.strip
+      jobflow.ready_at = (ready_at == "") ? (nil) : (Time.parse(ready_at))
+      jobflow.instance_count = xml_element.xpath("./Instances/InstanceCount").text.strip
+      jobflow.master_instance_type = xml_element.xpath("./Instances/MasterInstanceType").text.strip
+      jobflow.slave_instance_type = xml_element.xpath("./Instances/SlaveInstanceType").text.strip
+      jobflow
+    end
 
-      # Create JobFlows from a collection of AWS <member> nodes (Nokogiri::XML::NodeSet):
-      #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows
-      def from_members_nodeset(members_nodeset)
-        jobflows = []
-        members_nodeset.each do |member|
-          jobflows << from_member_element(member)
-        end
-        jobflows
+    # Create JobFlows from a collection of AWS <member> nodes (Nokogiri::XML::NodeSet):
+    #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows
+    def self.from_members_nodeset(members_nodeset)
+      jobflows = []
+      members_nodeset.each do |member|
+        jobflows << from_member_element(member)
       end
-
+      jobflows
     end
 
   end

data/lib/elasticity/job_flow_step.rb

@@ -4,29 +4,32 @@ module Elasticity
 
     attr_accessor :name
     attr_accessor :state
+    attr_accessor :started_at
+    attr_accessor :ended_at
 
-    class << self
-
-      # Create a job flow from an AWS <member> (Nokogiri::XML::Element):
-      #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows/member/Steps/member
-      def from_member_element(xml_element)
-        job_flow_step = JobFlowStep.new
-        job_flow_step.name = xml_element.xpath("./StepConfig/Name").text
-        job_flow_step.state = xml_element.xpath("./ExecutionStatusDetail/State").text
-        job_flow_step
-      end
+    # Create a job flow from an AWS <member> (Nokogiri::XML::Element):
+    #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows/member/Steps/member
+    def self.from_member_element(xml_element)
+      job_flow_step = JobFlowStep.new
+      job_flow_step.name = xml_element.xpath("./StepConfig/Name").text.strip
+      job_flow_step.state = xml_element.xpath("./ExecutionStatusDetail/State").text.strip
+      started_at = xml_element.xpath("./ExecutionStatusDetail/StartDateTime").text.strip
+      job_flow_step.started_at = (started_at == "") ? (nil) : (Time.parse(started_at))
+      ended_at = xml_element.xpath("./ExecutionStatusDetail/EndDateTime").text.strip
+      job_flow_step.ended_at = (ended_at == "") ? (nil) : (Time.parse(ended_at))
+      job_flow_step
+    end
 
-      # Create JobFlowSteps from a collection of AWS <member> nodes (Nokogiri::XML::NodeSet):
-      #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows/member/Steps/member
-      def from_members_nodeset(members_nodeset)
-        jobflow_steps = []
-        members_nodeset.each do |member|
-          jobflow_steps << from_member_element(member)
-        end
-        jobflow_steps
+    # Create JobFlowSteps from a collection of AWS <member> nodes (Nokogiri::XML::NodeSet):
+    #   /DescribeJobFlowsResponse/DescribeJobFlowsResult/JobFlows/member/Steps/member
+    def self.from_members_nodeset(members_nodeset)
+      jobflow_steps = []
+      members_nodeset.each do |member|
+        jobflow_steps << from_member_element(member)
       end
+      jobflow_steps
     end
 
   end
 
-end
+end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = "1.2.2"
+  VERSION = "1.2.3"
 end

data/spec/fixtures/vcr_cassettes/add_instance_groups/one_group_successful.yml

@@ -2,8 +2,8 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&InstanceGroups.member.1.InstanceCount=1&InstanceGroups.member.1.InstanceRole=TASK&InstanceGroups.member.1.InstanceType=m1.small&InstanceGroups.member.1.Market=ON_DEMAND&InstanceGroups.member.1.Name=Go%20Canucks%20Go!&JobFlowId=j-OALI7TZTQMHX&Operation=AddInstanceGroups/
-    body: 
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&InstanceGroups.member.1.InstanceCount=1&InstanceGroups.member.1.InstanceRole=TASK&InstanceGroups.member.1.InstanceType=m1.small&InstanceGroups.member.1.Market=ON_DEMAND&InstanceGroups.member.1.Name=Go%20Canucks%20Go!&JobFlowId=j-OALI7TZTQMHX&Operation=AddInstanceGroups/
+    body:
     headers: 
       accept: 
       - "*/*; q=0.5, application/xml"

data/spec/fixtures/vcr_cassettes/add_instance_groups/one_group_unsuccessful.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&InstanceGroups.member.1.BidPrice=0&InstanceGroups.member.1.InstanceCount=1&InstanceGroups.member.1.InstanceRole=TASK&InstanceGroups.member.1.InstanceType=m1.small&InstanceGroups.member.1.Market=ON_DEMAND&InstanceGroups.member.1.Name=Go%20Canucks%20Go!&JobFlowId=j-19WDDS68ZUENP&Operation=AddInstanceGroups/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&InstanceGroups.member.1.BidPrice=0&InstanceGroups.member.1.InstanceCount=1&InstanceGroups.member.1.InstanceRole=TASK&InstanceGroups.member.1.InstanceType=m1.small&InstanceGroups.member.1.Market=ON_DEMAND&InstanceGroups.member.1.Name=Go%20Canucks%20Go!&JobFlowId=j-19WDDS68ZUENP&Operation=AddInstanceGroups/
     body:
     headers: 
       accept: 

data/spec/fixtures/vcr_cassettes/add_jobflow_steps/add_multiple_steps.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?.*Operation=RunJobFlow/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?.*Operation=RunJobFlow/
     body:
     headers: 
       accept: 
@@ -36,7 +36,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?.*Operation=AddJobFlowSteps/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?.*Operation=AddJobFlowSteps/
     body:
     headers: 
       accept: 
@@ -67,7 +67,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?.*Operation=DescribeJobFlows/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?.*Operation=DescribeJobFlows/
     body:
     headers: 
       accept: 

data/spec/fixtures/vcr_cassettes/describe_jobflows/all_jobflows.yml

@@ -2,8 +2,8 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&Operation=DescribeJobFlows/
-    body: 
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&Operation=DescribeJobFlows/
+    body:
     headers: 
       accept: 
       - "*/*; q=0.5, application/xml"

data/spec/fixtures/vcr_cassettes/direct/terminate_jobflow.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&JobFlowIds.member.1=j-1MZ5TVWFJRSKN&Operation=TerminateJobFlows/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&JobFlowIds.member.1=j-1MZ5TVWFJRSKN&Operation=TerminateJobFlows/
     body:
     headers: 
       accept: 

data/spec/fixtures/vcr_cassettes/hive_job/hive_ads.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction
   request: !ruby/struct:VCR::Request
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&Instances\.Ec2KeyName=sharethrough_dev&Instances\.HadoopVersion=0\.20&Instances\.InstanceCount=2&Instances\.MasterInstanceType=m1\.small&Instances\.SlaveInstanceType=m1\.small&Name=Elasticity%20Hive%20Job&Operation=RunJobFlow&.*&Steps\.member\.1\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.1\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/hive\/hive-script&Steps\.member\.1\.HadoopJarStep\.Args\.member\.2=--base-path&Steps\.member\.1\.HadoopJarStep\.Args\.member\.3=s3:\/\/elasticmapreduce\/libs\/hive\/&Steps\.member\.1\.HadoopJarStep\.Args\.member\.4=--install-hive&Steps\.member\.1\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.1\.Name=Setup%20Hive&Steps\.member\.2\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.2\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/hive\/hive-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.10=-d&Steps\.member\.2\.HadoopJarStep\.Args\.member\.11=LIBS=s3n:\/\/elasticmapreduce\/samples\/hive-ads\/libs&Steps\.member\.2\.HadoopJarStep\.Args\.member\.2=--run-hive-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.3=--args&Steps\.member\.2\.HadoopJarStep\.Args\.member\.4=-f&Steps\.member\.2\.HadoopJarStep\.Args\.member\.5=s3n:\/\/elasticmapreduce\/samples\/hive-ads\/libs\/model-build\.q&Steps\.member\.2\.HadoopJarStep\.Args\.member\.6=-d&Steps\.member\.2\.HadoopJarStep\.Args\.member\.7=INPUT=s3n:\/\/elasticmapreduce\/samples\/hive-ads\/tables&Steps\.member\.2\.HadoopJarStep\.Args\.member\.8=-d&Steps\.member\.2\.HadoopJarStep\.Args\.member\.9=OUTPUT=s3n:\/\/slif-elasticity\/hive-ads\/output\/2011-04-19&Steps\.member\.2\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.2\.Name=Run%20Hive%20Script/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&Instances\.Ec2KeyName=sharethrough_dev&Instances\.HadoopVersion=0\.20&Instances\.InstanceCount=2&Instances\.MasterInstanceType=m1\.small&Instances\.SlaveInstanceType=m1\.small&Name=Elasticity%20Hive%20Job&Operation=RunJobFlow&.*&Steps\.member\.1\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.1\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/hive\/hive-script&Steps\.member\.1\.HadoopJarStep\.Args\.member\.2=--base-path&Steps\.member\.1\.HadoopJarStep\.Args\.member\.3=s3:\/\/elasticmapreduce\/libs\/hive\/&Steps\.member\.1\.HadoopJarStep\.Args\.member\.4=--install-hive&Steps\.member\.1\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.1\.Name=Setup%20Hive&Steps\.member\.2\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.2\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/hive\/hive-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.10=-d&Steps\.member\.2\.HadoopJarStep\.Args\.member\.11=LIBS=s3n:\/\/elasticmapreduce\/samples\/hive-ads\/libs&Steps\.member\.2\.HadoopJarStep\.Args\.member\.2=--run-hive-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.3=--args&Steps\.member\.2\.HadoopJarStep\.Args\.member\.4=-f&Steps\.member\.2\.HadoopJarStep\.Args\.member\.5=s3n:\/\/elasticmapreduce\/samples\/hive-ads\/libs\/model-build\.q&Steps\.member\.2\.HadoopJarStep\.Args\.member\.6=-d&Steps\.member\.2\.HadoopJarStep\.Args\.member\.7=INPUT=s3n:\/\/elasticmapreduce\/samples\/hive-ads\/tables&Steps\.member\.2\.HadoopJarStep\.Args\.member\.8=-d&Steps\.member\.2\.HadoopJarStep\.Args\.member\.9=OUTPUT=s3n:\/\/slif-elasticity\/hive-ads\/output\/2011-04-19&Steps\.member\.2\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.2\.Name=Run%20Hive%20Script/
     body:
     headers:
       accept:

data/spec/fixtures/vcr_cassettes/modify_instance_groups/set_instances_to_3.yml

@@ -2,8 +2,8 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&InstanceGroups.member.1.InstanceCount=2&InstanceGroups.member.1.InstanceGroupId=ig-2T1HNUO61BG3O&Operation=ModifyInstanceGroups/
-    body: 
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&InstanceGroups.member.1.InstanceCount=2&InstanceGroups.member.1.InstanceGroupId=ig-2T1HNUO61BG3O&Operation=ModifyInstanceGroups/
+    body:
     headers: 
       accept: 
       - "*/*; q=0.5, application/xml"

data/spec/fixtures/vcr_cassettes/pig_job/apache_log_reports.yml

@@ -2,8 +2,8 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&Instances.Ec2KeyName=sharethrough_dev&Instances.HadoopVersion=0.20&Instances.InstanceCount=2&Instances.MasterInstanceType=m1.small&Instances.SlaveInstanceType=m1.small&LogUri=s3n:\/\/slif-elasticity\/pig-apache\/logs&Name=Elasticity%20Pig%20Job&Operation=RunJobFlow&.*&Steps.member.1.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.1.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.1.HadoopJarStep.Args.member.2=--base-path&Steps.member.1.HadoopJarStep.Args.member.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps.member.1.HadoopJarStep.Args.member.4=--install-pig&Steps.member.1.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.1.Name=Setup%20Pig&Steps.member.2.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.2.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.2.HadoopJarStep.Args.member.10=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports.pig&Steps.member.2.HadoopJarStep.Args.member.2=--run-pig-script&Steps.member.2.HadoopJarStep.Args.member.3=--args&Steps.member.2.HadoopJarStep.Args.member.4=-p&Steps.member.2.HadoopJarStep.Args.member.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps.member.2.HadoopJarStep.Args.member.6=-p&Steps.member.2.HadoopJarStep.Args.member.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-05-04&Steps.member.2.HadoopJarStep.Args.member.8=-p&Steps.member.2.HadoopJarStep.Args.member.9=E_PARALLELS=1&Steps.member.2.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.2.Name=Run%20Pig%20Script/
-    body: 
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&Instances.Ec2KeyName=sharethrough_dev&Instances.HadoopVersion=0.20&Instances.InstanceCount=2&Instances.MasterInstanceType=m1.small&Instances.SlaveInstanceType=m1.small&LogUri=s3n:\/\/slif-elasticity\/pig-apache\/logs&Name=Elasticity%20Pig%20Job&Operation=RunJobFlow&.*&Steps.member.1.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.1.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.1.HadoopJarStep.Args.member.2=--base-path&Steps.member.1.HadoopJarStep.Args.member.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps.member.1.HadoopJarStep.Args.member.4=--install-pig&Steps.member.1.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.1.Name=Setup%20Pig&Steps.member.2.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.2.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.2.HadoopJarStep.Args.member.10=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports.pig&Steps.member.2.HadoopJarStep.Args.member.2=--run-pig-script&Steps.member.2.HadoopJarStep.Args.member.3=--args&Steps.member.2.HadoopJarStep.Args.member.4=-p&Steps.member.2.HadoopJarStep.Args.member.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps.member.2.HadoopJarStep.Args.member.6=-p&Steps.member.2.HadoopJarStep.Args.member.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-05-04&Steps.member.2.HadoopJarStep.Args.member.8=-p&Steps.member.2.HadoopJarStep.Args.member.9=E_PARALLELS=1&Steps.member.2.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.2.Name=Run%20Pig%20Script/
+    body:
     headers: 
       accept: 
       - "*/*; q=0.5, application/xml"

data/spec/fixtures/vcr_cassettes/pig_job/apache_log_reports_with_bootstrap.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&BootstrapActions.member.1.Name=Elasticity%20Bootstrap%20Action%20\(Configure%20Hadoop\)&BootstrapActions.member.1.ScriptBootstrapAction.Args.member.1=-m&BootstrapActions.member.1.ScriptBootstrapAction.Args.member.2=mapred.job.reuse.jvm.num.tasks=120&BootstrapActions.member.1.ScriptBootstrapAction.Path=s3n:\/\/elasticmapreduce\/bootstrap-actions\/configure-hadoop&Instances.Ec2KeyName=sharethrough_dev&Instances.HadoopVersion=0.20&Instances.InstanceCount=2&Instances.MasterInstanceType=m1.small&Instances.SlaveInstanceType=m1.small&Name=Elasticity%20Pig%20Job&Operation=RunJobFlow&.*&Steps.member.1.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.1.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.1.HadoopJarStep.Args.member.2=--base-path&Steps.member.1.HadoopJarStep.Args.member.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps.member.1.HadoopJarStep.Args.member.4=--install-pig&Steps.member.1.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.1.Name=Setup%20Pig&Steps.member.2.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.2.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.2.HadoopJarStep.Args.member.10=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports.pig&Steps.member.2.HadoopJarStep.Args.member.2=--run-pig-script&Steps.member.2.HadoopJarStep.Args.member.3=--args&Steps.member.2.HadoopJarStep.Args.member.4=-p&Steps.member.2.HadoopJarStep.Args.member.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps.member.2.HadoopJarStep.Args.member.6=-p&Steps.member.2.HadoopJarStep.Args.member.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-05-10&Steps.member.2.HadoopJarStep.Args.member.8=-p&Steps.member.2.HadoopJarStep.Args.member.9=E_PARALLELS=1&Steps.member.2.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.2.Name=Run%20Pig%20Script/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&BootstrapActions.member.1.Name=Elasticity%20Bootstrap%20Action%20\(Configure%20Hadoop\)&BootstrapActions.member.1.ScriptBootstrapAction.Args.member.1=-m&BootstrapActions.member.1.ScriptBootstrapAction.Args.member.2=mapred.job.reuse.jvm.num.tasks=120&BootstrapActions.member.1.ScriptBootstrapAction.Path=s3n:\/\/elasticmapreduce\/bootstrap-actions\/configure-hadoop&Instances.Ec2KeyName=sharethrough_dev&Instances.HadoopVersion=0.20&Instances.InstanceCount=2&Instances.MasterInstanceType=m1.small&Instances.SlaveInstanceType=m1.small&Name=Elasticity%20Pig%20Job&Operation=RunJobFlow&.*&Steps.member.1.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.1.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.1.HadoopJarStep.Args.member.2=--base-path&Steps.member.1.HadoopJarStep.Args.member.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps.member.1.HadoopJarStep.Args.member.4=--install-pig&Steps.member.1.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.1.Name=Setup%20Pig&Steps.member.2.ActionOnFailure=TERMINATE_JOB_FLOW&Steps.member.2.HadoopJarStep.Args.member.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps.member.2.HadoopJarStep.Args.member.10=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports.pig&Steps.member.2.HadoopJarStep.Args.member.2=--run-pig-script&Steps.member.2.HadoopJarStep.Args.member.3=--args&Steps.member.2.HadoopJarStep.Args.member.4=-p&Steps.member.2.HadoopJarStep.Args.member.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps.member.2.HadoopJarStep.Args.member.6=-p&Steps.member.2.HadoopJarStep.Args.member.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-05-10&Steps.member.2.HadoopJarStep.Args.member.8=-p&Steps.member.2.HadoopJarStep.Args.member.9=E_PARALLELS=1&Steps.member.2.HadoopJarStep.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner.jar&Steps.member.2.Name=Run%20Pig%20Script/
     body:
     headers: 
       accept: 

data/spec/fixtures/vcr_cassettes/run_jobflow/word_count.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&Instances\.Ec2KeyName=sharethrough-dev&Instances\.HadoopVersion=0\.20&Instances\.InstanceCount=2&Instances\.MasterInstanceType=m1\.small&Instances\.Placement\.AvailabilityZone=us-east-1a&Instances\.SlaveInstanceType=m1\.small&Name=Elasticity%20Test%20Flow%20\(EMR%20Pig%20Script\)&Operation=RunJobFlow&.*&Steps\.member\.1\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.1\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps\.member\.1\.HadoopJarStep\.Args\.member\.2=--base-path&Steps\.member\.1\.HadoopJarStep\.Args\.member\.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps\.member\.1\.HadoopJarStep\.Args\.member\.4=--install-pig&Steps\.member\.1\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.1\.Name=Setup%20Pig&Steps\.member\.2\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.2\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.2=--run-pig-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.3=--args&Steps\.member\.2\.HadoopJarStep\.Args\.member\.4=-p&Steps\.member\.2\.HadoopJarStep\.Args\.member\.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps\.member\.2\.HadoopJarStep\.Args\.member\.6=-p&Steps\.member\.2\.HadoopJarStep\.Args\.member\.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-04-19&Steps\.member\.2\.HadoopJarStep\.Args\.member\.8=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports\.pig&Steps\.member\.2\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.2\.Name=Run%20Pig%20Script/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&Instances\.Ec2KeyName=sharethrough-dev&Instances\.HadoopVersion=0\.20&Instances\.InstanceCount=2&Instances\.MasterInstanceType=m1\.small&Instances\.Placement\.AvailabilityZone=us-east-1a&Instances\.SlaveInstanceType=m1\.small&Name=Elasticity%20Test%20Flow%20\(EMR%20Pig%20Script\)&Operation=RunJobFlow&.*&Steps\.member\.1\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.1\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps\.member\.1\.HadoopJarStep\.Args\.member\.2=--base-path&Steps\.member\.1\.HadoopJarStep\.Args\.member\.3=s3:\/\/elasticmapreduce\/libs\/pig\/&Steps\.member\.1\.HadoopJarStep\.Args\.member\.4=--install-pig&Steps\.member\.1\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.1\.Name=Setup%20Pig&Steps\.member\.2\.ActionOnFailure=TERMINATE_JOB_FLOW&Steps\.member\.2\.HadoopJarStep\.Args\.member\.1=s3:\/\/elasticmapreduce\/libs\/pig\/pig-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.2=--run-pig-script&Steps\.member\.2\.HadoopJarStep\.Args\.member\.3=--args&Steps\.member\.2\.HadoopJarStep\.Args\.member\.4=-p&Steps\.member\.2\.HadoopJarStep\.Args\.member\.5=INPUT=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/input&Steps\.member\.2\.HadoopJarStep\.Args\.member\.6=-p&Steps\.member\.2\.HadoopJarStep\.Args\.member\.7=OUTPUT=s3n:\/\/slif-elasticity\/pig-apache\/output\/2011-04-19&Steps\.member\.2\.HadoopJarStep\.Args\.member\.8=s3n:\/\/elasticmapreduce\/samples\/pig-apache\/do-reports\.pig&Steps\.member\.2\.HadoopJarStep\.Jar=s3:\/\/elasticmapreduce\/libs\/script-runner\/script-runner\.jar&Steps\.member\.2\.Name=Run%20Pig%20Script/
     body:
     headers: 
       accept: 

data/spec/fixtures/vcr_cassettes/set_termination_protection/nonexistent_job_flows.yml

@@ -2,8 +2,8 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&JobFlowIds.member.1=j-1B4D1XP0C0A35&JobFlowIds.member.2=j-2&Operation=SetTerminationProtection.*&TerminationProtected=true/
-    body: 
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&JobFlowIds.member.1=j-1B4D1XP0C0A35&JobFlowIds.member.2=j-2&Operation=SetTerminationProtection.*&TerminationProtected=true/
+    body:
     headers: 
       accept: 
       - "*/*; q=0.5, application/xml"

data/spec/fixtures/vcr_cassettes/set_termination_protection/protect_multiple_job_flows.yml

@@ -2,8 +2,8 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&JobFlowIds.member.1=j-1B4D1XP0C0A35&JobFlowIds.member.2=j-1YG2MYL0HVYS5&Operation=SetTerminationProtection&.*TerminationProtected=true/
-    body: 
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&JobFlowIds.member.1=j-1B4D1XP0C0A35&JobFlowIds.member.2=j-1YG2MYL0HVYS5&Operation=SetTerminationProtection&.*TerminationProtected=true/
+    body:
     headers: 
       accept: 
       - "*/*; q=0.5, application/xml"

data/spec/fixtures/vcr_cassettes/terminate_jobflows/one_jobflow.yml

@@ -2,7 +2,7 @@
 - !ruby/struct:VCR::HTTPInteraction 
   request: !ruby/struct:VCR::Request 
     method: :get
-    uri: !ruby/regexp /^http:\/\/elasticmapreduce.amazonaws.com:80\/\?AWSAccessKeyId=AKIAI7HEMMNKGT6VFFSA&JobFlowIds.member.1=j-1MZ5TVWFJRSKN&Operation=TerminateJobFlows/
+    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&JobFlowIds.member.1=j-1MZ5TVWFJRSKN&Operation=TerminateJobFlows/
     body:
     headers: 
       accept: 

data/spec/lib/elasticity/aws_request_spec.rb

@@ -15,7 +15,7 @@ describe Elasticity::AwsRequest do
 
   describe "#sign_params" do
     before do
-      Time.should_receive(:now).and_return(Time.at(1302461096))
+      Time.stub(:now).and_return(Time.at(1302461096))
     end
     it "should sign according to Amazon's rules" do
       request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_access_key")
@@ -24,4 +24,50 @@ describe Elasticity::AwsRequest do
     end
   end
 
-end
+  describe "#aws_emr_request" do
+    before do
+      Time.stub(:now).and_return(Time.at(1302461096))
+    end
+
+    describe "options" do
+
+      describe "region" do
+        context "when :region is specified" do
+          it "should request against that region" do
+            request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_access_key",
+              :region => "eu-west-1")
+            RestClient.should_receive(:get).with(/elasticmapreduce\.eu\-west\-1\.amazonaws\.com/)
+            request.aws_emr_request({})
+          end
+        end
+        context "when :region is not specified" do
+          it "should use the default request url" do
+            request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_access_key")
+            RestClient.should_receive(:get).with(/elasticmapreduce\.amazonaws\.com/)
+            request.aws_emr_request({})
+          end
+        end
+      end
+
+      describe ":secure" do
+        context "when :secure is specified" do
+          it "should use the value to determine the request type" do
+            request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_access_key",
+              :secure => false)
+            RestClient.should_receive(:get).with(/^http:/)
+            request.aws_emr_request({})
+          end
+        end
+        context "when :secure is not specified" do
+          it "should default to secure connection" do
+            request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_access_key")
+            RestClient.should_receive(:get).with(/^https:/)
+            request.aws_emr_request({})
+          end
+        end
+      end
+
+    end
+  end
+
+end

data/spec/lib/elasticity/emr_spec.rb

@@ -293,6 +293,14 @@ describe Elasticity::EMR do
         jobflows.map(&:name).should == ["Pig Job", "Hive Job"]
       end
 
+      it "should accept additional parameters" do
+        aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
+        aws_request.should_receive(:aws_emr_request).with({"Operation" => "DescribeJobFlows","CreatedBefore" => "2011-10-04"}).and_return(@describe_jobflows_xml)
+        Elasticity::AwsRequest.should_receive(:new).and_return(aws_request)
+        emr = Elasticity::EMR.new("aws_access_key_id", "aws_secret_key")
+        emr.describe_jobflows(:CreatedBefore => "2011-10-04")
+      end
+
       context "when a block is provided" do
         it "should yield the XML result" do
           aws_request = Elasticity::AwsRequest.new("aws_access_key_id", "aws_secret_key")
@@ -783,4 +791,4 @@ describe Elasticity::EMR do
     end
   end
 
-end
+end

data/spec/lib/elasticity/job_flow_spec.rb

@@ -11,6 +11,15 @@ describe Elasticity::JobFlow do
               <JobFlowId>j-p</JobFlowId>
               <Name>Pig Job</Name>
               <ExecutionStatusDetail>
+                <CreationDateTime>
+                   2011-10-04T21:49:16Z
+                </CreationDateTime>
+                <StartDateTime>
+                   2011-10-04T21:49:17Z
+                </StartDateTime>
+                <ReadyDateTime>
+                   2011-10-04T21:49:18Z
+                </ReadyDateTime>
                 <State>TERMINATED</State>
               </ExecutionStatusDetail>
               <Steps>
@@ -31,13 +40,60 @@ describe Elasticity::JobFlow do
                   </ExecutionStatusDetail>
                 </member>
               </Steps>
+              <Instances>
+                 <Placement>
+                    <AvailabilityZone>
+                      eu-west-1a
+                    </AvailabilityZone>
+                 </Placement>
+                 <SlaveInstanceType>
+                    m1.small
+                 </SlaveInstanceType>
+                 <MasterInstanceType>
+                    m1.small
+                 </MasterInstanceType>
+                 <Ec2KeyName>
+                    myec2keyname
+                 </Ec2KeyName>
+                 <InstanceCount>
+                    4
+                 </InstanceCount>
+              </Instances>
             </member>
             <member>
               <JobFlowId>j-h</JobFlowId>
               <Name>Hive Job</Name>
               <ExecutionStatusDetail>
+                <CreationDateTime>
+                   2011-10-04T22:49:16Z
+                </CreationDateTime>
+                <StartDateTime>
+
+                </StartDateTime>
+                <ReadyDateTime>
+                   
+                </ReadyDateTime>
                 <State>TERMINATED</State>
               </ExecutionStatusDetail>
+              <Instances>
+                 <Placement>
+                    <AvailabilityZone>
+                      eu-west-1b
+                    </AvailabilityZone>
+                 </Placement>
+                 <SlaveInstanceType>
+                    c1.medium
+                 </SlaveInstanceType>
+                 <MasterInstanceType>
+                    c1.medium
+                 </MasterInstanceType>
+                 <Ec2KeyName>
+                    myec2keyname
+                 </Ec2KeyName>
+                 <InstanceCount>
+                    2
+                 </InstanceCount>
+              </Instances>
             </member>
           </JobFlows>
         </DescribeJobFlowsResult>
@@ -56,6 +112,12 @@ describe Elasticity::JobFlow do
       jobflow.state.should == "TERMINATED"
       jobflow.steps.map(&:name).should == ["Setup Hive", "Run Hive Script"]
       jobflow.steps.map(&:state).should == ["FAILED", "PENDING"]
+      jobflow.created_at.should == Time.parse("2011-10-04T21:49:16Z")
+      jobflow.started_at.should == Time.parse("2011-10-04T21:49:17Z")
+      jobflow.ready_at.should == Time.parse("2011-10-04T21:49:18Z")
+      jobflow.master_instance_type.should == "m1.small"
+      jobflow.slave_instance_type.should == "m1.small"
+      jobflow.instance_count.should == "4"
     end
   end
 
@@ -65,7 +127,13 @@ describe Elasticity::JobFlow do
       jobflow.map(&:name).should == ["Pig Job", "Hive Job"]
       jobflow.map(&:jobflow_id).should == ["j-p", "j-h"]
       jobflow.map(&:state).should == ["TERMINATED", "TERMINATED"]
+      jobflow.map(&:created_at).should == [Time.parse("2011-10-04T21:49:16Z"), Time.parse("2011-10-04T22:49:16Z")]
+      jobflow.map(&:started_at).should == [Time.parse("2011-10-04T21:49:17Z"), nil]
+      jobflow.map(&:ready_at).should == [Time.parse("2011-10-04T21:49:18Z"), nil]
+      jobflow.map(&:master_instance_type).should == ["m1.small","c1.medium"]
+      jobflow.map(&:slave_instance_type).should == ["m1.small", "c1.medium"]
+      jobflow.map(&:instance_count).should == ["4","2"]
     end
   end
 
-end
+end

data/spec/lib/elasticity/job_flow_step_spec.rb

@@ -20,6 +20,12 @@ describe Elasticity::JobFlowStep do
                   </StepConfig>
                   <ExecutionStatusDetail>
                     <State>FAILED</State>
+                    <StartDateTime>
+                       2011-10-04T21:49:16Z
+                    </StartDateTime>
+                    <EndDateTime>
+                       2011-10-04T21:51:16Z
+                    </EndDateTime>
                   </ExecutionStatusDetail>
                 </member>
                 <member>
@@ -28,6 +34,10 @@ describe Elasticity::JobFlowStep do
                   </StepConfig>
                   <ExecutionStatusDetail>
                     <State>PENDING</State>
+                    <StartDateTime>
+                    </StartDateTime>
+                    <EndDateTime>
+                    </EndDateTime>
                   </ExecutionStatusDetail>
                 </member>
               </Steps>
@@ -46,6 +56,8 @@ describe Elasticity::JobFlowStep do
       jobflow_step = Elasticity::JobFlowStep.from_member_element(@members_nodeset[0])
       jobflow_step.name.should == "Setup Hive"
       jobflow_step.state.should == "FAILED"
+      jobflow_step.started_at.should == Time.parse("2011-10-04T21:49:16Z")
+      jobflow_step.ended_at.should == Time.parse("2011-10-04T21:51:16Z")
     end
   end
 
@@ -54,8 +66,10 @@ describe Elasticity::JobFlowStep do
       jobflow_steps = Elasticity::JobFlowStep.from_members_nodeset(@members_nodeset)
       jobflow_steps.map(&:name).should == ["Setup Hive", "Run Hive Script"]
       jobflow_steps.map(&:state).should == ["FAILED", "PENDING"]
+      jobflow_steps.map(&:started_at).should == [Time.parse("2011-10-04T21:49:16Z"), nil]
+      jobflow_steps.map(&:ended_at).should == [Time.parse("2011-10-04T21:51:16Z"), nil]
     end
   end
 
 
-end
+end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: elasticity
 version: !ruby/object:Gem::Version 
-  hash: 27
+  hash: 25
   prerelease: 
   segments: 
   - 1
   - 2
-  - 2
-  version: 1.2.2
+  - 3
+  version: 1.2.3
 platform: ruby
 authors: 
 - Robert Slifka
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-05-10 00:00:00 -07:00
+date: 2011-10-09 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -166,7 +166,7 @@ files:
 - Gemfile
 - HISTORY.mediawiki
 - LICENSE
-- README.mediawiki
+- README.md
 - Rakefile
 - elasticity.gemspec
 - lib/elasticity.rb
@@ -227,7 +227,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: "0"
 requirements: []
 
-rubyforge_project: elasticity
+rubyforge_project: 
 rubygems_version: 1.4.1
 signing_key: 
 specification_version: 3