elasticity 1.4.1 → 1.5

data/.rvmrc

@@ -1 +1 @@
-rvm use ree-1.8.7-2010.02@elasticity --create
+rvm use ruby-1.9.2-p180@elasticity --create

data/.travis.yml

@@ -0,0 +1,11 @@
+language: ruby
+rvm:
+  - 1.8.7
+  - 1.9.2
+  - 1.9.3
+  - ree
+branches:
+  only:
+    - master
+    - travis-testing
+

data/HISTORY.md

@@ -1,3 +1,9 @@
+### 1.5
+
++ Added support for Hadoop bootstrap actions to all job types (Pig, Hive and Custom Jar).
++ Added support for REE 1.8.7-2011.12, Ruby 1.9.2 and 1.9.3.
++ Updated to the latest versions of all development dependencies (notably VCR 2).
+
 ### 1.4.1
 
 + Added Elasticity::EMR#describe_jobflow("jobflow_id") for describing a specific job.  If you happen to run hundreds of EMR jobs, this makes retrieving jobflow status much faster than using Elasticity::EMR#describe_jobflowS which pulls down and parses XML status for hundreds of jobs.

data/README.md

@@ -1,10 +1,6 @@
 Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently wrap the API operations in a manner that makes working with EMR job flows from Ruby more productive and more enjoyable, without having to understand the nuts and bolts of the EMR REST API.  At the very least, using Elasticity allows you to easily experiment with the EMR API :)
 
-**CREDITS**: AWS signing was used from [RightScale's](http://www.rightscale.com/) amazing [right_aws gem](https://github.com/rightscale/right_aws) which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.  Used camelize from ActiveSupport as well, thank you \Rails :)
-
-**CONTRIBUTIONS**:
-
-+ [Wouter Broekhof](https://github.com/wouter/) - HTTPS and AWS region support, additional params to describe_jobflows.
+[![Build Status](https://secure.travis-ci.org/rslifka/elasticity.png)](http://travis-ci.org/rslifka/elasticity)
 
 # Installation and Usage
 
@@ -99,7 +95,7 @@ Use this as you would any other Pig variable.
 Custom jar jobs are also available.  To kick off a custom job, specify the path to the jar and any arguments you'd like passed to the jar.
 
 <pre>
-  custom_jar = Elasticity::PigJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
+  custom_jar = Elasticity::CustomJarJob.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_KEY"])
   custom_jar.log_uri = "s3n://slif-test/output/logs"
   custom_jar.action_on_failure = "TERMINATE_JOB_FLOW"
   jobflow_id = custom_jar.run('s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar', [
@@ -111,6 +107,12 @@ Custom jar jobs are also available.  To kick off a custom job, specify the path
   > "j-1IU6NM8OUPS9I"
 </pre>
 
+Custom jar jobs support arbitrary entry points.  Specify the class on which to call main() either via the JAR manifest or as the first argument to the job:
+
+<pre>
+  Elasticity::CustomJarJob.new(key, secret).run(s3_jar_path, ['MyCustomClass', 'arg1', 'arg2'])
+</pre>
+
 # Amazon API Reference
 
 Elasticity wraps all of the EMR API calls.  Please see the Amazon guide for details on these operations because the default values aren't obvious (e.g. the meaning of <code>DescribeJobFlows</code> without parameters).
@@ -343,6 +345,19 @@ If you're chomping at the bit to initiate some EMR functionality that isn't wrap
   > &lt;DescribeJobFlowsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009...
 </pre>
 
+# Something Borrowed...
+
+AWS signing was used from [RightScale's](http://www.rightscale.com/) amazing [right_aws gem](https://github.com/rightscale/right_aws) which works extraordinarily well!  If you need access to any AWS service (EC2, S3, etc.), have a look.
+
+Used camelize from ActiveSupport as well, thank you \Rails :)
+
+# Thanks!
+
+Thanks to the following people who have contributed patches or helpful suggestions:
+
++ [Aram Price](https://github.com/aramprice/)
++ [Wouter Broekhof](https://github.com/wouter/)
+
 # License
 
 <pre>
@@ -363,6 +378,8 @@ If you're chomping at the bit to initiate some EMR functionality that isn't wrap
 
 ### Development Notes for Slif
 
+[Versioning Guide](http://docs.rubygems.org/read/chapter/7#page27), c/o [@brokenladder](https://twitter.com/#!/brokenladder)
+
 <pre>
   rake build    # Build lorem-0.0.2.gem into the pkg directory
   rake install  # Build and install lorem-0.0.2.gem into system gems

data/elasticity.gemspec

@@ -9,18 +9,15 @@ Gem::Specification.new do |s|
   s.authors     = ["Robert Slifka"]
   s.homepage    = "http://www.github.com/rslifka/elasticity"
   s.summary     = %q{Programmatic access to Amazon's Elastic Map Reduce service.}
-  s.description = %q{Programmatic access to Amazon's Elastic Map Reduce service.}
+  s.description = %q{Programmatic access to Amazon's Elastic Map Reduce service, driven by the Sharethrough team's requirements for belting out EMR jobs.}
 
   s.add_dependency("rest-client")
   s.add_dependency("nokogiri")
 
-  s.add_development_dependency("autotest-fsevent")
-  s.add_development_dependency("autotest-growl")
   s.add_development_dependency("rake")
-  s.add_development_dependency("rspec",   ">= 2.5.0")
-  s.add_development_dependency("vcr",     ">= 1.5.1")
-  s.add_development_dependency("webmock", ">= 1.6.2")
-  s.add_development_dependency("ZenTest")
+  s.add_development_dependency("rspec",   ">= 2.8.0")
+  s.add_development_dependency("vcr",     "~> 2.0")
+  s.add_development_dependency("webmock", "~> 1.8.0")
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")

data/lib/elasticity/custom_jar_job.rb

@@ -27,9 +27,10 @@ module Elasticity
           }
         ]
       }
-      jobflow_config.merge!(:log_uri => @log_uri) if @log_uri
+
       jobflow_config[:steps].first[:hadoop_jar_step][:args] = arguments if arguments
-      @emr.run_job_flow(jobflow_config)
+
+      run_job(jobflow_config)
     end
 
   end

data/lib/elasticity/hive_job.rb

@@ -61,9 +61,7 @@ module Elasticity
         ]
       }
 
-      jobflow_config.merge!(:log_uri => @log_uri) if @log_uri
-      
-      @emr.run_job_flow(jobflow_config)
+      run_job(jobflow_config)
     end
 
   end

data/lib/elasticity/pig_job.rb

@@ -75,10 +75,7 @@ module Elasticity
         ]
       }
 
-      jobflow_config.merge!(:log_uri => @log_uri) if @log_uri
-      jobflow_config.merge!(get_bootstrap_actions)
-
-      @emr.run_job_flow(jobflow_config)
+      run_job(jobflow_config)
     end
 
     private

data/lib/elasticity/simple_job.rb

@@ -40,9 +40,10 @@ module Elasticity
 
     private
 
-    def get_bootstrap_actions
-      return {} unless @hadoop_actions && !@hadoop_actions.empty?
-      { :bootstrap_actions => @hadoop_actions }
+    def run_job(jobflow_config)
+      jobflow_config.merge!(:log_uri => @log_uri) if @log_uri
+      jobflow_config.merge!(:bootstrap_actions => @hadoop_actions) if @hadoop_actions && !@hadoop_actions.empty?
+      @emr.run_job_flow(jobflow_config)
     end
 
   end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = "1.4.1"
+  VERSION = "1.5"
 end

data/spec/fixtures/vcr_cassettes/add_instance_groups/one_group_successful.yml

@@ -1,38 +1,44 @@
 --- 
-- !ruby/struct:VCR::HTTPInteraction 
-  request: !ruby/struct:VCR::Request 
-    method: :get
-    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&InstanceGroups.member.1.InstanceCount=1&InstanceGroups.member.1.InstanceRole=TASK&InstanceGroups.member.1.InstanceType=m1.small&InstanceGroups.member.1.Market=ON_DEMAND&InstanceGroups.member.1.Name=Go%20Canucks%20Go!&JobFlowId=j-OALI7TZTQMHX&Operation=AddInstanceGroups/
+http_interactions: 
+- request: 
+    method: get
+    uri: !ruby/regexp /^https/
     body:
+      encoding: US-ASCII
+      string: ""
     headers: 
-      accept: 
+      Accept: 
       - "*/*; q=0.5, application/xml"
-      accept-encoding: 
+      Accept-Encoding: 
       - gzip, deflate
-  response: !ruby/struct:VCR::Response 
-    status: !ruby/struct:VCR::ResponseStatus 
+  response: 
+    status: 
       code: 200
       message: OK
     headers: 
-      x-amzn-requestid: 
-      - ddd0d158-67eb-11e0-ba06-2b5c43005be2
-      content-type: 
-      - text/xml
-      date: 
-      - Sat, 16 Apr 2011 05:39:15 GMT
-      content-length: 
+      Content-Length: 
       - "411"
-    body: |
-      <AddInstanceGroupsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
-        <AddInstanceGroupsResult>
-          <JobFlowId>j-OALI7TZTQMHX</JobFlowId>
-          <InstanceGroupIds>
-            <member>ig-2GOVEN6HVJZID</member>
-          </InstanceGroupIds>
-        </AddInstanceGroupsResult>
-        <ResponseMetadata>
-          <RequestId>ddd0d158-67eb-11e0-ba06-2b5c43005be2</RequestId>
-        </ResponseMetadata>
-      </AddInstanceGroupsResponse>
+      Date: 
+      - Sat, 16 Apr 2011 05:39:15 GMT
+      Content-Type: 
+      - text/xml
+      X-Amzn-Requestid: 
+      - ddd0d158-67eb-11e0-ba06-2b5c43005be2
+    body: 
+      encoding: US-ASCII
+      string: |
+        <AddInstanceGroupsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+          <AddInstanceGroupsResult>
+            <JobFlowId>j-OALI7TZTQMHX</JobFlowId>
+            <InstanceGroupIds>
+              <member>ig-2GOVEN6HVJZID</member>
+            </InstanceGroupIds>
+          </AddInstanceGroupsResult>
+          <ResponseMetadata>
+            <RequestId>ddd0d158-67eb-11e0-ba06-2b5c43005be2</RequestId>
+          </ResponseMetadata>
+        </AddInstanceGroupsResponse>
 
     http_version: "1.1"
+  recorded_at: Sat, 03 Mar 2012 22:59:32 GMT
+recorded_with: VCR 2.0.0

data/spec/fixtures/vcr_cassettes/add_instance_groups/one_group_unsuccessful.yml

@@ -1,35 +1,41 @@
---- 
-- !ruby/struct:VCR::HTTPInteraction 
-  request: !ruby/struct:VCR::Request 
-    method: :get
-    uri: !ruby/regexp /^https:\/\/elasticmapreduce.amazonaws.com\/\?AWSAccessKeyId=\S+&InstanceGroups.member.1.BidPrice=0&InstanceGroups.member.1.InstanceCount=1&InstanceGroups.member.1.InstanceRole=TASK&InstanceGroups.member.1.InstanceType=m1.small&InstanceGroups.member.1.Market=ON_DEMAND&InstanceGroups.member.1.Name=Go%20Canucks%20Go!&JobFlowId=j-19WDDS68ZUENP&Operation=AddInstanceGroups/
+---
+http_interactions:
+- request:
+    method: get
+    uri: !ruby/regexp /^https/
     body:
-    headers: 
-      accept: 
+      encoding: US-ASCII
+      string: ""
+    headers:
+      Accept:
       - "*/*; q=0.5, application/xml"
-      accept-encoding: 
+      Accept-Encoding:
       - gzip, deflate
-  response: !ruby/struct:VCR::Response 
-    status: !ruby/struct:VCR::ResponseStatus 
+  response:
+    status:
       code: 400
       message: Bad Request
-    headers: 
-      x-amzn-requestid: 
-      - 0c8d744d-67ea-11e0-bf8a-ed57a5465c87
-      content-type: 
-      - text/xml
-      date: 
-      - Sat, 16 Apr 2011 05:26:15 GMT
-      content-length: 
+    headers:
+      Content-Length:
       - "337"
-    body: |
-      <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
-        <Error>
-          <Type>Sender</Type>
-          <Code>ValidationError</Code>
-          <Message>Task instance group already exists in the job flow, cannot add more task groups</Message>
-        </Error>
-        <RequestId>0c8d744d-67ea-11e0-bf8a-ed57a5465c87</RequestId>
-      </ErrorResponse>
+      Date:
+      - Sat, 16 Apr 2011 05:26:15 GMT
+      Content-Type:
+      - text/xml
+      X-Amzn-Requestid:
+      - 0c8d744d-67ea-11e0-bf8a-ed57a5465c87
+    body:
+      encoding: US-ASCII
+      string: |
+        <ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
+          <Error>
+            <Type>Sender</Type>
+            <Code>ValidationError</Code>
+            <Message>Task instance group already exists in the job flow, cannot add more task groups</Message>
+          </Error>
+          <RequestId>0c8d744d-67ea-11e0-bf8a-ed57a5465c87</RequestId>
+        </ErrorResponse>
 
     http_version: "1.1"
+  recorded_at: Sat, 03 Mar 2012 22:59:41 GMT
+recorded_with: VCR 2.0.0