elasticity 2.6 → 2.7

checksums.yaml

@@ -1,7 +1,15 @@
 ---
-SHA1:
-  metadata.gz: 3109d268557e25b4de8f9219f267bb9fcc2ee0c5
-  data.tar.gz: f42cd6f42d22afcc7fe14353afc0a15d6ca60a68
+!binary "U0hBMQ==":
+  metadata.gz: !binary |-
+    NWE3MTZjMzY1Mjc1NGM2MTBkMDgzNzlkZmRjZTllNWYxMzQ1MWE5Mg==
+  data.tar.gz: !binary |-
+    MzljODgxYWQ1YjhkOWZlZGVhZWViNzhhMzFlYjlhYTdjZmZiNjUyNg==
 SHA512:
-  metadata.gz: 8ff4ccbb6af6cffe65c60884685204db81c5d4be67dfafffc422d8d0e21e9ba710f75c05c4e86f37599c6b492424137a34b9cbf7907798900ea71779b60de611
-  data.tar.gz: b71875d6ca3db0f7ca14c431121c52bffbdf094deaed4fcb81929ef3e78f949d3796c98c9718a810230b96e653b518e113c7c6a0838b48b140985e54d80ae356
+  metadata.gz: !binary |-
+    OTU3OTk2ZjUxZTU1NTE2NGM0MTBhNjMzYTAxNjY3MzdhNTJiN2FlZmQzZDg3
+    NTI1Y2I2MDkzZmIwZDVmYzUzOTY1Y2U4YzY3MzkzNWJjOWI5YTVhNWRiNzQw
+    OWJmM2E1NjFkYTQ3ZDljNjZlZmExOWQ5Y2FmYTI3YTg4MThkMDM=
+  data.tar.gz: !binary |-
+    MTJmNWIxZWNiOGUxODI0N2MwYjE1OGVlNmZjYThmYjlkMDM0Yzg0MDQyZmNj
+    NTRiNGMxNmM0M2Y1ZDVhZGFjYTMxZGZkMzI2NjFlMTc0ZTk0ODdlYmFmNzU2
+    YjU4OWJmZDM3NDI5MWUwODFmYjBjN2M5NzNjODFlMzZkYjNmMDQ=

data/.ruby-version

@@ -1 +1 @@
-1.9.2-p320
+1.9.3-p484

data/.travis.yml

@@ -1,5 +1,4 @@
 language: ruby
 rvm:
-  - 1.9.2
   - 1.9.3
   - 2.0.0

data/HISTORY.md

@@ -1,12 +1,18 @@
+## 2.7 - December 4, 2013
+
+- `Elasticity::S3DistCp` and `Elasticity::ScriptStep` added to provide easy access to remote copying and arbitrary script execution.
+- `Elasticity::EMR` can now autodetect AWS credentials.
+- Baseline support now is now 1.9.3-p484.  Dropped support for Ruby 1.9.2 as it was [EOL on 2013/06/01](http://bugs.ruby-lang.org/projects/ruby/wiki/ReleaseEngineering).
+- Dev update: No longer using `guard-rspec`.
+- Dev update: Added `unf` as an explicit dependency as `fog` no longer includes it since it's AWS-only.
+
 ## 2.6 - August 17, 2013
 
 + Added debugging support via `JobFlow#enable_debugging`.  Note that this requires `JobFlow#log_uri` to be set.
 + Added job flow completion polling via `JobFlow#wait_for_completion`.
 + Added testing to support Ruby 2.0.
 + Removed support for REE and 1.8.7 as these are now unsupported versions of Ruby.  Common strategy in the Ruby community has been to only perform a minor version bump in this case; Elasticity is following suit.
-+ Now specifing minor versions in the gemspec.  With the release of Ruby 2.0, I'm anticipating breaking changes coming to many gems, and hoping that this mitigates those effects.
-
-### Development
++ Now specifying minor versions in the gemspec.  With the release of Ruby 2.0, I'm anticipating breaking changes coming to many gems, and hoping that this mitigates those effects.
 + Dev update: guard-rspec added.
 + Dev update: All development dependencies now require Ruby >= 1.9.2.
 + Dev update: Latest version of Ruby 1.9.2 (p320).

data/README.md

@@ -1,10 +1,10 @@
 [![Gem Version](https://badge.fury.io/rb/elasticity.png)](http://badge.fury.io/rb/elasticity)
 
-**(August 17, 2013)** Taking requests! I have a few ideas for what might be cool features though I'd rather work on what the community wants.  Go ahead and file an issue!
+**(December 4, 2013)** Taking requests! I have a few ideas for what might be cool features though I'd rather work on what the community wants.  Go ahead and file an issue!
 
 Elasticity provides programmatic access to Amazon's Elastic Map Reduce service.  The aim is to conveniently abstract away the complex EMR REST API and make working with job flows more productive and more enjoyable.
 
-[![Build Status](https://secure.travis-ci.org/rslifka/elasticity.png)](http://travis-ci.org/rslifka/elasticity) 1.9.2, 1.9.3, 2.0.0
+[![Build Status](https://secure.travis-ci.org/rslifka/elasticity.png)](http://travis-ci.org/rslifka/elasticity) 1.9.3, 2.0.0
 
 Elasticity provides two ways to access EMR:
 
@@ -20,7 +20,7 @@ gem install elasticity
 or in your Gemfile
 
 ```
-gem 'elasticity', '~> 2.6'
+gem 'elasticity', '~> 2.7'
 ```
 
 This will ensure that you protect yourself from API changes, which will only be made in major revisions.
@@ -281,13 +281,33 @@ jar_step.arguments = ['arg1', 'arg2']
 jobflow.add_step(jar_step)
 ```
 
+### Adding a Script Step
+
+```ruby
+# Path to your script, plus arguments
+script_step = Elasticity::ScriptStep.new('script_location', 'arg1', 'arg2')
+
+jobflow.add_step(script_step)
+```
+
+### Adding an S3DistCp Step
+
+For a complete list of supported arguments, please see the [Amazon EMR guide](http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html).
+
+```ruby
+# Path to your script, plus arguments (both symbols and strings are supported)
+copy_step = Elasticity::S3DistCpStep.new(:arg1 => 'value1', 'arg2' => 'value2')
+
+jobflow.add_step(copy_step)
+```
+
 ## 6 - Upload Assets (optional)
 
 This isn't part of ```JobFlow```; more of an aside.  Elasticity provides a very basic means of uploading assets to S3 so that your EMR job has access to them.  Most commonly this will be a set of resources to run the job (e.g. JAR files, streaming scripts, etc.) and a set of resources used by the job itself (e.g. a TSV file with a range of valid values, join tables, etc.).
 
 ```ruby
 # Specify the bucket name, AWS credentials and region
-s3 = Elasticity::SyncToS3('my-bucket', 'access', 'secret', 'region')
+s3 = Elasticity::SyncToS3.new('my-bucket', 'access', 'secret', 'region')
 
 # Alternatively, specify nothing :)
 # - Use the standard environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
@@ -331,7 +351,7 @@ And with a status callback, providing the elapsed time and an instance of ```Ela
 ```ruby
 # Blocks until status changes, calling back every 60 seconds
 jobflow.wait_for_completion do |elapsed_time, job_flow_status|
-  puts "Waiting for #{seconds}, jobflow status: #{status.state}"
+  puts "Waiting for #{elapsed_time}, jobflow status: #{job_flow_status.state}"
 end
 ```
 

data/elasticity.gemspec

@@ -14,9 +14,9 @@ Gem::Specification.new do |s|
   s.add_dependency('rest-client', '~> 1.0')
   s.add_dependency('nokogiri', '~> 1.0')
   s.add_dependency('fog', '~> 1.0')
+  s.add_dependency('unf', '~> 0.1')
 
   s.add_development_dependency('fakefs', '~> 0.4.0')
-  s.add_development_dependency('guard-rspec', '~> 3.0')
   s.add_development_dependency('rake', '~> 0.9')
   s.add_development_dependency('rspec', '~> 2.14.0')
   s.add_development_dependency('timecop', '~> 0.6')

data/lib/elasticity.rb

@@ -27,6 +27,8 @@ require 'elasticity/setup_hadoop_debugging_step'
 require 'elasticity/hive_step'
 require 'elasticity/pig_step'
 require 'elasticity/streaming_step'
+require 'elasticity/script_step'
+require 'elasticity/s3distcp_step'
 
 module Elasticity
 

data/lib/elasticity/emr.rb

@@ -4,7 +4,7 @@ module Elasticity
 
     attr_reader :aws_request
 
-    def initialize(aws_access_key_id, aws_secret_access_key, options = {})
+    def initialize(aws_access_key_id=nil, aws_secret_access_key=nil, options = {})
       @aws_request = Elasticity::AwsRequest.new(aws_access_key_id, aws_secret_access_key, options)
     end
 

data/lib/elasticity/s3distcp_step.rb

@@ -0,0 +1,16 @@
+module Elasticity
+
+  class S3DistCpStep < CustomJarStep
+
+    def initialize(options)
+      @name = 'Elasticity S3DistCp Step'
+      @jar = '/home/hadoop/lib/emr-s3distcp-1.0.jar'
+      @arguments = []
+      options.each do |argument, value|
+        @arguments << '--arg' << argument.to_s << '--arg' << value
+      end
+    end
+
+  end
+
+end

data/lib/elasticity/script_step.rb

@@ -0,0 +1,13 @@
+module Elasticity
+
+  class ScriptStep < CustomJarStep
+
+    def initialize(script_name, *script_args)
+      @name = 'Elasticity Script Step'
+      @jar = 's3://elasticmapreduce/libs/script-runner/script-runner.jar'
+      @arguments = ["#{script_name} #{script_args.join(' ')}"]
+    end
+
+  end
+
+end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = '2.6'
+  VERSION = '2.7'
 end

data/spec/lib/elasticity/emr_spec.rb

@@ -4,7 +4,24 @@ describe Elasticity::EMR do
     Elasticity::EMR.new('ACCESS', 'SECRET')
   end
 
-  its(:aws_request) { should == Elasticity::AwsRequest.new('ACCESS', 'SECRET', {}) }
+  describe '.new' do
+
+    context 'when arguments are provided' do
+      its(:aws_request) { should == Elasticity::AwsRequest.new('ACCESS', 'SECRET', {}) }
+    end
+
+    context 'when arguments are not provided' do
+      before do
+        ENV.stub(:[]).with('AWS_ACCESS_KEY_ID').and_return('ENV_ACCESS')
+        ENV.stub(:[]).with('AWS_SECRET_ACCESS_KEY').and_return('ENV_SECRET')
+      end
+      it 'should use environment variables' do
+        emr = Elasticity::EMR.new
+        emr.aws_request.should == Elasticity::AwsRequest.new('ENV_ACCESS', 'ENV_SECRET', {})
+      end
+    end
+
+  end
 
   describe '#add_instance_groups' do
 

data/spec/lib/elasticity/s3distcp_step_spec.rb

@@ -0,0 +1,11 @@
+describe Elasticity::S3DistCpStep do
+
+  it { should be_a Elasticity::CustomJarStep }
+
+  subject { Elasticity::S3DistCpStep.new({'arg1' => 'value1', :arg2 => 'value2'}) }
+
+  its(:name) { should == 'Elasticity S3DistCp Step' }
+  its(:jar) { should == '/home/hadoop/lib/emr-s3distcp-1.0.jar' }
+  its(:arguments) { should == %w(--arg arg1 --arg value1 --arg arg2 --arg value2) }
+
+end

data/spec/lib/elasticity/script_step_spec.rb

@@ -0,0 +1,11 @@
+describe Elasticity::ScriptStep do
+
+  it { should be_a Elasticity::CustomJarStep }
+
+  subject { Elasticity::ScriptStep.new('script_location', 'arg1', 'arg2') }
+
+  its(:name) { should == 'Elasticity Script Step' }
+  its(:jar) { should == 's3://elasticmapreduce/libs/script-runner/script-runner.jar' }
+  its(:arguments) { should == ['script_location arg1 arg2'] }
+
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: elasticity
 version: !ruby/object:Gem::Version
-  version: '2.6'
+  version: '2.7'
 platform: ruby
 authors:
 - Robert Slifka
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-17 00:00:00.000000000 Z
+date: 2013-12-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rest-client
@@ -53,33 +53,33 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency
-  name: fakefs
+  name: unf
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 0.4.0
-  type: :development
+        version: '0.1'
+  type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 0.4.0
+        version: '0.1'
 - !ruby/object:Gem::Dependency
-  name: guard-rspec
+  name: fakefs
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: 0.4.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: 0.4.0
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -156,6 +156,8 @@ files:
 - lib/elasticity/job_flow_step.rb
 - lib/elasticity/looper.rb
 - lib/elasticity/pig_step.rb
+- lib/elasticity/s3distcp_step.rb
+- lib/elasticity/script_step.rb
 - lib/elasticity/setup_hadoop_debugging_step.rb
 - lib/elasticity/streaming_step.rb
 - lib/elasticity/sync_to_s3.rb
@@ -175,6 +177,8 @@ files:
 - spec/lib/elasticity/job_flow_step_spec.rb
 - spec/lib/elasticity/looper_spec.rb
 - spec/lib/elasticity/pig_step_spec.rb
+- spec/lib/elasticity/s3distcp_step_spec.rb
+- spec/lib/elasticity/script_step_spec.rb
 - spec/lib/elasticity/setup_hadoop_debugging_step_spec.rb
 - spec/lib/elasticity/streaming_step_spec.rb
 - spec/lib/elasticity/sync_to_s3_spec.rb
@@ -199,7 +203,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.0.6
+rubygems_version: 2.1.11
 signing_key: 
 specification_version: 4
 summary: Streamlined, programmatic access to Amazon's Elastic Map Reduce service.
@@ -219,6 +223,8 @@ test_files:
 - spec/lib/elasticity/job_flow_step_spec.rb
 - spec/lib/elasticity/looper_spec.rb
 - spec/lib/elasticity/pig_step_spec.rb
+- spec/lib/elasticity/s3distcp_step_spec.rb
+- spec/lib/elasticity/script_step_spec.rb
 - spec/lib/elasticity/setup_hadoop_debugging_step_spec.rb
 - spec/lib/elasticity/streaming_step_spec.rb
 - spec/lib/elasticity/sync_to_s3_spec.rb