elasticity 5.0.3 → 6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f55ef4c0da52c5540b84f2f5c443a481dc3b0f63
-  data.tar.gz: 24d3f7bf29a3b61e0775d343b19c27fce4da6150
+  metadata.gz: 9e3ac59080cc54551c89cc60e8aa0eab2a83bfd5
+  data.tar.gz: 8533393df0bc0f7cfd5fb05e3e339dc98721b44e
 SHA512:
-  metadata.gz: e790d9f3b835d039b2c1a8522981bff5dfa7eb167e2a04e944c5a11c96bf3b44e881e61bc43bedaa80e0810da027f2eadf3fbb39730ccf98cc8c82a62995baa6
-  data.tar.gz: 8397623a0f99f13f6407e6619a22b33185a7ad2f480177d7bcef075d6e83e6dc1e3b32d928881f0546510d06232c73d62aa7c94c4184979f3037b9bb3f4d96f2
+  metadata.gz: b8db8baed719babf9489f48ca949283e8ca60c0262f864496a505f145b6490243ddfe3003563ce9720a72a6977066cf2ea3f60afcd6a654446591e984374a04b
+  data.tar.gz: e2e705bdb17457727cf12df927033edca67795ffe480e85b4863ef050a26999bf861fed9cd23e5f2f089c9225e2abe383a7543ce1b85e91371377abd414a0100

data/HISTORY.md

@@ -1,3 +1,29 @@
+## 6.0 - July 17, 2015
+
+Amazon is in the process of transitioning from the notion of "Job Flows" to "Clusters" and is updating their APIs as such.  You've already seen this in the EMR web UI as all mentions of "job flows" are gone and now you create "Clusters".
+
+On the API side, all of the newer commands take `cluster_id` rather than `job_flow_id`.  On the API submission side, they are transitioning from a 'flat' structure to a nested JSON structure requiring no transformation. Finally, XML is all gone and commands return JSON (i.e. no more Nokogiri as a dependency).
+
+They've also begun deprecating APIs, starting with `DescribeJobFlows`.  Given the sweeping set of changes, a major release was deemed appropriate.
+
+- [#88](https://github.com/rslifka/elasticity/issues/88) - Removed support for deprecated `DescribeJobFlows`. 
+- [#89](https://github.com/rslifka/elasticity/issues/89) - Add support for `AddTags`.
+- [#90](https://github.com/rslifka/elasticity/issues/90) - Add support for `RemoveTags`.
+- [#91](https://github.com/rslifka/elasticity/issues/91) - Add support for `SetVisibleToAllUsers`.
+- [#92](https://github.com/rslifka/elasticity/issues/92) - Add support for `ListSteps`.
+- [#93](https://github.com/rslifka/elasticity/issues/93) - Add support for `ListInstances`.
+- [#94](https://github.com/rslifka/elasticity/issues/94) - Add support for `ListInstanceGroups`.
+- [#95](https://github.com/rslifka/elasticity/issues/95) - Add support for `ListClusters`.
+- [#96](https://github.com/rslifka/elasticity/issues/96) - Add support for `ListBootstrapActions`.
+- [#97](https://github.com/rslifka/elasticity/issues/97) - Add support for `DescribeCluster`.
+- [#98](https://github.com/rslifka/elasticity/issues/98) - Add support for `DescribeStep`.
+- [#101](https://github.com/rslifka/elasticity/issues/101) - Fix plurality of `TerminateJobFlows`; now requires an array of IDs to terminate.
+- [#102](https://github.com/rslifka/elasticity/issues/102) - Simplify interface to `AddJobFlowSteps`; no longer require extraneous `:steps => []`.
+- [#104](https://github.com/rslifka/elasticity/issues/104) - Expose return value from `AddJobFlowSteps`.
+- [#105](https://github.com/rslifka/elasticity/issues/105) - `JobFlow#status` has been removed in favour of `JobFlow#cluster_status` and `JobFlow#cluster_step_status`.
+- [#107](https://github.com/rslifka/elasticity/issues/107) - Add support for temporary credentials via `Elasticity.configure`.
+- [#109](https://github.com/rslifka/elasticity/issues/109) - Credential specification relocated to `Elasticity.configure`.
+
 ## 5.0.3 - July 8, 2015
 
 - Fix for issue [#86](https://github.com/rslifka/elasticity/issues/86).

data/README.md

@@ -18,7 +18,7 @@ gem install elasticity
 or in your Gemfile
 
 ```
-gem 'elasticity', '~> 5.0'
+gem 'elasticity', '~> 6.0'
 ```
 
 This will ensure that you protect yourself from API changes, which will only be made in major revisions.
@@ -30,11 +30,14 @@ If you're familiar with the AWS EMR UI, you'll recall there are sample jobs Amaz
 ```ruby
 require 'elasticity'
 
-# Create a job flow with your AWS credentials
-jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
+# Specify your AWS credentials
+Elasticity.configure do |c|
+  c.access_key = ENV['AWS_ACCESS_KEY_ID']
+  c.secret_key = ENV['AWS_SECRET_ACCESS_KEY']
+end
 
-# Omit credentials to use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
-# jobflow = Elasticity::JobFlow.new
+# Create a job flow
+jobflow = Elasticity::JobFlow.new
 
 # NOTE: Amazon requires that all new accounts specify a VPC subnet when launching jobs.
 # If you're on an existing account, this is unnecessary however new AWS accounts require
@@ -44,7 +47,7 @@ jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
 # This is the first step in the jobflow - running a custom jar
 step = Elasticity::CustomJarStep.new('s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar')
 
-# Here are the arguments to pass to the jar
+# Here are the arguments to pass to the jar (replace OUTPUT_BUCKET)
 step.arguments = %w(s3n://elasticmapreduce/samples/cloudburst/input/s_suis.br s3n://elasticmapreduce/samples/cloudburst/input/100k.br s3n://OUTPUT_BUCKET/cloudburst/output/2012-06-22 36 3 0 1 240 48 24 24 128 16)
 
 # Add the step to the jobflow
@@ -60,6 +63,7 @@ Note that this example is only for ```CustomJarStep```.  Other steps will have d
 
 Job flows are the center of the EMR universe.  The general order of operations is:
 
+  1. Specify AWS credentials
   1. Create a job flow.
   1. Specify options.
   1. (optional) Configure instance groups.
@@ -71,31 +75,30 @@ Job flows are the center of the EMR universe.  The general order of operations i
   1. (optional) Wait for the job flow to complete.
   1. (optional) Shutdown the job flow.
 
-## 1 - Create a Job Flow
-
-Only your AWS credentials are needed.
+## 1 - Specify AWS Credentials
 
 ```ruby
-# Manually specify AWS credentials
-jobflow = Elasticity::JobFlow.new('AWS access key', 'AWS secret key')
+Elasticity.configure do |c|
+  c.access_key = ENV['AWS_ACCESS_KEY_ID']
+  c.secret_key = ENV['AWS_SECRET_ACCESS_KEY']
+end
+```
 
-# Use the standard environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
+## 2 - Create a Job Flow
+
+```ruby
 jobflow = Elasticity::JobFlow.new
 ```
 
 If you want to access a job flow that's already running:
 
 ```ruby
-# Manually specify AWS credentials
-jobflow = Elasticity::JobFlow.from_jobflow_id('AWS access key', 'AWS secret key', 'jobflow ID', 'region')
-
-# Use the standard environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
-jobflow = Elasticity::JobFlow.from_jobflow_id(nil, nil, 'jobflow ID', 'region')
+jobflow = Elasticity::JobFlow.from_jobflow_id('jobflow ID', 'region')
 ```
 
 This is useful if you'd like to attach to a running job flow and add more steps, etc.  The ```region``` parameter is necessary because job flows are only accessible from the the API when you connect to the same endpoint that created them (e.g. us-west-1).  If you don't specify the ```region``` parameter, us-east-1 is assumed.
 
-## 2 - Specifying Options
+## 3 - Specifying Options
 
 Configuration job flow options, shown below with default values.  Note that these defaults are subject to change - they are reasonable defaults at the time(s) I work on them (e.g. the latest version of Hadoop).
 
@@ -106,7 +109,7 @@ jobflow.name                              = 'Elasticity Job Flow'
 
 # For new AWS accounts, this is required to be set
 jobflow.ec2_subnet_id                     = nil
-jobflow.jobflow_role                      = nil
+jobflow.job_flow_role                     = nil
 jobflow.service_role                      = nil
 
 jobflow.action_on_failure                 = 'TERMINATE_JOB_FLOW'
@@ -124,7 +127,7 @@ jobflow.master_instance_type              = 'm1.small'
 jobflow.slave_instance_type               = 'm1.small'
 ```
 
-## 3 - Configure Instance Groups (optional)
+## 4 - Configure Instance Groups (optional)
 
 Technically this is optional since Elasticity creates MASTER and CORE instance groups for you (one m1.small instance in each).  If you'd like your jobs to finish in an appreciable amount of time, you'll want to at least add a few instances to the CORE group :)
 
@@ -170,7 +173,7 @@ ig.set_spot_instances(0.25)         # Makes this a SPOT group with a $0.25 bid p
 jobflow.set_core_instance_group(ig)
 ```
 
-## 4 - Add Bootstrap Actions (optional)
+## 5 - Add Bootstrap Actions (optional)
 
 Bootstrap actions are run as part of setting up the job flow, so be sure to configure these before running the job.
 
@@ -206,7 +209,7 @@ action = Elasticity::HadoopFileBootstrapAction.new('s3n://my-bucket/job-config.x
 jobflow.add_bootstrap_action(action)
 ```
 
-## 5 - Add Steps (optional)
+## 6 - Add Steps (optional)
 
 Each type of step has ```#name``` and ```#action_on_failure``` fields that can be specified.  Apart from that, steps are configured differently - exhaustively described below.
 
@@ -308,7 +311,7 @@ copy_step.arguments = [...]
 jobflow.add_step(copy_step)
 ```
 
-## 6 - Upload Assets (optional)
+## 7 - Upload Assets (optional)
 
 This isn't part of ```JobFlow```; more of an aside.  Elasticity provides a very basic means of uploading assets to S3 so that your EMR job has access to them.  Most commonly this will be a set of resources to run the job (e.g. JAR files, streaming scripts, etc.) and a set of resources used by the job itself (e.g. a TSV file with a range of valid values, join tables, etc.).
 
@@ -332,7 +335,7 @@ If the bucket doesn't exist, it will be created.
 
 If a file already exists, there is an MD5 checksum evaluation.  If the checksums are the same, the file will be skipped.  Now you can use something like ```s3n://my-bucket/remote-dir/this-job/tables/join.tsv``` in your EMR jobs.
 
-## 7 - Run the Job Flow
+## 8 - Run the Job Flow
 
 Submit the job flow to Amazon, storing the ID of the running job flow.
 
@@ -340,11 +343,11 @@ Submit the job flow to Amazon, storing the ID of the running job flow.
 jobflow_id = jobflow.run
 ```
 
-## 8 - Add Additional Steps (optional)
+## 9 - Add Additional Steps (optional)
 
 Steps can be added to a running jobflow just by calling ```#add_step``` on the job flow exactly how you add them prior to submitting the job.
 
-## 9 - Wait For the Job Flow to Complete (optional)
+## 10 - Wait For the Job Flow to Complete (optional)
 
 Elasticity has the ability to block until the status of a job flow is not STARTING or RUNNING.  There are two flavours.  Without a status callback:
 
@@ -362,7 +365,7 @@ jobflow.wait_for_completion do |elapsed_time, job_flow_status|
 end
 ```
 
-## 10 - Shut Down the Job Flow (optional)
+## 11 - Shut Down the Job Flow (optional)
 
 By default, job flows are set to terminate when there are no more running steps.  You can tell the job flow to stay alive when it has nothing left to do:
 
@@ -378,11 +381,15 @@ jobflow.shutdown
 
 # Elasticity Configuration
 
-Elasticity supports a wide range of configuration options :) all of which are shown below.
+Elasticity supports a handful of configuration options, all of which are shown below.
 
 ```ruby
 Elasticity.configure do |config|
 
+  # AWS credentials
+  config.access_key = ENV['AWS_ACCESS_KEY_ID']
+  config.secret_key = ENV['AWS_SECRET_ACCESS_KEY']
+
   # If using Hive, it will be configured via the directives here
   config.hive_site = 's3://bucket/hive-site.xml'
 

data/elasticity.gemspec

@@ -12,13 +12,13 @@ Gem::Specification.new do |s|
   s.description = %q{Streamlined, programmatic access to Amazon's Elastic Map Reduce service, driven by the Sharethrough team's requirements for belting out EMR jobs.}
 
   s.add_dependency('rest-client', '~> 1.0')
-  s.add_dependency('nokogiri', '~> 1.0')
   s.add_dependency('fog', '~> 1.0')
   s.add_dependency('unf', '~> 0.1')
 
+  s.add_development_dependency('factory_girl', '~> 4.0')
   s.add_development_dependency('fakefs', '~> 0.4.0')
   s.add_development_dependency('rake', '~> 0.9')
-  s.add_development_dependency('rspec', '~> 2.14.0')
+  s.add_development_dependency('rspec', '~> 3.0')
   s.add_development_dependency('timecop', '~> 0.6')
 
   s.files         = `git ls-files`.split("\n")

data/lib/elasticity.rb

@@ -9,7 +9,6 @@ require 'elasticity/version'
 
 require 'elasticity/aws_utils'
 require 'elasticity/aws_session'
-require 'elasticity/aws_request_v2'
 require 'elasticity/aws_request_v4'
 require 'elasticity/emr'
 
@@ -25,8 +24,8 @@ require 'elasticity/looper'
 require 'elasticity/job_flow'
 require 'elasticity/instance_group'
 
-require 'elasticity/job_flow_status'
-require 'elasticity/job_flow_status_step'
+require 'elasticity/cluster_status'
+require 'elasticity/cluster_step_status'
 
 require 'elasticity/custom_jar_step'
 require 'elasticity/setup_hadoop_debugging_step'
@@ -56,6 +55,9 @@ module Elasticity
 
   class Configuration
     attr_accessor :hive_site
+    attr_accessor :access_key
+    attr_accessor :secret_key
+    attr_accessor :security_token
   end
 
 end

data/lib/elasticity/aws_request_v4.rb

@@ -20,11 +20,21 @@ module Elasticity
       @ruby_service_hash.delete(:operation)
 
       @timestamp = Time.now.utc
+
+      @access_key = Elasticity.configuration.access_key
+      if @access_key == nil
+        raise ArgumentError, '.access_key must be set in the configuration block'
+      end
+
+      @secret_key = Elasticity.configuration.secret_key
+      if @secret_key == nil
+        raise ArgumentError, '.secret_key must be set in the configuration block'
+      end
     end
 
     def headers
-      {
-        'Authorization' => "AWS4-HMAC-SHA256 Credential=#{@aws_session.access_key}/#{credential_scope}, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=#{aws_v4_signature}",
+      headers = {
+        'Authorization' => "AWS4-HMAC-SHA256 Credential=#{@access_key}/#{credential_scope}, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=#{aws_v4_signature}",
         'Content-Type' => 'application/x-amz-json-1.1',
         'Host' => host,
         'User-Agent' => "elasticity/#{Elasticity::VERSION}",
@@ -32,6 +42,8 @@ module Elasticity
         'X-Amz-Date' => @timestamp.strftime('%Y%m%dT%H%M%SZ'),
         'X-Amz-Target' => "ElasticMapReduce.#{@operation}",
       }
+      headers.merge!('X-Amz-Security-Token' => Elasticity.configuration.security_token) if Elasticity.configuration.security_token
+      headers
     end
 
     def url
@@ -85,7 +97,7 @@ module Elasticity
     # Task 3: Calculate the AWS Signature Version 4
     #   http://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
     def aws_v4_signature
-      date = OpenSSL::HMAC.digest('sha256', 'AWS4' + @aws_session.secret_key, @timestamp.strftime('%Y%m%d'))
+      date = OpenSSL::HMAC.digest('sha256', 'AWS4' + @secret_key, @timestamp.strftime('%Y%m%d'))
       region = OpenSSL::HMAC.digest('sha256', date, @aws_session.region)
       service = OpenSSL::HMAC.digest('sha256', region, SERVICE_NAME)
       signing_key = OpenSSL::HMAC.digest('sha256', service, 'aws4_request')

data/lib/elasticity/aws_session.rb

@@ -7,15 +7,13 @@ module Elasticity
 
   class AwsSession
 
-    attr_reader :access_key
-    attr_reader :secret_key
     attr_reader :host
     attr_reader :region
 
     # Supported values for options:
     #  :region - AWS region (e.g. us-west-1)
     #  :secure - true or false, default true.
-    def initialize(access=nil, secret=nil, options={})
+    def initialize(options={})
       # There is a cryptic error if this isn't set
       if options.has_key?(:region) && options[:region] == nil
         raise MissingRegionError, 'A valid :region is required to connect to EMR'
@@ -23,8 +21,6 @@ module Elasticity
       options[:region] = 'us-east-1' unless options[:region]
       @region = options[:region]
 
-      @access_key = get_access_key(access)
-      @secret_key = get_secret_key(secret)
       @host = "elasticmapreduce.#@region.amazonaws.com"
     end
 
@@ -39,32 +35,17 @@ module Elasticity
 
     def ==(other)
       return false unless other.is_a? AwsSession
-      return false unless @access_key == other.access_key
-      return false unless @secret_key == other.secret_key
       return false unless @host == other.host
       true
     end
 
     private
 
-    def get_access_key(access)
-      return access if access
-      return ENV['AWS_ACCESS_KEY_ID'] if ENV['AWS_ACCESS_KEY_ID']
-      raise MissingKeyError, 'Please provide an access key or set AWS_ACCESS_KEY_ID.'
-    end
-
-    def get_secret_key(secret)
-      return secret if secret
-      return ENV['AWS_SECRET_ACCESS_KEY'] if ENV['AWS_SECRET_ACCESS_KEY']
-      raise MissingKeyError, 'Please provide a secret key or set AWS_SECRET_ACCESS_KEY.'
-    end
-
     # AWS error responses all follow the same form.  Extract the message from
     # the error document.
-    def self.parse_error_response(error_xml)
-      xml_doc = Nokogiri::XML(error_xml)
-      xml_doc.remove_namespaces!
-      xml_doc.xpath('/ErrorResponse/Error/Message').text
+    def self.parse_error_response(error_json)
+      error = JSON.parse(error_json)
+      "AWS EMR API Error (#{error['__type']}): #{error['message']}"
     end
 
   end

data/lib/elasticity/aws_utils.rb

@@ -27,35 +27,6 @@ module Elasticity
       end
     end
 
-    # Since we use the same structure as AWS, we can generate AWS param names
-    # from the Ruby versions of those names (and the param nesting).
-    def self.convert_ruby_to_aws(params)
-      result = {}
-      params.each do |key, value|
-        case value
-          when Array
-            prefix = "#{camelize(key.to_s)}.member"
-            value.each_with_index do |item, index|
-              if item.is_a?(String)
-                result["#{prefix}.#{index+1}"] = item
-              else
-                convert_ruby_to_aws(item).each do |nested_key, nested_value|
-                  result["#{prefix}.#{index+1}.#{nested_key}"] = nested_value
-                end
-              end
-            end
-          when Hash
-            prefix = "#{camelize(key.to_s)}"
-            convert_ruby_to_aws(value).each do |nested_key, nested_value|
-              result["#{prefix}.#{nested_key}"] = nested_value
-            end
-          else
-            result[camelize(key.to_s)] = value
-        end
-      end
-      result
-    end
-
     def self.camelize(word)
       word.to_s.gsub(/\/(.?)/) { '::' + $1.upcase }.gsub(/(^|_)(.)/) { $2.upcase }
     end

data/lib/elasticity/cluster_status.rb

@@ -0,0 +1,38 @@
+module Elasticity
+
+  class ClusterStatus
+
+    attr_accessor :name
+    attr_accessor :cluster_id
+    attr_accessor :state
+    attr_accessor :created_at
+    attr_accessor :ready_at
+    attr_accessor :ended_at
+    attr_accessor :last_state_change_reason
+    attr_accessor :master_public_dns_name
+    attr_accessor :normalized_instance_hours
+
+    # ClusterStatus is created via the results of the DescribeCluster API call
+    def self.from_aws_data(cluster_data)
+      cluster_data = cluster_data['Cluster']
+      ClusterStatus.new.tap do |c|
+        c.name = cluster_data['Name']
+        c.cluster_id = cluster_data['Id']
+        c.state = cluster_data['Status']['State']
+        c.created_at = Time.at(cluster_data['Status']['Timeline']['CreationDateTime'])
+        c.ready_at = Time.at(cluster_data['Status']['Timeline']['ReadyDateTime'])
+        c.ended_at = Time.at(cluster_data['Status']['Timeline']['EndDateTime'])
+        c.last_state_change_reason = cluster_data['Status']['StateChangeReason']['Code']
+        c.master_public_dns_name = cluster_data['MasterPublicDnsName']
+        c.normalized_instance_hours = cluster_data['NormalizedInstanceHours']
+      end
+    end
+
+    # http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/ProcessingCycle.html
+    def active?
+      %w{RUNNING STARTING BOOTSTRAPPING WAITING SHUTTING_DOWN}.include?(@state)
+    end
+
+  end
+
+end

data/lib/elasticity/cluster_step_status.rb

@@ -0,0 +1,51 @@
+module Elasticity
+
+  class ClusterStepStatus
+
+    attr_accessor :action_on_failure
+    attr_accessor :args
+    attr_accessor :jar
+    attr_accessor :main_class
+    attr_accessor :properties
+    attr_accessor :step_id
+    attr_accessor :name
+    attr_accessor :state
+    attr_accessor :state_change_reason
+    attr_accessor :state_change_reason_message
+    attr_accessor :created_at
+    attr_accessor :started_at
+    attr_accessor :ended_at
+
+    # Constructed from http://docs.aws.amazon.com/ElasticMapReduce/latest/API/API_ListSteps.html
+    def self.from_aws_list_data(cluster_step_data)
+      cluster_step_data['Steps'].map do |s|
+        ClusterStepStatus.new.tap do |c|
+          c.action_on_failure = s['ActionOnFailure']
+          c.args = s['Config']['Args']
+          c.jar = s['Config']['Jar']
+          c.main_class = s['Config']['MainClass']
+          c.properties = s['Config']['Properties']
+          c.step_id = s['Id']
+          c.name = s['Name']
+          c.state = s['Status']['State']
+          c.state_change_reason = s['Status']['StateChangeReason']['Code']
+          c.state_change_reason_message = s['Status']['StateChangeReason']['Message']
+          c.created_at = Time.at(s['Status']['Timeline']['CreationDateTime'])
+          c.started_at = Time.at(s['Status']['Timeline']['StartDateTime'])
+          c.ended_at   = Time.at(s['Status']['Timeline']['EndDateTime'])
+        end
+      end
+    end
+
+    def self.installed_steps(cluster_step_statuses)
+      step_names = cluster_step_statuses.map(&:name)
+      installed_steps = []
+      Elasticity::JobFlowStep.steps_requiring_installation.each do |step|
+        installed_steps << step if step_names.include?(step.aws_installation_step_name)
+      end
+      installed_steps
+    end
+
+  end
+
+end