elasticity 2.0 → 2.1

data/HISTORY.md

@@ -1,3 +1,9 @@
+## 2.1 - July 7, 2012
+
++ TASK instance group support added.
++ SPOT instance support added for all instance group types. 
++ Removed name of jar from default name of ```CustomJarStep``` since the AWS UI already calls it out in a separate column when looking at job flow steps.
+
 ## 2.0 - June 26, 2012
 
 2.0 is a rewrite of the simplified API after a year's worth of daily use at [Sharethrough](http://www.sharethrough.com/).  We're investing heavily in our data processing infrastucture and many Elasticity feature ideas have come from those efforts.
@@ -21,59 +27,61 @@ In order to move more quickly and support interesting features like a command-li
 + Drastic simplification of the testing around EMR submission, reducing LoC (however important that metric is you :) and complexity by ~50%.
 + Development dependency updates: updated to ruby-1.9.3-p194 and rspec-2.10.  Removed dependency on VCR and WebMock (no longer using either of these).
 
-## 1.5
+## 1.5 - March 5, 2012
 
 + Added support for Hadoop bootstrap actions to all job types (Pig, Hive and Custom Jar).
 + Added support for REE 1.8.7-2011.12, Ruby 1.9.2 and 1.9.3.
 + Updated to the latest versions of all development dependencies (notably VCR 2).
 
-## 1.4.1
+## 1.4.1 - December 17, 2011
 
 + Added ```Elasticity::EMR#describe_jobflow("jobflow_id")``` for describing a specific job.  If you happen to run hundreds of EMR jobs, this makes retrieving jobflow status much faster than using ```Elasticity::EMR#describe_jobflowS``` which pulls down and parses XML status for hundreds of jobs.
 
-## 1.4
+## 1.4 - December 9, 2011
 
 + Added ```Elasticity::CustomJarJob``` for launching "Custom Jar" jobs.
 
-## 1.3.1
+## 1.3.1 - November 16, 2011
 
 + Explicitly requiring 'time' (only a problem if you aren't running from within a Rails environment).
 + ```Elasticity::JobFlow``` now exposes ```last_state_change_reason```.
 
-## 1.3 (Contributions from Wouter Broekhof)
+## 1.3 - October 10, 2011
+
+This release primarily contains contributions from Wouter Broekhof
 
 + The default mode of communication is now via HTTPS.
 + ```Elasticity::AwsRequest``` new option ```:secure => true|false``` (whether to use HTTPS).
 + ```Elasticity::AwsRequest``` new option ```:region => eu-west-1|...``` (which region to run the EMR job).
 + ```Elasticity::EMR#describe_jobflows``` now accepts additional params for filtering the jobflow query (see docs).
 
-## 1.2.2
+## 1.2.2 - May 10, 2011
 
 + ```HiveJob``` and ```PigJob``` now support configuring Hadoop options via ```#add_hadoop_bootstrap_action()```.
 
-## 1.2.1
+## 1.2.1 - May 7, 2011
 
 + Shipping up E_PARALLELS Pig variable with each invocation; reasonable default value for PARALLEL based on the number and type of instances configured.
 
-## 1.2
+## 1.2 - May 4, 2011
 
 + Added ```PigJob```!
 
-## 1.1.1
+## 1.1.1 - April 25, 2011
 
 + ```HiveJob``` critical bug fixed, now it works :)
 + Added ```log_uri``` and ```action_on_failure``` as options to ```HiveJob```.
 + Added integration tests to ```HiveJob```.
 
-## 1.1
+## 1.1 - April 24, 2011
 
 + Added ```HiveJob```, a simplified way to launch basic Hive job flows.
 + Added HISTORY.
 
-## 1.0.1
+## 1.0.1 - April 22, 2011
 
 + Added LICENSE.
 
-## 1.0
+## 1.0 - April 22, 2011
 
 + Released!

data/README.md

@@ -52,7 +52,8 @@ Job flows are the center of the EMR universe.  The general order of operations i
 
   1. Create a job flow.
   1. Specify options.
-  1. Add bootstrap actions.
+  1. (optional) Configure instance groups.
+  1. (optional) Add bootstrap actions.
   1. Create steps.
   1. Run the job flow.
   1. (optional) Add additional steps.
@@ -78,15 +79,63 @@ jobflow.ami_version                       = 'latest'
 jobflow.ec2_key_name                      = 'default'
 jobflow.ec2_subnet_id                     = nil
 jobflow.hadoop_version                    = '0.20.205'
-jobflow.instance_count                    = 2
 jobflow.keep_job_flow_alive_when_no_steps = true
 jobflow.log_uri                           = nil
-jobflow.master_instance_type              = 'm1.small'
 jobflow.name                              = 'Elasticity Job Flow'
+jobflow.instance_count                    = 2
+jobflow.master_instance_type              = 'm1.small'
 jobflow.slave_instance_type               = 'm1.small'
 ```
 
-## 3 - Adding Bootstrap Actions
+## 3 - Configuring Instance Groups (optional)
+
+Technically this is optional since Elasticity creates MASTER and CORE instance groups for you (one m1.small instance in each).  If you'd like your jobs to finish in an appreciable amount of time, you'll want to at least add a few instances to the CORE group :)
+
+### The Easy Way™
+
+If all you'd like to do is change the type or number of instances, ```JobFlow``` provides a few shortcuts to do just that.
+
+```
+jobflow.instance_count       = 10
+jobflow.master_instance_type = 'm1.small'
+jobflow.slave_instance_type  = 'c1.medium'
+```
+
+This says "I want 10 instances from EMR: one m1.small MASTER instance and nine c1.medium CORE instances."
+
+### The Still-Easy Way™
+
+Elasticity supports all EMR instance group types and all configuration options. The MASTER, CORE and TASK instance groups can be configured via ```JobFlow#set_master_instance_group```, ```JobFlow#set_core_instance_group``` and ```JobFlow#set_task_instance_group``` respectively.
+
+#### On-Demand Instance Groups
+
+These instances will be available for the life of your EMR job, versus Spot instances which are transient depending on your bid price (see below).
+
+```
+ig = Elasticity::InstanceGroup.new
+ig.count = 10                       # Provision 10 instances
+ig.type  = 'c1.medium'              # See the EMR docs for a list of supported types
+ig.set_on_demand_instances          # This is the default setting
+
+
+jobflow.set_core_instance_group(ig)
+```
+
+#### Spot Instance Groups
+
+*When Amazon EC2 has unused capacity, it offers EC2 instances at a reduced cost, called the Spot Price. This price fluctuates based on availability and demand. You can purchase Spot Instances by placing a request that includes the highest bid price you are willing to pay for those instances. When the Spot Price is below your bid price, your Spot Instances are launched and you are billed the Spot Price. If the Spot Price rises above your bid price, Amazon EC2 terminates your Spot Instances.* - [EMR Developer Guide](http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_SpotInstances.html)
+
+```
+ig = Elasticity::InstanceGroup.new
+ig.count = 10                       # Provision 10 instances
+ig.type  = 'c1.medium'              # See the EMR docs for a list of supported types
+ig.set_spot_instances(0.25)         # Makes this a SPOT group with a $0.25 bid price
+
+
+jobflow.set_core_instance_group(ig)
+```
+
+## 4 - Adding Bootstrap Actions (optional)
 
 Bootstrap actions are run as part of setting up the job flow, so be sure to configure these before running the job.
 
@@ -100,7 +149,7 @@ Bootstrap actions are run as part of setting up the job flow, so be sure to conf
 end
 ```
 
-## 4 - Adding Steps
+## 5 - Adding Steps
 
 Each type of step has a default name that can be overridden (the :name field).  Apart from that, steps are configured differently - exhaustively described below.
 
@@ -170,7 +219,7 @@ jar_step.arguments = ['arg1', 'arg2']
 jobflow.add_step(jar_step)
 ```
 
-## 5 - Running the Job Flow
+## 6 - Running the Job Flow
 
 Submit the job flow to Amazon, storing the ID of the running job flow.
 
@@ -178,11 +227,11 @@ Submit the job flow to Amazon, storing the ID of the running job flow.
 jobflow_id = jobflow.run
 ```
 
-## 6 - Adding Additional Steps (optional)
+## 7 - Adding Additional Steps (optional)
 
 Steps can be added to a running jobflow just by calling ```#add_step``` on the job flow exactly how you add them prior to submitting the job.
 
-## 7 - Shutting Down the Job Flow (optional)
+## 8 - Shutting Down the Job Flow (optional)
 
 By default, job flows are set to terminate when there are no more running steps.  You can tell the job flow to stay alive when it has nothing left to do:
 

data/lib/elasticity.rb

@@ -13,6 +13,8 @@ require 'elasticity/hadoop_bootstrap_action'
 require 'elasticity/job_flow_step'
 
 require 'elasticity/job_flow'
+require 'elasticity/instance_group'
+
 require 'elasticity/job_flow_status'
 require 'elasticity/job_flow_status_step'
 

data/lib/elasticity/custom_jar_step.rb

@@ -10,7 +10,7 @@ module Elasticity
     attr_accessor :action_on_failure
 
     def initialize(jar)
-      @name = "Elasticity Custom Jar Step (#{jar})"
+      @name = 'Elasticity Custom Jar Step'
       @jar = jar
       @arguments = []
       @action_on_failure = 'TERMINATE_JOB_FLOW'

data/lib/elasticity/instance_group.rb

@@ -0,0 +1,57 @@
+module Elasticity
+
+  class InstanceGroup
+
+    ROLES = %w(MASTER CORE TASK)
+
+    attr_accessor :count
+    attr_accessor :type
+    attr_accessor :role
+
+    attr_reader :bid_price
+    attr_reader :market
+
+    def initialize
+      @count = 1
+      @type = 'm1.small'
+      @market = 'ON_DEMAND'
+      @role = 'CORE'
+    end
+
+    def count=(instance_count)
+      raise_if instance_count <= 0, ArgumentError, "Instance groups require at least 1 instance (#{instance_count} requested)"
+      raise_if @role == 'MASTER' && instance_count != 1, ArgumentError, "MASTER instance groups can only have 1 instance (#{instance_count} requested)"
+      @count = instance_count
+    end
+
+    def role=(group_role)
+      raise_unless ROLES.include?(group_role), ArgumentError, "Role must be one of MASTER, CORE or TASK (#{group_role} was requested)"
+      @count = 1 if group_role == 'MASTER'
+      @role = group_role
+    end
+
+    def set_spot_instances(bid_price)
+      raise_unless bid_price > 0, ArgumentError, "The bid price for spot instances should be greater than 0 (#{bid_price} requested)"
+      @bid_price = bid_price
+      @market = 'SPOT'
+    end
+
+    def set_on_demand_instances
+      @bid_price = nil
+      @market = 'ON_DEMAND'
+    end
+
+    def to_aws_instance_config
+      {
+        :market => @market,
+        :instance_count => @count,
+        :instance_type => @type,
+        :instance_role => @role,
+      }.tap do |config|
+        config.merge!(:bid_price => @bid_price) if @market == 'SPOT'
+      end
+    end
+
+  end
+
+end

data/lib/elasticity/job_flow.rb

@@ -22,30 +22,61 @@ module Elasticity
       @action_on_failure = 'TERMINATE_JOB_FLOW'
       @ec2_key_name = 'default'
       @hadoop_version = '0.20.205'
-      @instance_count = 2
-      @master_instance_type = 'm1.small'
       @name = 'Elasticity Job Flow'
-      @slave_instance_type = 'm1.small'
       @ami_version = 'latest'
       @keep_job_flow_alive_when_no_steps = false
 
-      @emr = Elasticity::EMR.new(access, secret)
-
       @bootstrap_actions = []
       @jobflow_steps = []
       @installed_steps = []
+
+      @instance_groups = {}
+      set_master_instance_group(Elasticity::InstanceGroup.new)
+      set_core_instance_group(Elasticity::InstanceGroup.new)
+
+      @instance_count = 2
+      @master_instance_type = 'm1.small'
+      @slave_instance_type = 'm1.small'
+
+      @emr = Elasticity::EMR.new(access, secret)
     end
 
     def instance_count=(count)
       raise ArgumentError, 'Instance count cannot be set to less than 2 (requested 1)' unless count > 1
+      @instance_groups[:core].count = count - 1
       @instance_count = count
     end
 
+    def master_instance_type=(type)
+      @instance_groups[:master].type = type
+      @master_instance_type = type
+    end
+
+    def slave_instance_type=(type)
+      @instance_groups[:core].type = type
+      @slave_instance_type = type
+    end
+
     def add_bootstrap_action(bootstrap_action)
       raise_if is_jobflow_running?, JobFlowRunningError, 'To modify bootstrap actions, please create a new job flow.'
       @bootstrap_actions << bootstrap_action
     end
 
+    def set_master_instance_group(instance_group)
+      instance_group.role = 'MASTER'
+      @instance_groups[:master] = instance_group
+    end
+
+    def set_core_instance_group(instance_group)
+      instance_group.role = 'CORE'
+      @instance_groups[:core] = instance_group
+    end
+
+    def set_task_instance_group(instance_group)
+      instance_group.role = 'TASK'
+      @instance_groups[:task] = instance_group
+    end
+
     def add_step(jobflow_step)
       if is_jobflow_running?
         jobflow_steps = []
@@ -90,20 +121,18 @@ module Elasticity
     end
 
     def jobflow_preamble
-      {
+      preamble = {
         :name => @name,
         :ami_version => @ami_version,
         :instances => {
           :keep_job_flow_alive_when_no_steps => @keep_job_flow_alive_when_no_steps,
           :ec2_key_name => @ec2_key_name,
           :hadoop_version => @hadoop_version,
-          :instance_count => @instance_count,
-          :master_instance_type => @master_instance_type,
-          :slave_instance_type => @slave_instance_type,
+          :instance_groups => jobflow_instance_groups
         }
-      }.tap do |preamble|
-        preamble.merge!(:ec2_subnet_id => @ec2_subnet_id) if @ec2_subnet_id
-      end
+      }
+      preamble.merge!(:ec2_subnet_id => @ec2_subnet_id) if @ec2_subnet_id
+      preamble
     end
 
     def jobflow_steps
@@ -118,6 +147,11 @@ module Elasticity
       steps
     end
 
+    def jobflow_instance_groups
+      groups = [:master, :core, :task].map{|role| @instance_groups[role]}.compact
+      groups.map(&:to_aws_instance_config)
+    end
+
   end
 
 end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = "2.0"
+  VERSION = '2.1'
 end

data/spec/lib/elasticity/custom_jar_step_spec.rb

@@ -6,7 +6,7 @@ describe Elasticity::CustomJarStep do
 
   it { should be_a Elasticity::JobFlowStep }
 
-  its(:name) { should == 'Elasticity Custom Jar Step (jar)' }
+  its(:name) { should == 'Elasticity Custom Jar Step' }
   its(:jar) { should == 'jar' }
   its(:arguments) { should == [] }
   its(:action_on_failure) { should == 'TERMINATE_JOB_FLOW' }
@@ -24,7 +24,7 @@ describe Elasticity::CustomJarStep do
           :hadoop_jar_step => {
             :jar => 'jar'
           },
-          :name => 'Elasticity Custom Jar Step (jar)'
+          :name => 'Elasticity Custom Jar Step'
         }
       end
     end
@@ -43,7 +43,7 @@ describe Elasticity::CustomJarStep do
             :jar => 'jar',
             :args => ['arg1', 'arg2',],
           },
-          :name => 'Elasticity Custom Jar Step (jar)'
+          :name => 'Elasticity Custom Jar Step'
         }
       end
     end

data/spec/lib/elasticity/instance_group_spec.rb

@@ -0,0 +1,140 @@
+describe Elasticity::InstanceGroup do
+
+  its(:bid_price) { should == nil }
+  its(:count) { should == 1 }
+  its(:type) { should == 'm1.small' }
+  its(:market) { should == 'ON_DEMAND' }
+  its(:role) { should == 'CORE' }
+
+  describe '#count=' do
+
+    it 'should set the count' do
+      subject.count = 10
+      subject.count.should == 10
+    end
+
+    context 'when the role is not MASTER' do
+      context 'and the count is <= 0' do
+        it 'should be an error' do
+          subject.role = 'CORE'
+          expect {
+            subject.count = 0
+          }.to raise_error(ArgumentError, 'Instance groups require at least 1 instance (0 requested)')
+        end
+      end
+    end
+
+    context 'when the role is MASTER' do
+      context 'and a count != 1 is attempted' do
+        it 'should be an error' do
+          subject.role = 'MASTER'
+          expect {
+            subject.count = 2
+          }.to raise_error(ArgumentError, 'MASTER instance groups can only have 1 instance (2 requested)')
+        end
+      end
+    end
+
+  end
+
+  describe '#role=' do
+
+    it 'should set the role' do
+      subject.role = 'MASTER'
+      subject.role.should == 'MASTER'
+    end
+
+    context 'when the role is unknown' do
+      it 'should be an error' do
+        expect {
+          subject.role = '_'
+        }.to raise_error(ArgumentError, 'Role must be one of MASTER, CORE or TASK (_ was requested)')
+      end
+    end
+
+    context 'when the role is switching to MASTER' do
+      context 'and the count is != 1' do
+        it 'should set the count to 1' do
+          subject.role = 'CORE'
+          subject.count = 2
+          expect {
+            subject.role = 'MASTER'
+          }.to change { subject.count }.to(1)
+        end
+      end
+    end
+
+  end
+
+  describe '#set_spot_instances' do
+
+    it 'should set the type and price' do
+      subject.set_spot_instances(0.25)
+      subject.market.should == 'SPOT'
+      subject.bid_price.should == 0.25
+    end
+
+    context 'when the price is <= 0' do
+      it 'should be an error' do
+        expect {
+          subject.set_spot_instances(-1)
+        }.to raise_error(ArgumentError, 'The bid price for spot instances should be greater than 0 (-1 requested)')
+      end
+    end
+
+  end
+
+  describe '#set_on_demand_instances' do
+
+    it 'should set the type and price' do
+      subject.set_on_demand_instances
+      subject.market.should == 'ON_DEMAND'
+      subject.bid_price.should == nil
+    end
+
+  end
+
+  describe '#to_aws_instance_config' do
+
+    context 'when an ON_DEMAND group' do
+      let(:on_demand_instance_group) do
+        Elasticity::InstanceGroup.new.tap do |i|
+          i.count = 5
+          i.type = 'c1.medium'
+          i.role = 'CORE'
+          i.set_on_demand_instances
+        end
+      end
+      it 'should generate an AWS config' do
+        on_demand_instance_group.to_aws_instance_config.should == {
+          :market => 'ON_DEMAND',
+          :instance_count => 5,
+          :instance_type => 'c1.medium',
+          :instance_role => 'CORE',
+        }
+      end
+    end
+
+    context 'when a SPOT group' do
+      let(:on_demand_instance_group) do
+        Elasticity::InstanceGroup.new.tap do |i|
+          i.count = 5
+          i.type = 'c1.medium'
+          i.role = 'CORE'
+          i.set_spot_instances(0.25)
+        end
+      end
+      it 'should generate an AWS config' do
+        on_demand_instance_group.to_aws_instance_config.should == {
+          :market => 'SPOT',
+          :bid_price => 0.25,
+          :instance_count => 5,
+          :instance_type => 'c1.medium',
+          :instance_role => 'CORE',
+        }
+      end
+    end
+
+  end
+
+end

data/spec/lib/elasticity/job_flow_integration_spec.rb

@@ -31,9 +31,20 @@ describe 'Elasticity::JobFlow Integration Examples' do
           :keep_job_flow_alive_when_no_steps => false,
           :ec2_key_name => 'default',
           :hadoop_version => '0.20.205',
-          :instance_count => 2,
-          :master_instance_type => 'm1.small',
-          :slave_instance_type => 'm1.small',
+          :instance_groups => [
+            {
+              :instance_count => 1,
+              :instance_role => 'MASTER',
+              :instance_type => 'm1.small',
+              :market => 'ON_DEMAND',
+            },
+              {
+                :instance_count => 1,
+                :instance_role => 'CORE',
+                :instance_type => 'm1.small',
+                :market => 'ON_DEMAND'
+              },
+          ],
         },
         :steps => [
           {
@@ -98,9 +109,20 @@ describe 'Elasticity::JobFlow Integration Examples' do
           :keep_job_flow_alive_when_no_steps => false,
           :ec2_key_name => 'default',
           :hadoop_version => '0.20.205',
-          :instance_count => 8,
-          :master_instance_type => 'm1.small',
-          :slave_instance_type => 'm1.xlarge',
+          :instance_groups => [
+            {
+              :instance_count => 1,
+              :instance_role => 'MASTER',
+              :instance_type => 'm1.small',
+              :market => 'ON_DEMAND',
+            },
+              {
+                :instance_count => 7,
+                :instance_role => 'CORE',
+                :instance_type => 'm1.xlarge',
+                :market => 'ON_DEMAND'
+              },
+          ]
         },
         :steps => [
           {
@@ -169,9 +191,20 @@ describe 'Elasticity::JobFlow Integration Examples' do
           :keep_job_flow_alive_when_no_steps => false,
           :ec2_key_name => 'default',
           :hadoop_version => '0.20.205',
-          :instance_count => 2,
-          :master_instance_type => 'm1.small',
-          :slave_instance_type => 'm1.small',
+          :instance_groups => [
+            {
+              :instance_count => 1,
+              :instance_role => 'MASTER',
+              :instance_type => 'm1.small',
+              :market => 'ON_DEMAND',
+            },
+              {
+                :instance_count => 1,
+                :instance_role => 'CORE',
+                :instance_type => 'm1.small',
+                :market => 'ON_DEMAND'
+              },
+          ]
         },
         :steps => [
           {
@@ -184,7 +217,7 @@ describe 'Elasticity::JobFlow Integration Examples' do
                   's3n://slif_hadoop_test/cloudburst/output/2011-12-09',
               ],
             },
-            :name => 'Elasticity Custom Jar Step (s3n://elasticmapreduce/samples/cloudburst/cloudburst.jar)'
+            :name => 'Elasticity Custom Jar Step'
           }
         ]
       }).and_return('CUSTOM_JAR_JOBFLOW_ID')

data/spec/lib/elasticity/job_flow_spec.rb

@@ -23,18 +23,69 @@ describe Elasticity::JobFlow do
   describe '#instance_count=' do
 
     context 'when set to more than 1' do
+
       it 'should set the number of instances' do
         subject.instance_count = 10
         subject.instance_count.should == 10
       end
+
+      it 'should set the CORE group instance count to COUNT-1 instances' do
+        instance_group = Elasticity::InstanceGroup.new
+        instance_group.count = 4
+        instance_group.role = 'CORE'
+
+        subject.instance_count = 5
+        subject.send(:jobflow_instance_groups).should be_include(instance_group.to_aws_instance_config)
+      end
+
     end
 
     context 'when set to less than 2' do
-      it 'should be an error' do
+
+      it 'should be an error and not set the instance count' do
+        subject.instance_count = 10
         expect {
           subject.instance_count = 1
         }.to raise_error(ArgumentError, 'Instance count cannot be set to less than 2 (requested 1)')
+        subject.instance_count.should == 10
       end
+
+    end
+
+  end
+
+  describe '#master_instance_type=' do
+
+    it 'should set the master_instance_type' do
+      subject.master_instance_type = '_'
+      subject.master_instance_type.should == '_'
+    end
+
+    it 'should set the MASTER group instance type' do
+      instance_group = Elasticity::InstanceGroup.new
+      instance_group.type = 'c1.medium'
+      instance_group.role = 'MASTER'
+
+      subject.master_instance_type = 'c1.medium'
+      subject.send(:jobflow_instance_groups).should be_include(instance_group.to_aws_instance_config)
+    end
+
+  end
+
+  describe '#slave_instance_type=' do
+
+    it 'should set the slave_instance_type' do
+      subject.slave_instance_type = '_'
+      subject.slave_instance_type.should == '_'
+    end
+
+    it 'should set the CORE group instance type' do
+      instance_group = Elasticity::InstanceGroup.new
+      instance_group.type = 'c1.medium'
+      instance_group.role = 'CORE'
+
+      subject.slave_instance_type = 'c1.medium'
+      subject.send(:jobflow_instance_groups).should be_include(instance_group.to_aws_instance_config)
     end
 
   end
@@ -231,6 +282,59 @@ describe Elasticity::JobFlow do
 
   end
 
+  describe '#jobflow_instance_groups' do
+
+    describe 'default instance groups' do
+
+      let(:default_instance_groups) do
+        [
+          {
+            :instance_count => 1,
+            :instance_role => 'MASTER',
+            :instance_type => 'm1.small',
+            :market => 'ON_DEMAND',
+          },
+            {
+              :instance_count => 1,
+              :instance_role => 'CORE',
+              :instance_type => 'm1.small',
+              :market => 'ON_DEMAND'
+            },
+        ]
+      end
+
+      it 'should create a properly specified instance group config' do
+        subject.send(:jobflow_instance_groups).should == default_instance_groups
+      end
+
+    end
+
+    context 'when a task instance group is specified' do
+
+      let(:task_instance_group) do
+        Elasticity::InstanceGroup.new.tap do |i|
+          i.count = 2
+          i.type = 'c1.medium'
+        end
+      end
+
+      let(:task_instance_group_config) do
+        {
+          :instance_count => 2,
+          :instance_role => 'TASK',
+          :instance_type => 'c1.medium',
+          :market => 'ON_DEMAND'
+        }
+      end
+
+      it 'should include it in the group config' do
+        subject.set_task_instance_group(task_instance_group)
+        subject.send(:jobflow_instance_groups).should be_include(task_instance_group_config)
+      end
+
+    end
+  end
+
   describe '#jobflow_preamble' do
 
     let(:basic_preamble) do
@@ -241,13 +345,15 @@ describe Elasticity::JobFlow do
           :keep_job_flow_alive_when_no_steps => false,
           :ec2_key_name => 'default',
           :hadoop_version => '0.20.205',
-          :instance_count => 2,
-          :master_instance_type => 'm1.small',
-          :slave_instance_type => 'm1.small',
+          :instance_groups => ['INSTANCE_GROUP_CONFIGURATION']
         }
       }
     end
 
+    before do
+      subject.stub(:jobflow_instance_groups).and_return(['INSTANCE_GROUP_CONFIGURATION'])
+    end
+
     it 'should create a jobflow configuration section' do
       subject.send(:jobflow_preamble).should == basic_preamble
     end
@@ -353,7 +459,7 @@ describe Elasticity::JobFlow do
   describe '#shutdown' do
 
     context 'when the jobflow has not yet been started' do
-      let(:unstarted_job_flow) { Elasticity::JobFlow.new('_', '_')}
+      let(:unstarted_job_flow) { Elasticity::JobFlow.new('_', '_') }
       it 'should be an error' do
         expect {
           unstarted_job_flow.shutdown

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: elasticity
 version: !ruby/object:Gem::Version
-  version: '2.0'
+  version: '2.1'
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-07-02 00:00:00.000000000 Z
+date: 2012-07-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rest-client
@@ -98,6 +98,7 @@ files:
 - lib/elasticity/emr.rb
 - lib/elasticity/hadoop_bootstrap_action.rb
 - lib/elasticity/hive_step.rb
+- lib/elasticity/instance_group.rb
 - lib/elasticity/job_flow.rb
 - lib/elasticity/job_flow_status.rb
 - lib/elasticity/job_flow_status_step.rb
@@ -110,6 +111,7 @@ files:
 - spec/lib/elasticity/emr_spec.rb
 - spec/lib/elasticity/hadoop_bootstrap_action_spec.rb
 - spec/lib/elasticity/hive_step_spec.rb
+- spec/lib/elasticity/instance_group_spec.rb
 - spec/lib/elasticity/job_flow_integration_spec.rb
 - spec/lib/elasticity/job_flow_spec.rb
 - spec/lib/elasticity/job_flow_status_spec.rb
@@ -133,7 +135,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 4070655408797594419
+      hash: -3013863256678255822
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -142,7 +144,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 4070655408797594419
+      hash: -3013863256678255822
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.24
@@ -155,6 +157,7 @@ test_files:
 - spec/lib/elasticity/emr_spec.rb
 - spec/lib/elasticity/hadoop_bootstrap_action_spec.rb
 - spec/lib/elasticity/hive_step_spec.rb
+- spec/lib/elasticity/instance_group_spec.rb
 - spec/lib/elasticity/job_flow_integration_spec.rb
 - spec/lib/elasticity/job_flow_spec.rb
 - spec/lib/elasticity/job_flow_status_spec.rb