right_aws 2.1.0 → 3.0.0

data/History.txt

@@ -4,7 +4,7 @@ Initial release.
 == 1.2.0 2007-09-12
 
 * r1718, todd, 2007-09-12 15:34:37
-  * # 458, Extensive documentation review, rework, and expansion.  Also added 
+  * # 458, Extensive documentation review, rework, and expansion.  Also added
     coverage analysis to the test suite using RCov.
 
 * r1690, todd, 2007-09-07 15:23:11
@@ -39,7 +39,7 @@ Initial release.
   * # 487, # 488, Consolidate a lot of code that was repeated in three places.
     Fix error handling path when using streaming GET interfaces with S3.
     Also add a stub for RightHttpConnection which allows more control over the
-    unit tests.  Expand the unit tests and coverage tests. 
+    unit tests.  Expand the unit tests and coverage tests.
 
 * r1755, todd, 2007-09-19 14:29:19
   * # 487, RDoc fixes after code consolidation
@@ -59,13 +59,13 @@ Initial release.
   * # 524, blocks added to ec2_describe_xxx to support aws_cache
 
 * r1924, konstantin, 2007-10-12 11:35:06
-  * # 536, user_data bug fix 
+  * # 536, user_data bug fix
 
 * r1929, tve,  2007-10-15 00:00:11
   * Fix libxml/rexml selection bug
 
 * r1938, konstantin, 2007-10-16 10:53:56
-  * instance type support is set to default 
+  * instance type support is set to default
 
 == 1.4.3 2007-10-25
 
@@ -82,11 +82,11 @@ Initial release.
 * r2001, konstantin, 2007-11-01 12:03:13  +0300
   * Fixed multiple permissions assignment on Grantee#grant/revoke
   * Fixed new grantee permissions set ingnore (Grantee#apply)
-  * S3::Grantee#exists? method added 
+  * S3::Grantee#exists? method added
 
 * r2109, konstantin, 2007-11-12 21:49:36  +0300
   * RightAwsBaseInterface: caching implemented.
-    (The Ec2 functions are being cached: describe_images, describe_instances, 
+    (The Ec2 functions are being cached: describe_images, describe_instances,
     describe_security_groups and describe_key_pairs)
 
 == 1.4.5 - 1.4.6
@@ -107,7 +107,7 @@ Initial release.
 == 1.5.0
 * r 2688, konstantin, 02-30-08 15:42:00  +0300
   * SDB support added.
-  * RightAws::S3::bucket and RightAws::S3::Bucket.create methods behaviour 
+  * RightAws::S3::bucket and RightAws::S3::Bucket.create methods behaviour
     changed: param +create+ is set to +false+ by default.
 
 == 1.6.0
@@ -151,7 +151,7 @@ Initial release.
 == 1.7.2
 
   Release Notes:
-  
+
   RightAws includes some new features, including:
   - Support in RightAws::S3 and RightAws::S3Interface for S3 key copy, move, and rename
   - Support for signature version 0 request authentication to EC2, SQS, and SDB
@@ -165,7 +165,7 @@ Initial release.
     We now reset the seek pointer of the streaming IO object to its initial position.
   - Removal of an accidental dependency on ActiveSupport in RightAws::S3Interface.get_link().
   - Monkey-patch of the Ruby File class on Windows platforms to correct a problem in lstat.
-    The lstat bug was causing failure of very large file uploads on Windows [ Contributed by Benjamin Allfree ] 
+    The lstat bug was causing failure of very large file uploads on Windows [ Contributed by Benjamin Allfree ]
   - Fixed parsing of the ETag field for S3 objects
 
 == 1.7.3
@@ -174,7 +174,7 @@ Initial release.
 
   - Removed the 1.7.2 monkey-patch of the Ruby File class on Windows.  This patch broke Rails 2.0.
     The patch is now included in the README for anyone to use at their own risk.
- 
+
 == 1.8.0
 
   Release Notes:
@@ -186,7 +186,7 @@ Initial release.
 
   Bug fixes include correction of RightAws::S3 copy's failure to url-encode
 the source key.
- 
+
 == 1.8.1
 
   Release Notes:
@@ -197,7 +197,7 @@ the source key.
     result sorting and attributes auto loading
   - RightAws::ActiveSdb::Base#find_all_by_ and find_by_ helpers improved to support
     :order, :auto_load, :limit and :next_token options
-  - RightAws::SdbInterface#delete_attributes bug fixed 
+  - RightAws::SdbInterface#delete_attributes bug fixed
   - SdbInterface allows specification of a string value to use for
     representing Ruby nil in SDB.
   - Sdb tests fixed and improved
@@ -215,7 +215,7 @@ the source key.
    - Ec::cancel_bundle_task
 
   - Full Amazon CloudFront support added with RightAws::AcfInterface
-  - Bug fixes to S3Interface::store_object_and_verify and 
+  - Bug fixes to S3Interface::store_object_and_verify and
     S3Interface::retrieve_object_and_verify (thanks to numerous user reports)
   - Updates to caching for Ec2::describe_images_by methods
   - Ec2 now has Ec2::last_request_id
@@ -228,7 +228,7 @@ the source key.
   - CloudFront: docs fixes
   - SDB: added: SQL-like query, select and query_with_attributes support
   - SDB: fixed no method error when searching for id that doesn't exist
-  
+
 === 1.11.0
 
   Release Notes:
@@ -303,3 +303,27 @@ the source key.
     - SDB: uuid gem dependency
   - this gem requires right_http_connection 0bc3343232133bdb38c237d8285525d74495d3f5 or later
   - "Raise On Timeout On Action" feature added to avoid duplicate resources creation if a timeout error occures and a retry is performed
+
+=== 3.0.0
+  Release Notes:
+  - Fixed/Added:
+    - ClientToken (launch_instances, run_instances) is not used for Eucalyptus clouds
+    - VPC2, stage1. Next methods were updated:
+      - associate_address, modify_security_group, create_security_group, create_vpc, delete_security_group,
+        describe_addresses, describe_images, describe_instance_attribute, describe_regions, describe_reserved_instances_offerings,
+        describe_security_groups, describe_snapshots, describe_spot_instance_requests, describe_spot_price_history,
+        describe_vpcs, disassociate_address, modify_image_attribute, modify_snapshot_attribute, release_address,
+        request_spot_instances, stop_instances
+    - EC2: ClientToken (launch_instances, run_instances) is not used for Eucalyptus clouds
+    - RDS:
+      - RDS: API 2011-04-01
+      - Make :instance_class param more consistent: :db_instance_class --> :instance_class
+      - Issue 53: regression in latest master version of right_rds_interface
+      - Issue 73: Can't get list of instances with RdsInterface
+    - EBS: Issue 54: regression in right_ec2_ebs.rb
+    - Add the port number with server name in the v2 signature string if it is not the RFC standard number (author: unakatsuo).
+    - EMR (Elastic Map Reduce) support
+    - SNS (Simple Notification Service) support
+    - SDB: ConsistentRead support
+    - bunch of micro bugs
+

data/Manifest.txt

@@ -13,6 +13,7 @@ lib/ec2/right_ec2_spot_instances.rb
 lib/ec2/right_ec2_ebs.rb
 lib/ec2/right_ec2_reserved_instances.rb
 lib/ec2/right_ec2_vpc.rb
+lib/ec2/right_ec2_vpc2.rb
 lib/ec2/right_ec2_monitoring.rb
 lib/ec2/right_ec2_placement_groups.rb
 lib/ec2/right_ec2_windows_mobility.rb

data/Rakefile

@@ -2,7 +2,6 @@
 
 require 'rubygems'
 require "rake/testtask"
-require 'rcov/rcovtask'
 require 'rake/gempackagetask'
 require 'rake/clean'
 $: << File.dirname(__FILE__)
@@ -13,7 +12,19 @@ begin
 rescue LoadError => e
   STDERR.puts("Bundler is not available, some rake tasks will not be defined: #{e.message}")
 else
-  Bundler::GemHelper.install_tasks
+  Bundler::GemHelper.install_tasks :name => 'right_aws'
+end
+
+begin
+  require 'rcov/rcovtask'
+rescue LoadError => e
+  STDERR.puts("RCov is not available, some rake tasks will not be defined: #{e.message}")
+else
+  desc "Analyze code coverage of the unit tests."
+  Rcov::RcovTask.new do |t|
+    t.test_files = FileList[testglobs]
+    #t.verbose = true     # uncomment to see the executed command
+  end
 end
 
 # == Gem == #
@@ -28,12 +39,6 @@ directory gemtask.package_dir
 
 CLEAN.include(gemtask.package_dir)
 
-desc "Analyze code coverage of the unit tests."
-Rcov::RcovTask.new do |t|
-  t.test_files = FileList[testglobs]
-  #t.verbose = true     # uncomment to see the executed command
-end
-
 desc "Test just the SQS interface"
 task :testsqs do
   require 'test/test_credentials'
@@ -101,4 +106,25 @@ task :testrds do
   require 'test/rds/test_right_rds.rb'
 end
 
+desc "Test just the SNS interface"
+task :testsns do
+  require 'test/test_credentials'
+  TestCredentials.get_credentials
+  require 'test/sns/test_right_sns.rb'
+end
+
+desc "Test Route 53 interface"
+task :testroute53 do
+  require 'test/test_credentials'
+  TestCredentials.get_credentials
+  require 'test/route_53/test_right_route_53'
+end
+
+desc "Test ELB interface"
+task :testelb do
+  require 'test/test_credentials'
+  TestCredentials.get_credentials
+  require 'test/elb/test_right_elb'
+end
+
 # vim: syntax=Ruby

data/lib/awsbase/right_awsbase.rb

@@ -24,7 +24,6 @@
 # Test
 module RightAws
   require 'digest/md5'
-  require 'pp'
   
   class AwsUtils #:nodoc:
     @@digest1   = OpenSSL::Digest::Digest.new("sha1")
@@ -274,20 +273,26 @@ module RightAws
       @aws_secret_access_key = aws_secret_access_key
       # if the endpoint was explicitly defined - then use it
       if @params[:endpoint_url]
-        @params[:server]   = URI.parse(@params[:endpoint_url]).host
-        @params[:port]     = URI.parse(@params[:endpoint_url]).port
-        @params[:service]  = URI.parse(@params[:endpoint_url]).path
+        uri = URI.parse(@params[:endpoint_url])
+        @params[:server]   = uri.host
+        @params[:port]     = uri.port
+        @params[:service]  = uri.path
+        @params[:protocol] = uri.scheme
         # make sure the 'service' path is not empty
         @params[:service]  = service_info[:default_service] if @params[:service].right_blank?
-        @params[:protocol] = URI.parse(@params[:endpoint_url]).scheme
         @params[:region]   = nil
+        default_port       = uri.default_port
       else
         @params[:server]   ||= service_info[:default_host]
         @params[:server]     = "#{@params[:region]}.#{@params[:server]}" if @params[:region]
         @params[:port]     ||= service_info[:default_port]
         @params[:service]  ||= service_info[:default_service]
         @params[:protocol] ||= service_info[:default_protocol]
+        default_port         = @params[:protocol] == 'https' ? 443 : 80
       end
+      # build a host name to sign
+      @params[:host_to_sign]  = @params[:server].dup
+      @params[:host_to_sign] << ":#{@params[:port]}" unless default_port == @params[:port].to_i
       # a set of options to be passed to RightHttpConnection object
       @params[:connection_options] = {} unless @params[:connection_options].is_a?(Hash) 
       @with_connection_options = {}
@@ -331,7 +336,8 @@ module RightAws
         # get rid of requestId (this bad boy was added for API 2008-08-08+ and it is uniq for every response)
         # feb 04, 2009 (load balancer uses 'RequestId' hence use 'i' modifier to hit it also)
         response = response.sub(%r{<requestId>.+?</requestId>}i, '')
-        response_md5 = MD5.md5(response).to_s
+        # this should work for both ruby 1.8.x and 1.9.x
+        response_md5 = Digest::MD5::new.update(response).to_s
         # check for changes
         unless @cache[function] && @cache[function][:response_md5] == response_md5
           # well, the response is new, reset cache data
@@ -416,7 +422,7 @@ module RightAws
 
     # ACF, AMS, EC2, LBS and SDB uses this guy
     # SQS and S3 use their own methods
-    def generate_request_impl(verb, action, options={}) #:nodoc:
+    def generate_request_impl(verb, action, options={}, custom_options={}) #:nodoc:
       # Form a valid http verb: 'GET' or 'POST' (all the other are not supported now)
       http_verb = verb.to_s.upcase
       # remove empty keys from request options
@@ -424,16 +430,16 @@ module RightAws
       # prepare service data
       service_hash = {"Action"         => action,
                       "AWSAccessKeyId" => @aws_access_key_id,
-                      "Version"        => @params[:api_version] }
+                      "Version"        => custom_options[:api_version] || @params[:api_version] }
       service_hash.merge!(options)
       # Sign request options
-      service_params = signed_service_params(@aws_secret_access_key, service_hash, http_verb, @params[:server], @params[:service])
+      service_params = signed_service_params(@aws_secret_access_key, service_hash, http_verb, @params[:host_to_sign], @params[:service])
       # Use POST if the length of the query string is too large
       # see http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/MakingRESTRequests.html
       if http_verb != 'POST' && service_params.size > 2000
         http_verb = 'POST'
         if signature_version == '2'
-          service_params = signed_service_params(@aws_secret_access_key, service_hash, http_verb, @params[:server], @params[:service])
+          service_params = signed_service_params(@aws_secret_access_key, service_hash, http_verb, @params[:host_to_sign], @params[:service])
         end
       end
       # create a request
@@ -614,10 +620,11 @@ module RightAws
     #     "Filter.2.Value.2"=>"bb"}
     def amazonize_list(masks, list, options={}) #:nodoc:
       groups = {}
-      Array(list).each_with_index do |list_item, i|
+      list_idx = options[:index] || 1
+      Array(list).each do |list_item|
         Array(masks).each_with_index do |mask, mask_idx|
           key = mask[/\?/] ? mask.dup : mask.dup + '.?'
-          key.sub!('?', (i+1).to_s)
+          key.sub!('?', list_idx.to_s)
           value = Array(list_item)[mask_idx]
           if value.is_a?(Array)
             groups.merge!(amazonize_list(key, value, options))
@@ -633,6 +640,7 @@ module RightAws
             groups[key] = value
           end
         end
+        list_idx += 1
       end
       groups
     end
@@ -662,6 +670,162 @@ module RightAws
       result
     end
 
+    # Build API request keys set.
+    #
+    # Options is a hash, expectations is a set of keys [and rules] how to represent options.
+    # Mappings is an Array (may include hashes) or a Hash.
+    #
+    #  Example:
+    #
+    #  options = { :valid_from              => Time.now - 10,
+    #              :instance_count          => 3,
+    #              :image_id                => 'ami-08f41161',
+    #              :spot_price              => 0.059,
+    #              :instance_type           => 'c1.medium',
+    #              :instance_count          => 1,
+    #              :key_name                => 'tim',
+    #              :availability_zone       => 'us-east-1a',
+    #              :monitoring_enabled      => true,
+    #              :launch_group            => 'lg1',
+    #              :availability_zone_group => 'azg1',
+    #              :groups                  => ['a', 'b', 'c'],
+    #              :group_ids               => 'sg-1',
+    #              :user_data               => 'konstantin',
+    #              :block_device_mappings   => [ { :device_name     => '/dev/sdk',
+    #                                              :ebs_snapshot_id => 'snap-145cbc7d',
+    #                                              :ebs_delete_on_termination => true,
+    #                                              :ebs_volume_size => 3,
+    #                                              :virtual_name => 'ephemeral2' }]}
+    #  mappings = { :spot_price,
+    #               :availability_zone_group,
+    #               :launch_group,
+    #               :type,
+    #               :instance_count,
+    #               :image_id              => 'LaunchSpecification.ImageId',
+    #               :instance_type         => 'LaunchSpecification.InstanceType',
+    #               :key_name              => 'LaunchSpecification.KeyName',
+    #               :addressing_type       => 'LaunchSpecification.AddressingType',
+    #               :kernel_id             => 'LaunchSpecification.KernelId',
+    #               :ramdisk_id            => 'LaunchSpecification.RamdiskId',
+    #               :subnet_id             => 'LaunchSpecification.SubnetId',
+    #               :availability_zone     => 'LaunchSpecification.Placement.AvailabilityZone',
+    #               :monitoring_enabled    => 'LaunchSpecification.Monitoring.Enabled',
+    #               :valid_from            => { :value => Proc.new { !options[:valid_from].right_blank?  && AwsUtils::utc_iso8601(options[:valid_from]) }},
+    #               :valid_until           => { :value => Proc.new { !options[:valid_until].right_blank? && AwsUtils::utc_iso8601(options[:valid_until]) }},
+    #               :user_data             => { :name  => 'LaunchSpecification.UserData',
+    #                                           :value => Proc.new { !options[:user_data].right_blank? && Base64.encode64(options[:user_data]).delete("\n") }},
+    #               :groups                => { :amazonize_list => 'LaunchSpecification.SecurityGroup'},
+    #               :group_ids             => { :amazonize_list => 'LaunchSpecification.SecurityGroupId'},
+    #               :block_device_mappings => { :amazonize_bdm  => 'LaunchSpecification.BlockDeviceMapping'})
+    #
+    #  map_api_keys_and_values( options, mappings) #=>
+    #    {"LaunchSpecification.BlockDeviceMapping.1.Ebs.DeleteOnTermination" => true,
+    #     "LaunchSpecification.BlockDeviceMapping.1.VirtualName"             => "ephemeral2",
+    #     "LaunchSpecification.BlockDeviceMapping.1.Ebs.VolumeSize"          => 3,
+    #     "LaunchSpecification.BlockDeviceMapping.1.Ebs.SnapshotId"          => "snap-145cbc7d",
+    #     "LaunchSpecification.BlockDeviceMapping.1.DeviceName"              => "/dev/sdk",
+    #     "LaunchSpecification.SecurityGroupId.1"                            => "sg-1",
+    #     "LaunchSpecification.InstanceType"                                 => "c1.medium",
+    #     "LaunchSpecification.KeyName"                                      => "tim",
+    #     "LaunchSpecification.ImageId"                                      => "ami-08f41161",
+    #     "LaunchSpecification.SecurityGroup.1"                              => "a",
+    #     "LaunchSpecification.SecurityGroup.2"                              => "b",
+    #     "LaunchSpecification.SecurityGroup.3"                              => "c",
+    #     "LaunchSpecification.Placement.AvailabilityZone"                   => "us-east-1a",
+    #     "LaunchSpecification.Monitoring.Enabled"                           => true,
+    #     "LaunchGroup"                                                      => "lg1",
+    #     "InstanceCount"                                                    => 1,
+    #     "SpotPrice"                                                        => 0.059,
+    #     "AvailabilityZoneGroup"                                            => "azg1",
+    #     "ValidFrom"                                                        => "2011-06-30T08:06:30.000Z",
+    #     "LaunchSpecification.UserData"                                     => "a29uc3RhbnRpbg=="}
+    #
+    def map_api_keys_and_values(options, *mappings) # :nodoc:
+      result = {}
+      vars   = {}
+      # Fix inputs and make them all to be hashes
+      mappings.flatten.each do |mapping|
+        unless mapping.is_a?(Hash)
+          # mapping is just a :key_name
+          mapping = { mapping => { :name  => mapping.to_s.right_camelize, :value => options[mapping] }}
+        else
+          mapping.each do |local_key, api_opts|
+            unless api_opts.is_a?(Hash)
+              # mapping is a { :key_name => 'ApiKeyName' }
+              mapping[local_key] = { :name  => api_opts.to_s, :value => options[local_key]}
+            else
+              # mapping is a { :key_name => { :name => 'ApiKeyName', :value => 'Value', ... etc}  }
+              api_opts[:name]  = local_key.to_s.right_camelize if (api_opts.keys & [:name, :amazonize_list, :amazonize_bdm]).right_blank?
+              api_opts[:value] = options[local_key] unless api_opts.has_key?(:value)
+            end
+          end
+        end
+        vars.merge! mapping
+      end
+      # Build API keys set
+      #  vars now is a Hash:
+      #    { :key1 => { :name           => 'ApiKey1',   :value => 'BlahBlah'},
+      #      :key2 => { :amazonize_list => 'ApiKey2.?', :value => [1, ...] },
+      #      :key3 => { :amazonize_bdm  => 'BDM',       :value => [{..}, ...] }, ... }
+      #
+      vars.each do |local_key, api_opts|
+        if api_opts[:amazonize_list]
+          result.merge!(amazonize_list( api_opts[:amazonize_list], api_opts[:value] )) unless api_opts[:value].right_blank?
+        elsif api_opts[:amazonize_bdm]
+          result.merge!(amazonize_block_device_mappings( api_opts[:value], api_opts[:amazonize_bdm] )) unless api_opts[:value].right_blank?
+        else
+          api_key = api_opts[:name]
+          value   = api_opts[:value]
+          value   = value.call if value.is_a?(Proc)
+          next if value.right_blank?
+          result[api_key] = value
+        end
+      end
+      #
+      result
+    end
+
+    # Transform a hash of parameters into a hash suitable for sending
+    # to Amazon using a key mapping.
+    #
+    #  amazonize_hash_with_key_mapping('Group.Filter',
+    #    {:some_param => 'SomeParam'},
+    #    {:some_param => 'value'}) #=> {'Group.Filter.SomeParam' => 'value'}
+    #
+    def amazonize_hash_with_key_mapping(key, mapping, hash, options={})
+      result = {}
+      unless hash.right_blank?
+        mapping.each do |local_name, remote_name|
+          value = hash[local_name]
+          next if value.nil?
+          result["#{key}.#{remote_name}"] = value
+        end
+      end
+      result
+    end
+
+    # Transform a list of hashes of parameters into a hash suitable for sending
+    # to Amazon using a key mapping.
+    #
+    #  amazonize_list_with_key_mapping('Group.Filter',
+    #    [{:some_param => 'SomeParam'}, {:some_param => 'SomeParam'}],
+    #    {:some_param => 'value'}) #=>
+    #      {'Group.Filter.1.SomeParam' => 'value',
+    #       'Group.Filter.2.SomeParam' => 'value'}
+    #
+    def amazonize_list_with_key_mapping(key, mapping, list, options={})
+      result = {}
+      unless list.right_blank?
+        list.each_with_index do |item, index|
+          mapping.each do |local_name, remote_name|
+            value = item[local_name]
+            next if value.nil?
+            result["#{key}.#{index+1}.#{remote_name}"] = value
+          end
+        end
+      end
+    end
+    
     # Execute a block of code with custom set of settings for right_http_connection.
     # Accepts next options (see Rightscale::HttpConnection for explanation):
     #  :raise_on_timeout