appoxy-aws 1.11.13

data/README.txt

@@ -0,0 +1,166 @@
+= RightScale Amazon Web Services Ruby Gems
+
+Published by RightScale, Inc. under the MIT License.
+For information about RightScale, see http://www.rightscale.com
+
+== DESCRIPTION:
+
+The RightScale AWS gems have been designed to provide a robust, fast, and secure interface to Amazon EC2, EBS, S3, SQS, SDB, and CloudFront. 
+These gems have been used in production by RightScale since late 2006 and are being maintained to track enhancements made by Amazon. 
+The RightScale AWS gems comprise:
+
+- RightAws::Ec2 -- interface to Amazon EC2 (Elastic Compute Cloud) and the
+  associated EBS (Elastic Block Store)
+- RightAws::S3 and RightAws::S3Interface -- interface to Amazon S3 (Simple Storage Service)
+- RightAws::Sqs and RightAws::SqsInterface -- interface to first-generation Amazon SQS (Simple Queue Service) (API version 2007-05-01)
+- RightAws::SqsGen2 and RightAws::SqsGen2Interface -- interface to second-generation Amazon SQS (Simple Queue Service) (API version 2008-01-01)
+- RightAws::SdbInterface and RightAws::ActiveSdb -- interface to Amazon SDB (SimpleDB)
+- RightAws::AcfInterface -- interface to Amazon CloudFront, a content distribution service
+
+== FEATURES:
+
+- Full programmmatic access to EC2, EBS, S3, SQS, SDB, and CloudFront.
+- Complete error handling: all operations check for errors and report complete
+  error information by raising an AwsError.
+- Persistent HTTP connections with robust network-level retry layer using
+  RightHttpConnection).  This includes socket timeouts and retries.
+- Robust HTTP-level retry layer.  Certain (user-adjustable) HTTP errors returned
+  by Amazon's services are classified as temporary errors.
+  These errors are automaticallly retried using exponentially increasing intervals.
+  The number of retries is user-configurable.
+- Fast REXML-based parsing of responses (as fast as a pure Ruby solution allows).
+- Uses libxml (if available) for faster response parsing. 
+- Support for large S3 list operations.  Buckets and key subfolders containing
+  many (> 1000) keys are listed in entirety.  Operations based on list (like
+  bucket clear) work on arbitrary numbers of keys.
+- Support for streaming GETs from S3, and streaming PUTs to S3 if the data source is a file.
+- Support for single-threaded usage, multithreaded usage, as well as usage with multiple
+  AWS accounts.
+- Support for both first- and second-generation SQS (API versions 2007-05-01
+  and 2008-01-01).  These versions of SQS are not compatible.
+- Support for signature versions 0, 1 and 2 on all services.
+- Interoperability with any cloud running Eucalyptus (http://eucalyptus.cs.ucsb.edu)
+- Test suite (requires AWS account to do "live" testing).
+
+== THREADING:
+
+All RightScale AWS interfaces offer two threading options:
+1. Use a single persistent HTTP connection per process.
+2. Use a persistent HTTP connection per Ruby thread.
+ 
+Either way, it doesn't matter how many (for example) RightAws::S3 objects you create,
+they all use the same per-program or per-thread
+connection. The purpose of sharing the connection is to keep a single
+persistent HTTP connection open to avoid paying connection
+overhead on every request. However, if you have multiple concurrent
+threads, you may want or need an HTTP connection per thread to enable
+concurrent requests to AWS. The way this plays out in practice is:
+1. If you have a non-multithreaded Ruby program, use the non-multithreaded setting.
+2. If you have a multi-threaded Ruby program, use the multithreaded setting to enable
+   concurrent requests to S3 (or SQS, or SDB, or EC2).
+3. For running under Mongrel/Rails, use the non-multithreaded setting even though
+   mongrel is multithreaded.  This is because only one Rails handler is invoked at
+   time (i.e. it acts like a single-threaded program)
+
+Note that due to limitations in the I/O of the Ruby interpreter you
+may not get the degree of parallelism you may expect with the multi-threaded setting.
+
+By default, EC2/S3/SQS/SDB/ACF interface instances are created in single-threaded mode.  Set
+"params[:multi_thread]" to "true" in the initialization arguments to use
+multithreaded mode.
+
+== GETTING STARTED:
+
+* For EC2 read RightAws::Ec2 and consult the Amazon EC2 API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=87
+* For S3 read RightAws::S3 and consult the Amazon S3 API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=48
+* For first generation SQS read RightAws::Sqs and consult the Amazon SQS API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=31
+* For second generation SQS read RightAws::SqsGen2, RightAws::SqsGen2Interface, and consult the Amazon SQS API documentation at
+  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1214&categoryID=31 
+
+  Amazon's Migration Guide for moving from first to second generation SQS is
+  avalable at:
+  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1148
+* For SDB read RightAws::SdbInterface, RightAws::ActiveSdb, and consult the Amazon SDB API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=141
+* For CloudFront (ACF) read RightAws::AcfInterface and consult the Amazon CloudFront API documentation at 
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=213
+
+== KNOWN ISSUES:
+
+- 7/08: A user has reported that uploads of large files on Windows may be broken on some
+  Win platforms due to a buggy File.lstat.size.  Use the following monkey-patch at your own risk, 
+  as it has been proven to break Rails 2.0 on Windows:
+
+    require 'win32/file'
+    class File
+      def lstat
+        self.stat
+      end
+    end
+
+
+- Attempting to use the Gibberish plugin (used by the Beast forum app) 
+  will break right_aws as well as lots of other code.  Gibberish
+  changes the semantics of core Ruby (specifically, the String class) and thus presents a reliability
+  problem for most Ruby programs.
+
+- 2/11/08: If you use RightAws in conjunction with attachment_fu, the
+  right_aws gem must be included (using the require statement) AFTER
+  attachment_fu.  If right_aws is loaded before attachment_fu, you'll
+  encounter errors similar to:
+
+  s3.amazonaws.com temporarily unavailable: (wrong number of arguments (5 for 4))
+
+  or
+
+  'incompatible Net::HTTP monkey-patch'
+
+  This is due to a conflict between the right_http_connection gem and another
+  gem required by attachment_fu.  It may be possible to require right_aws (and
+  thus right_http_connection) in the .after_initialize method of the config object in
+  environment.rb (check the docs for Rails::Configuration.after_initialize).
+
+- 8/07: Amazon has changed the semantics of the SQS service.  A
+  new queue may not be created within 60 seconds of the destruction of any
+  older queue with the same name.  Certain methods of RightAws::Sqs and
+  RightAws::SqsInterface will fail with the message:
+  "AWS.SimpleQueueService.QueueDeletedRecently: You must wait 60 seconds after deleting a queue before you can create another with the same name."
+  
+== REQUIREMENTS:
+
+RightAws requires REXML and the right_http_connection gem.
+If libxml and its Ruby bindings (distributed in the libxml-ruby gem) are
+present, RightAws can be configured to use them:
+  RightAws::RightAWSParser.xml_lib = 'libxml'
+Any error with the libxml installation will result in RightAws failing-safe to
+REXML parsing.
+
+== INSTALL:
+
+sudo gem install right_aws
+
+== LICENSE:
+
+Copyright (c) 2007-2009 RightScale, Inc. 
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/test/acf/test_helper.rb

@@ -0,0 +1,2 @@
+require 'test/unit'
+require File.dirname(__FILE__) + '/../../lib/right_aws'

data/test/acf/test_right_acf.rb

@@ -0,0 +1,146 @@
+require File.dirname(__FILE__) + '/test_helper.rb'
+
+class TestAcf < Test::Unit::TestCase
+
+  RIGHT_OBJECT_TEXT     = 'Right test message'
+  
+  STDOUT.sync = true
+
+  def setup
+    @acf= Rightscale::AcfInterface.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
+    @s3 = Rightscale::S3.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key)
+    @bucket_name   = "right-acf-awesome-test-bucket-0001"
+    @bucket_domain = "#{@bucket_name}.s3.amazonaws.com"
+  end
+
+  def test_01_list_distributions_part1
+    distributions = nil
+    assert_nothing_raised(Rightscale::AwsError) do
+      distributions = @acf.list_distributions
+    end
+    assert distributions.is_a?(Array)
+  end
+
+  def test_02_try_to_create_for_bad_bucket
+    # a bucket does not exist
+    assert_raise(Rightscale::AwsError) do
+      @acf.create_distribution("right-cloudfront-awesome-test-bucket-not-exist", "Mustn't to be born", true)
+    end
+    # a bucket is not a domain naming complied guy
+    bucket_name = 'right_cloudfront_awesome_test_bucket_BAD'
+    @s3.bucket(bucket_name, :create)
+    assert_raise(Rightscale::AwsError) do
+      @acf.create_distribution(bucket_name, "Mustn't to be born", true)
+    end
+  end
+
+  def test_03_create
+    comment = 'WooHoo!!!'
+    # create a test bucket
+    @s3.bucket(@bucket_name, :create)
+    # create a distribution
+    distribution = @acf.create_distribution(@bucket_domain, comment, true)
+    assert_equal comment, distribution[:comment]
+    assert       distribution[:cnames].size == 0
+    assert       distribution[:enabled]
+  end
+
+  def test_04_list_distributions_part2
+    distributions = @acf.list_distributions
+    assert distributions.size > 0
+  end
+
+  def get_test_distribution
+    @acf.list_distributions.select{ |d| d[:origin] == @bucket_domain }.first
+  end
+
+  def test_05_get_distribution
+    old = get_test_distribution
+    assert_nothing_raised do
+      @acf.get_distribution(old[:aws_id])
+    end
+  end
+
+  def test_06_get_and_set_config
+    config = nil
+    old = get_test_distribution
+    assert_nothing_raised do
+      config = @acf.get_distribution_config(old[:aws_id])
+    end
+    # change a config
+    config[:enabled] = false
+    config[:cnames] << 'x1.myawesomesite.com'
+    config[:cnames] << 'x2.myawesomesite.com'
+    # set config
+    set_config_result = nil
+    assert_nothing_raised do
+      set_config_result = @acf.set_distribution_config(old[:aws_id], config)
+    end
+    assert set_config_result
+    # reget the config and check
+    new_config = nil
+    assert_nothing_raised do
+      new_config = @acf.get_distribution_config(old[:aws_id])
+    end
+    assert           !new_config[:enabled]
+    assert_equal     new_config[:cnames].sort, ['x1.myawesomesite.com', 'x2.myawesomesite.com']
+    assert_not_equal config[:e_tag], new_config[:e_tag]
+
+    # try to update the old config again (must fail because ETAG has changed)
+    assert_raise(Rightscale::AwsError) do
+      @acf.set_distribution_config(old[:aws_id], config)
+    end
+  end
+
+  def test_07_caching
+    # enable caching
+    @acf.params[:cache] = true
+    # list distributions
+    @acf.list_distributions
+    # list the distributions again - cache should hit
+    assert_raise(Rightscale::AwsNoChange) do
+      @acf.list_distributions
+    end
+    # disable caching
+    @acf.params[:cache] = true
+  end
+
+  def test_08_delete_distribution
+    # we need ETAG so use get_distribution
+    distribution = @acf.get_distribution(get_test_distribution[:aws_id])
+    # try to delete a distribution
+    # should fail because
+    if distribution[:status] == 'InProgress'
+      # should fail because the distribution is not deployed yet
+      assert_raise(Rightscale::AwsError) do
+        @acf.delete_distribution(distribution[:aws_id], distribution[:e_tag])
+      end
+      # wait for a deployed state
+      print "waiting up to 5 min while the distribution is being deployed: "
+      100.times do
+        print '.'
+        distribution = @acf.get_distribution(distribution[:aws_id])
+        if distribution[:status] == 'Deployed'
+          print ' done'
+          break
+        end
+        sleep 3
+      end
+      puts
+    end
+
+    # only disabled and deployed distribution can be deleted
+    assert_equal 'Deployed', distribution[:status]
+    assert       !distribution[:enabled]
+
+    # delete the distribution
+    assert_nothing_raised do
+      @acf.delete_distribution(distribution[:aws_id], distribution[:e_tag])
+    end  
+  end
+
+  def test_09_drop_bucket
+    assert @s3.bucket(@bucket_name).delete
+  end
+
+end

data/test/ec2/test_helper.rb

@@ -0,0 +1,2 @@
+require 'test/unit'
+require File.dirname(__FILE__) + '/../../lib/right_aws'

data/test/ec2/test_right_ec2.rb

@@ -0,0 +1,108 @@
+require File.dirname(__FILE__) + '/test_helper.rb'
+require 'pp'
+
+class TestEc2 < Test::Unit::TestCase
+
+    # Some of RightEc2 instance methods concerning instance launching and image registration
+    # are not tested here due to their potentially risk.
+  
+  def setup
+    @ec2   = Rightscale::Ec2.new(TestCredentials.aws_access_key_id,
+                                 TestCredentials.aws_secret_access_key)
+    @key   = 'right_ec2_awesome_test_key'
+    @group = 'right_ec2_awesome_test_security_group'
+  end
+  
+  def test_01_create_describe_key_pairs
+    new_key = @ec2.create_key_pair(@key)
+    assert new_key[:aws_material][/BEGIN RSA PRIVATE KEY/], "New key material is absent"
+    keys = @ec2.describe_key_pairs
+    assert keys.map{|key| key[:aws_key_name] }.include?(@key), "#{@key} must exist"
+  end
+  
+  def test_02_create_security_group
+    assert @ec2.create_security_group(@group,'My awesone test group'), 'Create_security_group fail'
+    group = @ec2.describe_security_groups([@group])[0]
+    assert_equal @group, group[:aws_group_name], 'Group must be created but does not exist'
+  end
+  
+  def test_03_perms_add
+    assert @ec2.authorize_security_group_named_ingress(@group, TestCredentials.account_number, 'default')
+    assert @ec2.authorize_security_group_IP_ingress(@group, 80,80,'udp','192.168.1.0/8')
+  end
+  
+  def test_04_check_new_perms_exist
+    assert_equal 2, @ec2.describe_security_groups([@group])[0][:aws_perms].size
+  end
+
+  def test_05_perms_remove
+    assert @ec2.revoke_security_group_IP_ingress(@group, 80,80,'udp','192.168.1.0/8')
+    assert @ec2.revoke_security_group_named_ingress(@group,
+                                                    TestCredentials.account_number, 'default')
+  end
+
+  def test_06_describe_images
+    images = @ec2.describe_images
+    assert images.size>0, 'Amazon must have at least some public images'
+      # unknown image
+    assert_raise(Rightscale::AwsError){ @ec2.describe_images(['ami-ABCDEFGH'])}
+  end
+
+  def test_07_describe_instanses
+    assert @ec2.describe_instances
+      # unknown image
+    assert_raise(Rightscale::AwsError){ @ec2.describe_instances(['i-ABCDEFGH'])}
+  end
+
+  def test_08_delete_security_group
+    assert @ec2.delete_security_group(@group), 'Delete_security_group fail'
+  end
+  
+  def test_09_delete_key_pair
+    assert @ec2.delete_key_pair(@key), 'Delete_key_pair fail'
+##  Hmmm... Amazon does not through the exception any more. It now just returns a 'true' if the key does not exist any more...
+##      # key must be deleted already
+##    assert_raise(Rightscale::AwsError) { @ec2.delete_key_pair(@key) }
+  end
+
+  def test_10_signature_version_0
+    ec2 = Rightscale::Ec2.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key, :signature_version => '0')
+    images = ec2.describe_images
+    assert images.size>0, 'Amazon must have at least some public images'
+    # check that the request has correct signature version
+    assert ec2.last_request.path.include?('SignatureVersion=0')
+  end
+
+  def test_11_regions
+    regions = nil
+    assert_nothing_raised do
+      regions = @ec2.describe_regions
+    end
+    # check we got more that 0 regions
+    assert regions.size > 0
+    # check an access to regions
+    regions.each do |region|
+      regional_ec2 = Rightscale::Ec2.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key, :region => region)
+      # do we have a correct endpoint server?
+      assert_equal "#{region}.ec2.amazonaws.com", regional_ec2.params[:server]
+      # get a list of images from every region
+      images = nil
+      assert_nothing_raised do
+        images = regional_ec2.describe_regions
+      end
+      # every region must have images
+      assert images.size > 0
+    end
+  end
+  
+  def test_12_endpoint_url
+    ec2 = Rightscale::Ec2.new(TestCredentials.aws_access_key_id, TestCredentials.aws_secret_access_key, :endpoint_url => 'a://b.c:0/d/', :region => 'z')
+    # :endpoint_url has a priority hence :region should be ommitted
+    assert_equal 'a',   ec2.params[:protocol]
+    assert_equal 'b.c', ec2.params[:server]
+    assert_equal '/d/', ec2.params[:service]
+    assert_equal 0,     ec2.params[:port]
+    assert_nil          ec2.params[:region]
+  end
+
+end

data/test/http_connection.rb

@@ -0,0 +1,87 @@
+=begin
+Copyright (c) 2007 RightScale, Inc. 
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+=end
+
+# Stub extension/redefinition of RightHttpConnection for testing purposes.
+require 'net/http'
+require 'rubygems'
+require 'right_http_connection'
+
+module Net
+  class HTTPResponse
+  alias_method :real_body, :body
+    def setmsg(msg)
+      @mymsg = msg
+    end
+
+    def body
+      # defined?() helps us to get rid of a bunch of 'warnings'
+      (defined?(@mymsg) && @mymsg) ? @mymsg : real_body
+    end
+  end
+end
+
+module Rightscale
+
+  class HttpConnection
+    @@response_stack = []
+
+    alias_method :real_request, :request
+
+    def request(request_params, &block)
+      if(@@response_stack.length == 0)
+        return real_request(request_params, &block)
+      end
+
+      if(block)
+        # Do something special
+      else
+        next_response = HttpConnection::pop() 
+        classname = Net::HTTPResponse::CODE_TO_OBJ["#{next_response[:code]}"]
+        response = classname.new("1.1", next_response[:code], next_response[:msg])
+        if(next_response[:msg])
+          response.setmsg(next_response[:msg])
+        end
+        response
+      end
+    end
+
+    def self.reset
+      @@response_stack = []
+    end
+
+    def self.push(code, msg=nil)
+      response = {:code => code, :msg => msg}
+      @@response_stack << response
+    end
+
+    def self.pop
+      @@response_stack.pop
+    end
+
+    def self.length
+      @@response_stack.length
+    end
+
+  end
+
+end