right_aws 1.1.0 → 1.2.0

data/History.txt

@@ -1,3 +1,50 @@
 == 1.1.0 / 2007-08-10
 Initial release.
 
+== 1.2.0 / 2007-09-12
+------------------------------------------------------------------------
+r1718 | todd | 2007-09-12 15:34:37 -0700 (Wed, 12 Sep 2007) | 3 lines
+
+(#458) Extensive documentation review, rework, and expansion.  Also added
+coverage analysis to the test suite using RCov.
+
+------------------------------------------------------------------------
+r1690 | todd | 2007-09-07 15:23:11 -0700 (Fri, 07 Sep 2007) | 2 lines
+
+(#447) Add support.rb to manifest
+
+------------------------------------------------------------------------
+r1688 | todd | 2007-09-07 13:57:39 -0700 (Fri, 07 Sep 2007) | 4 lines
+
+#(447)  Use Active Support if available, but don't require it.  Load our own
+extensions if it's not present.  This keeps us from overloading ActiveSupport if
+a user's already using it.
+
+------------------------------------------------------------------------
+r1687 | todd | 2007-09-07 11:36:43 -0700 (Fri, 07 Sep 2007) | 2 lines
+
+(#447) Removed dependency on activesupport
+
+------------------------------------------------------------------------
+r1676 | konstantin | 2007-09-06 01:27:09 -0700 (Thu, 06 Sep 2007) | 1 line
+
+paid AMIs, small fix
+------------------------------------------------------------------------
+r1667 | konstantin | 2007-09-05 12:58:10 -0700 (Wed, 05 Sep 2007) | 1 line
+
+#427, Gems,  paid AMI support for ec2_instances
+------------------------------------------------------------------------
+r1658 | konstantin | 2007-09-05 01:02:25 -0700 (Wed, 05 Sep 2007) | 1 line
+
+RightGems: a couple of params improvements, now ones can use a String as well as an Array.
+------------------------------------------------------------------------
+r1653 | konstantin | 2007-09-04 12:31:19 -0700 (Tue, 04 Sep 2007) | 1 line
+
+Right gems: libxml and paid AMI support added
+------------------------------------------------------------------------
+r1581 | tve | 2007-08-24 16:21:45 -0700 (Fri, 24 Aug 2007) | 1 line
+
+Improved RightAws documentation
+------------------------------------------------------------------------
+
+

data/Manifest.txt

@@ -4,6 +4,7 @@ README.txt
 Rakefile
 lib/right_aws.rb
 lib/awsbase/benchmark_fix.rb
+lib/awsbase/support.rb
 lib/awsbase/right_awsbase.rb
 lib/ec2/right_ec2.rb
 lib/s3/right_s3.rb

data/README.txt

@@ -1,56 +1,74 @@
-RightAws
-    by RightScale, Inc.
-    www.RightScale.com 
+= RightScale Amazon Web Services Ruby Gems
 
-== DESCRIPTION:
-
-RightAws::Ec2 is a Ruby library for the Amazon EC2 (Elastic Compute Cloud)
-service.
+Published by RightScale, Inc. under the MIT License.
+For information about RightScale, see http://www.rightscale.com
 
-RightAws::S3 and RightAws::S3Interface are Ruby libraries for the Amazon S3
-(Simple Storage Service) service.
-
-RightAws::Sqs and RightAws::SqsInterface is a Ruby library for the Amazon SQS (Simple Queue Service)
-service.
+== DESCRIPTION:
 
-  
-All RightAws interfaces work in one of two ways:
-1) They use a single persistent HTTP connection per
-process or 2) per Ruby thread. Foir example, it doesn't matter how many RightAws::S3
-objects you create, they all use the same per-program or per-thread
+The RightScale AWS gems have been designed to provide a robust, fast, and secure interface to Amazon EC2, Amazon, S3, and Amazon SQS. These gems have been used in production by RightScale since late 2006 and are being maintained to track enhancements made by Amazon. The RightScale AWS gems comprise:
+
+- RightAws::Ec2 -- interface to Amazon EC2 (Elastic Compute Cloud)
+- RightAws::S3 and RightAws::S3Interface -- interface to Amazon S3 (Simple Storage Service)
+- RightAws::Sqs and RightAws::SqsInterface -- interface to Amazon SQS (Simple Queue Service)
+
+== FEATURES:
+
+- Full programmmatic access to EC2, S3, and SQS.
+- Complete error handling: all operations check for errors and report complete
+  error information by raising an AwsError.
+- Persistent HTTP connections with robust network-level retry layer using
+  RightHttpConnection).  This includes socket timeouts and retries.
+- Robust HTTP-level retry layer.  Certain (user-adjustable) HTTP errors returned
+  by Amazon's services are classified as temporary errors.
+  These errors are automaticallly retried using exponentially increasing intervals.
+  The number of retries is user-configurable.
+- Fast REXML-based parsing of responses (as fast as a pure Ruby solution allows).
+- Uses libxml (if available) for faster response parsing. 
+- Support for large S3 list operations.  Buckets and key subfolders containing
+  many (> 1000) keys are listed in entirety.  Operations based on list (like
+  bucket clear) work on arbitrary numbers of keys.
+- Support for streaming GETs from S3, and streaming PUTs to S3 if the data source is a file.
+- Support for single-threaded usage, multithreaded usage, as well as usage with multiple
+  AWS accounts.
+- Test suite (requires AWS account to do "live" testing).
+
+== THREADING:
+
+All RightScale AWS interfaces offer two threading options:
+1. Use a single persistent HTTP connection per process.
+2. Use a persistent HTTP connection per Ruby thread.
+ 
+Either way, it doesn't matter how many (for example) RightAws::S3 objects you create,
+they all use the same per-program or per-thread
 connection. The purpose of sharing the connection is to keep a single
 persistent HTTP connection open to avoid paying connection
 overhead on every request. However, if you have multiple concurrent
 threads, you may want or need an HTTP connection per thread to enable
-concurrent requests to S3. The way this plays out in practice is:
-1) If you have a non-multithreaded Ruby program, use the non-multithreaded setting for Gem.
-2) If you have a multi-threaded Ruby program, use the multithreaded setting to enable
-concurrent requests to S3 (SQS, EC2).
-3) For running under Mongrel/Rails, use thhe non-multithreaded setting for Gem even though
-Mongrel is multithreaded.  This is because only one Rails handler is invoked at
-any time (i.e. it acts like a single-threaded program)
-
-By default, Ec2/S3/Sqs interface instances are created in single-threaded mode.  Set
+concurrent requests to AWS. The way this plays out in practice is:
+1. If you have a non-multithreaded Ruby program, use the non-multithreaded setting.
+2. If you have a multi-threaded Ruby program, use the multithreaded setting to enable
+   concurrent requests to S3 (or SQS, or EC2).
+3. For running under Mongrel/Rails, use the non-multithreaded setting even though
+   mongrel is multithreaded.  This is because only one Rails handler is invoked at
+   time (i.e. it acts like a single-threaded program)
+
+Note that due to limitations in the I/O of the Ruby interpreter you
+may not get the degree of parallelism you may expect with the multi-threaded setting.
+
+By default, EC2/S3/SQS interface instances are created in single-threaded mode.  Set
 "params[:multi_thread]" to "true" in the initialization arguments to use
 multithreaded mode.
 
-== FEATURES/PROBLEMS:
+== GETTING STARTED:
 
-- Full programmmatic access to Ec2, S3, and Sqs
-- Robust network-level retry layer (using Rightscale::HttpConnection).  This includes
-  socket connect and read timeouts and retries.
-- Robust HTTP-level retry layer.  Certain (user-adjustable) HTTP errors are
-  classified as temporary errors.  These errors are automaticallly retried
-  over exponentially increasing intervals.  The number of retries is
-  user-configurable.
-- Support for large S3 list operations.  Buckets and key subfolders containing
-  many (> 1000) keys are listed in entirety.  Operations based on list (like
-  bucket clear) work on arbitrary numbers of keys.
-- Support for streaming PUTs to S3 if the data source is a file.
-- Support for streaming GETs from S3.
-- Interfaces for HTML link generation.
+* For EC2 read RightAws::Ec2 and consult the Amazon EC2 API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=87
+* For S3 read RightAws::S3 and consult the Amazon S3 API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=48
+* For SQS read RightAws::Sqs and consult the Amazon SQS API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=31
 
-Known Problems:
+== KNOWN ISSUES:
 
 - Amazon recently (8/07) changed the semantics of the SQS service.  A
   new queue may not be created within 60 seconds of the destruction of any
@@ -58,15 +76,14 @@ Known Problems:
   RightAws::SqsInterface will fail with the message:
   "AWS.SimpleQueueService.QueueDeletedRecently: You must wait 60 seconds after deleting a queue before you can create another with the same name."
   
-== SYNOPSIS:
-
-
-
-
-
 == REQUIREMENTS:
 
-RightAws requires activesupport and RightScale's right_http_connection gem.
+RightAws requires REXML and the RightHttpConnection gem.
+If libxml and its Ruby bindings (distributed in the libxml-ruby gem) are
+present, RightAws can be configured to use them:
+  RightAws::RightAWSParser.xml_lib = 'libxml'
+Any error with the libxml installation will result in RightAws failing-safe to
+REXML parsing.
 
 == INSTALL:
 

data/Rakefile

@@ -2,7 +2,10 @@
 
 require 'rubygems'
 require 'hoe'
-require './lib/right_aws.rb'
+require "rake/testtask"
+require 'rcov/rcovtask'
+$: << File.dirname(__FILE__)
+require 'lib/right_aws.rb'
 
 Hoe.new('right_aws', RightAws::VERSION::STRING) do |p|
   p.rubyforge_name = 'rightaws'
@@ -13,13 +16,15 @@ Hoe.new('right_aws', RightAws::VERSION::STRING) do |p|
   p.url = p.paragraphs_of('README.txt', 0).first.split(/\n/)[1..-1]
   p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
   p.remote_rdoc_dir = "/right_aws_gem_doc"
-  p.extra_deps = [['activesupport',       '>= 1.4.1'],
-                  ['right_http_connection','>= 0.1.4']]
+  p.extra_deps = [['right_http_connection','>= 0.1.4']]
+  p.test_globs = ['test/ts_right_aws.rb']
 end
 
-task :test do
-  puts "Hi There"
-  require './test/ts_right_aws'
+desc "Analyze code coverage of the unit tests."
+Rcov::RcovTask.new do |t|
+  t.test_files = FileList["test/ts*.rb"]
+  #t.verbose = true     # uncomment to see the executed command
 end
 
+
 # vim: syntax=Ruby

data/lib/awsbase/right_awsbase.rb

@@ -21,36 +21,48 @@
 # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 #
 
-
+# Test
 module RightAws
 
-    AMAZON_PROBLEMS = [ 'internal service error', 
-                        'is currently unavailable', 
-                        'no response from', 
-                        'Please try again',
-                        'InternalError',
-                        'ServiceUnavailable', #from SQS docs
-                        'Unavailable',
-                        'This application is not currently available',
-                        'InsufficientInstanceCapacity'
-                      ]
- 
-    # Exception class to handle any Amazon errors
-    # Attributes:
-    #  message    - the text of error
-    #  errors     - a list of errors as array or a string(==message if raised manually as AwsError.new('err_text'))
-    #  request_id - amazon's request id (if exists)
-    #  http_code  - HTTP response error code (if exists)
+  # Text, if found in an error message returned by AWS, indicates that this may be a transient
+  # error. Transient errors are automatically retried with exponential back-off.
+  AMAZON_PROBLEMS = [ 'internal service error', 
+                      'is currently unavailable', 
+                      'no response from', 
+                      'Please try again',
+                      'InternalError',
+                      'ServiceUnavailable', #from SQS docs
+                      'Unavailable',
+                      'This application is not currently available',
+                      'InsufficientInstanceCapacity'
+                    ]
+
+  # Exception class to signal any Amazon errors. All errors occuring during calls to Amazon's
+  # web services raise this type of error.
+  # Attribute inherited by RuntimeError:
+  #  message    - the text of the error, generally as returned by AWS in its XML response.
+
   class AwsError < RuntimeError
-    attr_reader :errors       # Array of errors list(each item is an array - [code,message]) or error string
-    attr_reader :request_id   # Request id (if exists)
-    attr_reader :http_code    # Response HTTP error code
+    
+    # either an array of errors where each item is itself an array of [code, message]),
+    # or an error string if the error was raised manually, as in <tt>AwsError.new('err_text')</tt>
+    attr_reader :errors
+    
+    # Request id (if exists)
+    attr_reader :request_id
+    
+    # Response HTTP error code
+    attr_reader :http_code
+    
     def initialize(errors=nil, http_code=nil, request_id=nil)
       @errors      = errors
       @request_id  = request_id
       @http_code   = http_code
       super(@errors.is_a?(Array) ? @errors.map{|code, msg| "#{code}: #{msg}"}.join("; ") : @errors.to_s)
     end
+    
+    # Does any of the error messages include the regexp +pattern+?
+    # Used to determine whether to retry request.
     def include?(pattern)
       if @errors.is_a?(Array)
         @errors.each{ |code, msg| return true if code =~ pattern } 
@@ -60,8 +72,14 @@ module RightAws
       false
     end
     
+    # Generic handler for AwsErrors. +aws+ is the RightAws::S3, RightAws::EC2, or RightAws::SQS
+    # object that caused the exception (it must provide last_request and last_response). Supported
+    # boolean options are:
+    # * <tt>:log</tt> print a message into the log using aws.logger to access the Logger
+    # * <tt>:puts</tt> do a "puts" of the error
+    # * <tt>:raise</tt> re-raise the error after logging
     def self.on_aws_exception(aws, options={:raise=>true, :log=>true})
- 	        # Only log & notify if not user error
+ 	    # Only log & notify if not user error
       if !options[:raise] || system_error?($!)
         error_text = "#{$!.inspect}\n#{$@}.join('\n')}"
         puts error_text if options[:puts]
@@ -78,6 +96,8 @@ module RightAws
       return nil
     end
     
+    # True if e is an AWS system error, i.e. something that is for sure not the caller's fault.
+    # Used to force logging.
     def self.system_error?(e)
  	    !e.is_a?(self) || e.message =~ /InternalError|InsufficientInstanceCapacity|Unavailable/
  	  end
@@ -127,7 +147,7 @@ module RightAws
       if response.body && response.body[/<\?xml/]         # ... it is a xml document
         @aws.class.bench_xml.add! do
           error_parser = RightErrorResponseParser.new
-          REXML::Document.parse_stream(response.body, error_parser)
+          error_parser.parse(response)
           @aws.last_errors     = error_parser.errors
           @aws.last_request_id = error_parser.requestID
           last_errors_text     = @aws.last_errors.flatten.join("\n")
@@ -166,13 +186,25 @@ module RightAws
 
   #-----------------------------------------------------------------
 
-  class RightAWSParser #:nodoc:
+  class RightAWSParser  #:nodoc:
+    @@xml_lib = 'rexml' # xml library name: 'rexml' | 'libxml'
+    def self.xml_lib
+      @@xml_lib
+    end
+    def self.xml_lib=(new_lib_name)
+      @@xml_lib = new_lib_name
+    end
+    
     attr_accessor :result
     attr_reader   :xmlpath
-    def initialize
+    attr_accessor :xml_lib
+    
+    def initialize(params={})
       @xmlpath = ''
       @result  = false
       @text    = ''
+      @xml_lib = params[:xml_lib] || @@xml_lib
+      @logger  = params[:logger]
       reset
     end
     def tag_start(name, attributes)
@@ -188,6 +220,43 @@ module RightAws
     def text(text)
       @text = text
       tagtext(text)
+    end
+      # Parser method.
+      # Params:
+      #   xml_text         - xml message text(String) or Net:HTTPxxx instance (response)
+      #   params[:xml_lib] - library name: 'rexml' | 'libxml'
+    def parse(xml_text, params={})
+        # Get response body
+      xml_text = xml_text.body unless xml_text.is_a?(String)
+      @xml_lib = params[:xml_lib] || @xml_lib
+        # load xml library
+      if @xml_lib=='libxml' && !defined?(XML::SaxParser)
+        begin
+          require 'xml/libxml' 
+        rescue LoadError => e
+          @xml_lib = 'rexml'
+          if @logger
+            @logger.error e.inspect
+            @logger.error e.backtrace
+            @logger.info "Can not load 'libxml' library. 'Rexml' is used for parsing."
+          end
+        end
+      end
+        # Parse the xml text
+      case @xml_lib
+      when 'libxml'  
+        xml = XML::SaxParser.new
+        xml.string = xml_text
+        xml.on_start_element{|name, attr_hash| self.tag_start(name, attr_hash)}
+        xml.on_characters{   |text|            self.text(text)}
+        xml.on_end_element{  |name|            self.tag_end(name)}
+        xml.parse
+      else
+        if @logger && @xml_lib!='rexml'
+          @logger.info "RightAWSParser#parse: Unknown xml library ('#{@xml_lib}') is selected. The default 'rexml' is used."
+        end
+        REXML::Document.parse_stream(xml_text, self)
+      end
     end
       # Parser must have a lots of methods 
       # (see /usr/lib/ruby/1.8/rexml/parsers/streamparser.rb)

data/lib/awsbase/support.rb

@@ -0,0 +1,106 @@
+# If ActiveSupport is present, then great - require it and use it.  But we
+# don't want a dependency on it, so if it's not present, define the few
+# extensions that we want to use...
+begin
+  require 'rubygems'
+  require 'active_support'
+rescue LoadError => e
+  # These are ActiveSupport-;like extensions to do a few handy things in the gems
+  # Derived from ActiveSupport, so the AS copyright notice applies:
+  #
+  #
+  #
+  # Copyright (c) 2005 David Heinemeier Hansson
+  #
+  # Permission is hereby granted, free of charge, to any person obtaining
+  # a copy of this software and associated documentation files (the
+  # "Software"), to deal in the Software without restriction, including
+  # without limitation the rights to use, copy, modify, merge, publish,
+  # distribute, sublicense, and/or sell copies of the Software, and to
+  # permit persons to whom the Software is furnished to do so, subject to
+  # the following conditions:
+  #
+  # The above copyright notice and this permission notice shall be
+  # included in all copies or substantial portions of the Software.
+  #
+  # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+  # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+  # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+  # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+  # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+  # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+  # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+  #++
+  #
+  #
+  class String #:nodoc:
+
+    # Constantize tries to find a declared constant with the name specified
+    # in the string. It raises a NameError when the name is not in CamelCase
+    # or is not initialized.
+    #
+    # Examples
+    #   "Module".constantize #=> Module
+    #   "Class".constantize #=> Class
+    def constantize()
+      unless /\A(?:::)?([A-Z]\w*(?:::[A-Z]\w*)*)\z/ =~ self
+        raise NameError, "#{self.inspect} is not a valid constant name!"
+      end
+
+      Object.module_eval("::#{$1}", __FILE__, __LINE__)
+    end
+
+  end
+
+
+  class Object #:nodoc:
+    # "", "   ", nil, [], and {} are blank
+    def blank?
+      if respond_to?(:empty?) && respond_to?(:strip)
+        empty? or strip.empty?
+      elsif respond_to?(:empty?)
+        empty?
+      else
+        !self
+      end
+    end
+  end
+
+  class NilClass #:nodoc:
+    def blank?
+      true
+    end
+  end
+
+  class FalseClass #:nodoc:
+    def blank?
+      true
+    end
+  end
+
+  class TrueClass #:nodoc:
+    def blank?
+      false
+    end
+  end
+
+  class Array #:nodoc:
+    alias_method :blank?, :empty?
+  end
+
+  class Hash #:nodoc:
+    alias_method :blank?, :empty?
+  end
+
+  class String #:nodoc:
+    def blank?
+      empty? || strip.empty?
+    end
+  end
+
+  class Numeric #:nodoc:
+    def blank?
+      false
+    end
+  end
+end