kerryb-right_aws 1.7.3

data/History.txt

@@ -0,0 +1,172 @@
+== 1.1.0 2007-08-10
+Initial release.
+
+== 1.2.0 2007-09-12
+
+* r1718, todd, 2007-09-12 15:34:37
+  * # 458, Extensive documentation review, rework, and expansion.  Also added 
+    coverage analysis to the test suite using RCov.
+
+* r1690, todd, 2007-09-07 15:23:11
+  * # 447, Add support.rb to manifest
+
+* r1688, todd, 2007-09-07 13:57:39
+  * # 447, Use Active Support if available, but don't require it.  Load our own
+    extensions if it's not present.  This keeps us from overloading ActiveSupport if
+    a user's already using it.
+
+* r1687, todd, 2007-09-07 11:36:43
+  * # 447, Removed dependency on activesupport
+
+* r1676, konstantin, 2007-09-06 01:27:09
+  * paid AMIs, small fix
+
+* r1667, konstantin, 2007-09-05 12:58:10
+  * # 427, paid AMI support for ec2_instances
+
+* r1658, konstantin, 2007-09-05 01:02:25
+  * params improvements for ec2.describe_xxx, now ones can use a String as well as an Array.
+
+* r1653, konstantin, 2007-09-04 12:31:19
+  * libxml and paid AMI support added
+
+* r1581, tve, 2007-08-24 16:21:45
+  * Improved RightAws documentation
+
+== 1.3.0 2007-09-26
+
+* r1754, todd, 2007-09-19 13:48:34
+  * # 487, # 488, Consolidate a lot of code that was repeated in three places.
+    Fix error handling path when using streaming GET interfaces with S3.
+    Also add a stub for RightHttpConnection which allows more control over the
+    unit tests.  Expand the unit tests and coverage tests. 
+
+* r1755, todd, 2007-09-19 14:29:19
+  * # 487, RDoc fixes after code consolidation
+
+* r1866, konstantin, 2007-10-05 06:17:36
+  * # 220, close connection on HTTP 5xx/4xx errors
+
+== 1.4.0 2007-10-10
+
+* r1868, konstantin, 2007-10-05 06:38:37
+  * # 220, inst_type branch merge into 1.3.0 release
+
+* r1869, konstantin, 2007-10-05 07:32:34
+  * right_http_connection 1.1.1 requirements fixed
+
+* r1879, konstantin, 2007-10-07 02:11:24
+  * # 524, blocks added to ec2_describe_xxx to support aws_cache
+
+* r1924, konstantin, 2007-10-12 11:35:06
+  * # 536, user_data bug fix 
+
+* r1929, tve,  2007-10-15 00:00:11
+  * Fix libxml/rexml selection bug
+
+* r1938, konstantin, 2007-10-16 10:53:56
+  * instance type support is set to default 
+
+== 1.4.3 2007-10-25
+
+* r1983, konstantin, 2007-10-25 22:33:00 +0400
+  * Fixed ActiveSupport requirement bug (thanks to Toby)
+  * Fixed HttpConnection logging to stdout bug (thanks to Toby)
+
+== 1.4.4
+
+* r1999, tve, 2007-11-01 00:07:00 -0700
+  * Fixed escaping issue affecting key names in S3 gem
+  * Fixed duplicate marker in S3 incremental bucket listing
+
+* r2001, konstantin, 2007-11-01 12:03:13  +0300
+  * Fixed multiple permissions assignment on Grantee#grant/revoke
+  * Fixed new grantee permissions set ingnore (Grantee#apply)
+  * S3::Grantee#exists? method added 
+
+* r2109, konstantin, 2007-11-12 21:49:36  +0300
+  * RightAwsBaseInterface: caching implemented.
+    (The Ec2 functions are being cached: describe_images, describe_instances, 
+    describe_security_groups and describe_key_pairs)
+
+== 1.4.5 - 1.4.6
+* r 2619, konstantin, 01-17-08 16:18:36  +0300
+  * S3 Location constraints support added.
+  * Fixed bug with trailing '/' in the bucket name for 'EU' located buckets
+  * Added: S3Interface#bucket_location, S3::Bucket#location
+
+== 1.4.7
+* r 2622, konstantin, 01-18-08 13:52:20  +0300
+  * Virtual domains doc added
+  * S3 Query API fixed to support virtual domains.
+
+== 1.4.8
+* r 2650, konstantin, 01-24-08 11:12:00  +0300
+  * net_fix.rb moved to right_http_connection
+
+== 1.5.0
+* r 2688, konstantin, 02-30-08 15:42:00  +0300
+  * SDB support added.
+  * RightAws::S3::bucket and RightAws::S3::Bucket.create methods behaviour 
+    changed: param +create+ is set to +false+ by default.
+
+== 1.6.0
+
+* r2780, todd, 2008-02-11 11:41:07 -0800 (Mon, 11 Feb 2008), 4 lines
+  * Some doc updates & tweaks: we now support ultra-large PUTs to S3, small SDB
+    cleanups, add SDB to the list of interfaces in RightAws, update some copyrights,
+    warn about loading attachment_fu AFTER right_aws (big no-no).
+
+* r2784, todd, 2008-02-11 13:46:47 -0800 (Mon, 11 Feb 2008), 2 lines
+  * One final clarification: you may get a Net::HTTP bad monkey patch exception
+
+* r2880, todd, 2008-02-25 18:06:22 -0800 (Mon, 25 Feb 2008), 2 lines
+  * Add SQS 'Gen 2' interface implementation
+
+* r2913, todd, 2008-02-29 16:57:07 -0800 (Fri, 29 Feb 2008), 3 lines
+  * SqsGen2 (object interface), unit tests for both interfaces, documentation updates.
+
+* r2922, todd, 2008-03-03 15:26:42 -0800 (Mon, 03 Mar 2008), 2 lines
+  * couple of documentation tweaks in prep for 1.6.0
+
+== 1.6.1
+
+* r2963, todd, 2008-03-06 19:10:23 -0800 (Thu, 06 Mar 2008), 3 lines
+  * (#950) Many minor fixes in incrementally_list_bucket to prevent a death loop
+    in certain rare conditions
+
+== 1.7.0
+
+* r3051, konstantin, 2008-03-14 21:26:12 +0300 (Fri, 14 Mar 2008), 1 line
+  * #897, ActiveSdb alpha release
+
+== 1.7.1
+
+  Do not autoload right_sdb with the rest of the modules; it requires uuidtools.
+    We want the user to explicly request ActiveSdb, at least as long as it is
+    alpha/beta.
+
+  Fix escaping problem in SqsGen2Interface: POST bodies did not properly escape the '&' character
+
+== 1.7.2
+
+  Release Notes:
+  
+  RightAws includes some new features, including:
+  - Support in RightAws::S3 and RightAws::S3Interface for S3 key copy, move, and rename
+  - Support for signature version 0 request authentication to EC2, SQS, and SDB
+  - Enhanced S3 object meta-header read and update
+  - Interoperability with clouds running Eucalyptus (http://eucalyptus.cs.ucsb.edu)
+    [ Contributed by the Eucalyptus group ]
+  - Support for c1.medium and c1.xlarge instance types
+
+  Bug fixes include:
+  - Corrected the failure, under certain conditions, of retries of streaming PUTs to S3.
+    We now reset the seek pointer of the streaming IO object to its initial position.
+  - Removal of an accidental dependency on ActiveSupport in RightAws::S3Interface.get_link().
+  - Monkey-patch of the Ruby File class on Windows platforms to correct a problem in lstat.
+    The lstat bug was causing failure of very large file uploads on Windows [ Contributed by Benjamin Allfree ] 
+  - Fixed parsing of the ETag field for S3 objects
+  
+
+

data/Manifest.txt

@@ -0,0 +1,32 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+lib/awsbase/benchmark_fix.rb
+lib/awsbase/file_fix.rb
+lib/awsbase/right_awsbase.rb
+lib/awsbase/support.rb
+lib/ec2/right_ec2.rb
+lib/right_aws.rb
+lib/s3/right_s3.rb
+lib/s3/right_s3_interface.rb
+lib/sdb/active_sdb.rb
+lib/sdb/right_sdb_interface.rb
+lib/sqs/right_sqs.rb
+lib/sqs/right_sqs_gen2.rb
+lib/sqs/right_sqs_gen2_interface.rb
+lib/sqs/right_sqs_interface.rb
+test/ec2/test_helper.rb
+test/ec2/test_right_ec2.rb
+test/http_connection.rb
+test/s3/test_helper.rb
+test/s3/test_right_s3.rb
+test/s3/test_right_s3_stubbed.rb
+test/sdb/test_active_sdb.rb
+test/sdb/test_helper.rb
+test/sdb/test_right_sdb.rb
+test/sqs/test_helper.rb
+test/sqs/test_right_sqs.rb
+test/sqs/test_right_sqs_gen2.rb
+test/test_credentials.rb
+test/ts_right_aws.rb

data/README.txt

@@ -0,0 +1,146 @@
+= RightScale Amazon Web Services Ruby Gems
+
+Published by RightScale, Inc. under the MIT License.
+For information about RightScale, see http://www.rightscale.com
+
+== DESCRIPTION:
+
+The RightScale AWS gems have been designed to provide a robust, fast, and secure interface to Amazon EC2, Amazon S3, Amazon SQS, and Amazon SDB. These gems have been used in production by RightScale since late 2006 and are being maintained to track enhancements made by Amazon. The RightScale AWS gems comprise:
+
+- RightAws::Ec2 -- interface to Amazon EC2 (Elastic Compute Cloud)
+- RightAws::S3 and RightAws::S3Interface -- interface to Amazon S3 (Simple Storage Service)
+- RightAws::Sqs and RightAws::SqsInterface -- interface to first-generation Amazon SQS (Simple Queue Service) (API version 2007-05-01)
+- RightAws::SqsGen2 and RightAws::SqsGen2Interface -- interface to second-generation Amazon SQS (Simple Queue Service) (API version 2008-01-01)
+- RightAws::SdbInterface and RightAws::ActiveSdb -- interface to Amazon SDB (SimpleDB)
+
+== FEATURES:
+
+- Full programmmatic access to EC2, S3, SQS, and SDB.
+- Complete error handling: all operations check for errors and report complete
+  error information by raising an AwsError.
+- Persistent HTTP connections with robust network-level retry layer using
+  RightHttpConnection).  This includes socket timeouts and retries.
+- Robust HTTP-level retry layer.  Certain (user-adjustable) HTTP errors returned
+  by Amazon's services are classified as temporary errors.
+  These errors are automaticallly retried using exponentially increasing intervals.
+  The number of retries is user-configurable.
+- Fast REXML-based parsing of responses (as fast as a pure Ruby solution allows).
+- Uses libxml (if available) for faster response parsing. 
+- Support for large S3 list operations.  Buckets and key subfolders containing
+  many (> 1000) keys are listed in entirety.  Operations based on list (like
+  bucket clear) work on arbitrary numbers of keys.
+- Support for streaming GETs from S3, and streaming PUTs to S3 if the data source is a file.
+- Support for single-threaded usage, multithreaded usage, as well as usage with multiple
+  AWS accounts.
+- Support for both first- and second-generation SQS (API versions 2007-05-01
+  and 2008-01-01).  These versions of SQS are not compatible.
+- Support for signature versions 0 and 1 on SQS, SDB, and EC2.
+- Interoperability with any cloud running Eucalyptus (http://eucalyptus.cs.ucsb.edu)
+- Test suite (requires AWS account to do "live" testing).
+
+== THREADING:
+
+All RightScale AWS interfaces offer two threading options:
+1. Use a single persistent HTTP connection per process.
+2. Use a persistent HTTP connection per Ruby thread.
+ 
+Either way, it doesn't matter how many (for example) RightAws::S3 objects you create,
+they all use the same per-program or per-thread
+connection. The purpose of sharing the connection is to keep a single
+persistent HTTP connection open to avoid paying connection
+overhead on every request. However, if you have multiple concurrent
+threads, you may want or need an HTTP connection per thread to enable
+concurrent requests to AWS. The way this plays out in practice is:
+1. If you have a non-multithreaded Ruby program, use the non-multithreaded setting.
+2. If you have a multi-threaded Ruby program, use the multithreaded setting to enable
+   concurrent requests to S3 (or SQS, or SDB, or EC2).
+3. For running under Mongrel/Rails, use the non-multithreaded setting even though
+   mongrel is multithreaded.  This is because only one Rails handler is invoked at
+   time (i.e. it acts like a single-threaded program)
+
+Note that due to limitations in the I/O of the Ruby interpreter you
+may not get the degree of parallelism you may expect with the multi-threaded setting.
+
+By default, EC2/S3/SQS/SDB interface instances are created in single-threaded mode.  Set
+"params[:multi_thread]" to "true" in the initialization arguments to use
+multithreaded mode.
+
+== GETTING STARTED:
+
+* For EC2 read RightAws::Ec2 and consult the Amazon EC2 API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=87
+* For S3 read RightAws::S3 and consult the Amazon S3 API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=48
+* For first generation SQS read RightAws::Sqs and consult the Amazon SQS API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=31
+* For second generation SQS read RightAws::SqsGen2, RightAws::SqsGen2Interface, and consult the Amazon SQS API documentation at
+  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1214&categoryID=31 
+
+  Amazon's Migration Guide for moving from first to second generation SQS is
+  avalable at:
+  http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1148
+* For SDB read RightAws::SdbInterface, RightAws::ActiveSdb, and consult the Amazon SDB API documentation at
+  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=141
+
+== KNOWN ISSUES:
+
+- Attempting to use the Gibberish plugin (used by the Beast forum app) 
+  will break right_aws as well as lots of other code.  Gibberish
+  changes the semantics of core Ruby (specifically, the String class) and thus presents a reliability
+  problem for most Ruby programs.
+
+- 2/11/08: If you use RightAws in conjunction with attachment_fu, the
+  right_aws gem must be included (using the require statement) AFTER
+  attachment_fu.  If right_aws is loaded before attachment_fu, you'll
+  encounter errors similar to:
+
+  s3.amazonaws.com temporarily unavailable: (wrong number of arguments (5 for 4))
+
+  or
+
+  'incompatible Net::HTTP monkey-patch'
+
+  This is due to a conflict between the right_http_connection gem and another
+  gem required by attachment_fu.
+
+- 8/07: Amazon has changed the semantics of the SQS service.  A
+  new queue may not be created within 60 seconds of the destruction of any
+  older queue with the same name.  Certain methods of RightAws::Sqs and
+  RightAws::SqsInterface will fail with the message:
+  "AWS.SimpleQueueService.QueueDeletedRecently: You must wait 60 seconds after deleting a queue before you can create another with the same name."
+  
+== REQUIREMENTS:
+
+RightAws requires REXML and the RightHttpConnection gem.
+If libxml and its Ruby bindings (distributed in the libxml-ruby gem) are
+present, RightAws can be configured to use them:
+  RightAws::RightAWSParser.xml_lib = 'libxml'
+Any error with the libxml installation will result in RightAws failing-safe to
+REXML parsing.
+
+== INSTALL:
+
+sudo gem install right_aws
+
+== LICENSE:
+
+Copyright (c) 2007-2008 RightScale, Inc. 
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,96 @@
+# -*- ruby -*-
+
+require 'rubygems'
+require 'hoe'
+require "rake/testtask"
+require 'rcov/rcovtask'
+$: << File.dirname(__FILE__)
+require 'lib/right_aws.rb'
+
+testglobs =     ["test/ts_right_aws.rb"]
+
+
+# Suppress Hoe's self-inclusion as a dependency for our Gem.  This also keeps
+# Rake & rubyforge out of the dependency list.  Users must manually install
+# these gems to run tests, etc.
+class Hoe
+  def extra_deps
+    @extra_deps.reject do |x|
+      Array(x).first == 'hoe'
+    end
+  end
+end
+
+Hoe.new('right_aws', RightAws::VERSION::STRING) do |p|
+  p.rubyforge_name = 'rightaws'
+  p.author = 'RightScale, Inc.'
+  p.email = 'support@rightscale.com'
+  p.summary = 'Interface classes for the Amazon EC2, SQS, and S3 Web Services'
+  p.description = p.paragraphs_of('README.txt', 2..5).join("\n\n")
+  p.url = p.paragraphs_of('README.txt', 0).first.split(/\n/)[1..-1]
+  p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
+  p.remote_rdoc_dir = "/right_aws_gem_doc"
+  p.extra_deps = [['right_http_connection','>= 1.2.1']]
+  p.test_globs = testglobs 
+end
+
+desc "Analyze code coverage of the unit tests."
+Rcov::RcovTask.new do |t|
+  t.test_files = FileList[testglobs]
+  #t.verbose = true     # uncomment to see the executed command
+end
+ 
+desc "Test just the SQS interface"
+task :testsqs do
+  require 'test/test_credentials'
+  require 'test/http_connection'
+  TestCredentials.get_credentials
+  require 'test/sqs/test_right_sqs.rb'
+end
+
+desc "Test just the second generation SQS interface"
+task :testsqs2 do
+  require 'test/test_credentials'
+  require 'test/http_connection'
+  TestCredentials.get_credentials
+  require 'test/sqs/test_right_sqs_gen2.rb'
+end
+
+desc "Test just the S3 interface"
+task :tests3 do
+  require 'test/test_credentials'
+  require 'test/http_connection'
+  TestCredentials.get_credentials
+  require 'test/s3/test_right_s3.rb'
+end
+
+desc "Test just the S3 interface using local stubs"
+task :tests3local do
+  require 'test/test_credentials'
+  require 'test/http_connection'
+  TestCredentials.get_credentials
+  require 'test/s3/test_right_s3_stubbed.rb'
+end
+
+desc "Test just the EC2 interface"
+task :testec2 do
+  require 'test/test_credentials'
+  TestCredentials.get_credentials
+  require 'test/ec2/test_right_ec2.rb'
+end
+
+desc "Test just the SDB interface"
+task :testsdb do
+  require 'test/test_credentials'
+  TestCredentials.get_credentials
+  require 'test/sdb/test_right_sdb.rb'
+end
+
+desc "Test active SDB interface"
+task :testactivesdb do
+  require 'test/test_credentials'
+  TestCredentials.get_credentials
+  require 'test/sdb/test_active_sdb.rb'
+end
+
+# vim: syntax=Ruby

data/lib/awsbase/benchmark_fix.rb

@@ -0,0 +1,39 @@
+#
+# Copyright (c) 2007-2008 RightScale Inc
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+#
+
+
+# A hack because there's a bug in add! in Benchmark::Tms
+module Benchmark  #:nodoc:
+  class Tms #:nodoc:
+    def add!(&blk)
+      t = Benchmark::measure(&blk)
+      @utime  = utime + t.utime
+      @stime  = stime + t.stime
+      @cutime = cutime + t.cutime
+      @cstime = cstime + t.cstime
+      @real   = real + t.real
+      self
+    end
+  end
+end

data/lib/awsbase/file_fix.rb

@@ -0,0 +1,33 @@
+#
+# Copyright (c) 2007-2008 RightScale Inc
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+
+# File.lstat.size doesn't report large sizes correctly in Ruby 1.8 (Windows).
+if RUBY_PLATFORM[/mswin|mingw|bccwin|wince/i]
+  require 'win32/file'
+  class File
+    def lstat
+      self.stat
+    end
+  end
+end