simple_aws 1.0.0.pre1 → 1.0.0.pre2

data/README.md

@@ -1,40 +1,36 @@
 SimpleAWS [![Travis CI Build Status](https://secure.travis-ci.org/jameskilton/simple_aws.png)](http://travis-ci.org/jameskilton/simple_aws)
 =========
 
-A thin, simple, forward compatible Ruby wrapper around the various Amazon AWS APIs. Unless otherwise mentioned below, this library will wrap every API available service listed on this page: http://aws.amazon.com/documentation/
+A thin, simple, forward compatible Ruby wrapper around the various Amazon Web Service APIs.
 
 What?! Why?
 -----------
 
 Do we really need another Ruby library to talk to Amazon Web Services? Aren't there enough libraries out there that could just use some more help to make them better? What about [fog](http://fog.io), or [aws-sdk](http://rubygems.org/gems/aws-sdk), or [aws](http://rubygems.org/gems/aws), or [right_aws](http://rubygems.org/gems/right_aws)?
 
-While there are a number of well used libraries, I feel they have all fallen prey to the same two problems: complexity and forward incompatibility.
+While there are a number of well used libraries, I feel they have all fallen prey to the same two problems: they are far too complex for what they are and they all lack forward compatibility.
 
 ### Complexity
 
-Every Ruby AWS library in use today is simply too complex. Every library I've tried to use has ended up hurting my productivity as I often have to dive into the code to find out what the parameter list is, what object I need to work with, or what hash keys map to the actual AWS API parameters. Every library that tries to build an Object Abstraction wrapper on top of AWS's APIs suffers from leaky abstractions, which results in yet more time lost trying to figure out what needs to be called, with what, and what gets returned. Software is supposed to be simple, it's supposed to make your life easier. I've yet to find an AWS library that does this.
+Every Ruby AWS library in use today is simply too complex. Each one I've tried to use has ended up hurting my productivity as I often found myself diving into the library's code to figure out how to call the API method in question. Instead of just working with Amazon's API, I end up fighting the library, constantly having to re-learn whatever abstraction said library is trying to provide. Every library mentioned either hard-codes parameter lists, has it's own mapping from hash keys to AWS parameters, or wraps up an Object API around everything, leading to confusion and more lost productivity when that abstraction leaks. Software is supposed to be simple to use; it's supposed to make your life easier. I've yet to find an AWS library that does this.
 
 ### Forward Compatibility
 
-Ignoring the complexity argument above, what finally drove me to create this library is the complete lack of forward compatibility in all existing Ruby AWS libraries. Any time I wanted to use a new parameter, new action, or new API, I would need to jump into the library itself and implement the missing pieces. In the usual OSS circles this doesn't sound that bad, and is in fact praised, but when you're dealing with an API library, this requirement quickly becomes a frustration.
+Outside of the pervasive complexity of these libraries, what finally drove me to create this library is the complete lack of forward compatibility in all of them. Any time I wanted to use a new parameter, new action, or new API, I would need to jump into the library itself to implement the missing pieces. In normal OSS fashion, this is of course to be lauded, contributing back to libraries is what makes software better. However, in the case of API wrappers, this very quickly becomes a frustration.
 
-Amazon constantly updates AWS APIs, adding parameters and actions, and at times entire new APIs. An AWS library should work *with* the API in question, not fight against it. The only thing a hard-coded parameter list does is add confusion as you have to keep two things in your mind: what you want to call, and how the library lets you call it. SimpleAWS simply says no, it won't force anything on the user. Use the names of the API methods and parameters and them alone. If a new parameter is added to the API you're using, just use it; SimpleAWS doesn't care, it will just work.
+Amazon constantly updates AWS APIs, adding parameters and actions, and at times entire new APIs. An AWS library should work *with* the API in question, not fight against it. The only thing a hard-coded parameter list does is add confusion. When you have to figure out how AWS parameters map to library parameters, or hash keys, your productivity drops. When you have to figure out how an object is calling an AWS library, and how you're supposed to use that object, your productivity drops. Likewise when you finally realize that the library does not currently support the action, parameter, or API you're trying to use at the time, your productivity is now at a complete stop.
 
-The name SimpleAWS isn't a wish or hope, it's the core philosophy. This library focuses on being a very thin communication layer between your Ruby code and Amazon's AWS APIs. Let SimpleAWS handle the messy communication details so your code can do what it needs to do letting you be more productive.
-
-In short, it's the Unix philosophy, SimpleAWS does one thing and does it well and nothing else.
+SimpleAWS simply says no, no more leaky abstractions and confusing APIs. Just use the names of the API methods and parameters as defined in Amazon's documentation! If a new parameter is added to the API you're using, just use it. The name SimpleAWS isn't a wish or hope, it's the core philosophy. This library focuses on being a very thin communication layer between your Ruby code and Amazon's AWS APIs. Let SimpleAWS handle the messy communication details so your code can do what it needs to do letting you be more productive.
 
 
 Surely SimpleAWS isn't just `curl`?
 -----------------------------------
 
-It's well know that AWS has its share of warts and wtfs. SimpleAWS doesn't try to fix these, as that's the path towards being yet another over-complicated and rigid library that SimpleAWS is trying to solve.
-
-What SimpleAWS does do is add some logic to ensure it follows the Principle of Least Surprise.
+It's well know that AWS has its share of warts and wtfs. SimpleAWS doesn't try to fix these. What SimpleAWS does do is add as little logic as possible to make you productive and to ensure it follows the Principle of Least Surprise.
 
 ### Calling
 
-First of all, calling actions are implemented as ruby methods, handled through mainly through `method_missing` (S3 and CloudFront are the two exceptions). You can call the AWS actions by AWSName or by ruby_name, they both work:
+First of all, calling actions are implemented as ruby methods, handled through mainly through `method_missing` (S3 and CloudFront are the two current exceptions). You can call the AWS actions by AWSName or by ruby_name, they both work:
 
 ```ruby
 ec2 = AWS::EC2.new key, secret
@@ -50,7 +46,7 @@ ec2.DescribeInstances
 
 ### Parameters
 
-With that in mind, here's what SimpleAWS allows. The following three are equivalent:
+Adding parameters to your method calls follows similar rules, with some Quality of Life improvements. The following three are equivalent:
 
 #### Just Call It
 
@@ -92,9 +88,9 @@ ec2.describe_instances({
 
 ### Response Parsing
 
-All requests return an AWS::Response which does a few cleanup tasks on the resulting XML to make it easier to query and to hide some of the worst warts an XML body tends to have.
+All requests return an AWS::Response object which does a few cleanup tasks on the resulting XML to make it easier to query and to hide some of the worst warts an XML body tends to have.
 
-First of all, wherever AWS returns a list of items, they almost always get wrapped up in an `<item>` or `<member>` wrapper tag. AWS::Response gets rid of those tags and gives you a flat array to work with.
+First of all, wherever AWS returns a list of items, they often get wrapped up in an `<item>` or `<member>` wrapper tag. AWS::Response gets rid of those tags and gives you a flat array to work with.
 
 Second, the resulting XML can have one or two wrapper objects that do nothing but encapsulate the information you're interested in. AWS::Response also jumps past these wrapper objects so you have direct access to the data in the response.
 
@@ -123,6 +119,11 @@ These are the Amazon APIs that SimpleAWS currently handles:
 * {AWS::SES SES}
 * {AWS::CloudFront CloudFront}
 
+Not Currently Implemented
+-------------------------
+
+* DynamoDB (API docs aren't up yet, and the samples are hard to follow)
+
 Currently Out-Of-Scope
 ----------------------
 

data/lib/aws/api.rb

@@ -11,12 +11,15 @@ module AWS
   # implementing various APIs:
   #
   #   http://docs.amazonwebservices.com/general/latest/gr/index.html?rande.html
+  #
   ##
   class API
     class << self
 
       ##
       # Define the AWS endpoint for the API being wrapped.
+      #
+      # @param endpoint [String] Subdomain endpoint for this API. E.g. "s3" for Amazon's S3
       ##
       def endpoint(endpoint)
         @endpoint = endpoint
@@ -24,8 +27,8 @@ module AWS
 
       ##
       # Specify a default region for all requests for this API.
-      # This region will be used if no region is given to the
-      # constructor
+      #
+      # @param region [String] Specify the region this API defaults to
       ##
       def default_region(region)
         @default_region = region
@@ -35,13 +38,17 @@ module AWS
       # Specify whether this API uses HTTPS for requests. If not set,
       # the system will use HTTP. Some API endpoints are not available under
       # HTTP and some are only HTTP.
+      #
+      # @param value [Boolean] Set whether this API uses HTTPS by default or not
       ##
       def use_https(value)
         @use_https = value
       end
 
       ##
-      # Specify the AWS version of the API in question. This will be a date string.
+      # Specify the AWS version of the API in question.
+      #
+      # @param version [String] The version this API currently uses.
       ##
       def version(version)
         @version = version
@@ -53,8 +60,10 @@ module AWS
 
     ##
     # Construct a new access object for the API in question.
-    # +access_key+ and +secret_key+ are as defined in AWS security standards.
-    # Use +region+ if you need to explicitly talk to a certain AWS region
+    #
+    # @param access_key [String] Amazon access key
+    # @param secret_key [String] Amazon secret key
+    # @param region [String] Give a specific region to talk to
     ##
     def initialize(access_key, secret_key, region = nil)
       @access_key = access_key
@@ -68,6 +77,8 @@ module AWS
 
     ##
     # Get the full host name for the current API
+    #
+    # @return [String] Full URI for this API
     ##
     def uri
       return @uri if @uri

data/lib/aws/cloud_front.rb

@@ -63,12 +63,73 @@ module AWS
       super(key, secret)
     end
 
-    [:get, :post, :put, :delete].each do |method|
-      define_method(method) do |*args|
-        self.call method, *args
-      end
+    ##
+    # Send a request using HTTP GET
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def get(path, options = {})
+      call :get, path, options
     end
 
+    ##
+    # Send a request using HTTP POST
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def post(path, options = {})
+      call :post, path, options
+    end
+
+    ##
+    # Send a request using HTTP PUT
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def put(path, options = {})
+      call :put, path, options
+    end
+
+    ##
+    # Send a request using HTTP DELETE
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def delete(path, options = {})
+      call :delete, path, options
+    end
+
+    ##
+    # Execute an HTTP request against CloudFront.
+    #
+    # @param method [Symbol, String] The HTTP method to use
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
     def call(method, path, options = {})
       request = AWS::Request.new method, self.uri, "/#{self.version}#{path}"
 

data/lib/aws/core/connection.rb

@@ -29,8 +29,24 @@ end
 
 module AWS
 
+  ##
+  # Custom response parser to handle the various craziness of the AWS API
+  ##
+  class SimpleAWSParser < HTTParty::Parser
+    def parse
+      if supports_format?
+        super
+      elsif body =~ %r{<\?xml}
+        xml
+      else
+        body
+      end
+    end
+  end
+
   class HTTP
     include HTTParty
+    parser SimpleAWSParser
   end
 
   ##

data/lib/aws/import_export.rb

@@ -10,7 +10,7 @@ module AWS
   # http://docs.amazonwebservices.com/AWSImportExport/latest/API/Welcome.html
   #
   # All requests are POST and always through HTTPS.
-  # This API does not support region specifiers
+  # This API does not support region specifiers.
   #
   # @see AWS::CallTypes::ActionParam Calling rules
   # @see AWS::Response Response handling

data/lib/aws/s3.rb

@@ -55,7 +55,13 @@ module AWS
   # As such, this API does not offer streaming downloads of file data from S3.
   # That is up to you to implement at this time, by running a HEAD to get
   # Content-Length then repeated GETs using the "Range:bytes" header to specify
-  # which parts to download next.
+  # which parts to download next. You can see an example of this in samples/s3_batch_download.rb.
+  #
+  # Quality of Life note: if you forget the leading / (forward slash) in the path
+  # of a resource when# working with a bucket, this library will recognize this and
+  # fix the path for you. Thus, the following is also a valid call:
+  #
+  #   s3.put "object/name.txt", :bucket => "bucket_name", :body => File.open()
   #
   # Raw file data in a response will be available in the #body method on the Response
   # returned by the method call.
@@ -67,12 +73,18 @@ module AWS
 
     ##
     # Build a full URL for the resource at +path+.
-    # If options includes :expires, this url will be a signed url. :expires
-    # needs to be the raw Unix timestamp at which this URL will expire, as
-    # defined in the S3 documentation.
     #
-    # Otherwise, +options+ can take anything as described above, but it
-    # will not use :headers or anything related to :body.
+    # @param path [String] The path of the resource that needs a URL
+    # @param options [Hash] Options on how this URL will be generated.
+    #
+    #   If options includes +:expires+, this url will be a signed url. +:expires+
+    #   needs to be the raw Unix timestamp at which this URL will expire, as
+    #   defined in the S3 documentation.
+    #
+    #   Otherwise, +options+ can take anything as described above, but it
+    #   will not use +:headers+ or anything related to +:body+.
+    #
+    # @return [String] The URL to the requested resource
     ##
     def url_for(path, options = {})
       request = build_request(:get, path, options)
@@ -88,7 +100,8 @@ module AWS
 
       if expires_at = options[:expires]
         # Small hack, expires is in the Date section of the
-        # signing string, so we just do that here
+        # signing string, so we just do that here so that we don't
+        # muddy up build_signature_for
         request.headers["Date"] = expires_at.to_i
 
         signature = "Signature=#{build_signature_for(request)}"
@@ -101,18 +114,92 @@ module AWS
       url
     end
 
-    [:get, :put, :delete, :head].each do |method|
-      define_method(method) do |*args|
-        request = self.build_request method, *args
+    ##
+    # Send a request using HTTP GET
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def get(path, options = {})
+      call :get, path, options
+    end
 
-        connection = AWS::Connection.new
-        connection.call finish_and_sign_request(request)
-      end
+    ##
+    # Send a request using HTTP PUT
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def put(path, options = {})
+      call :put, path, options
     end
 
+    ##
+    # Send a request using HTTP DELETE
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def delete(path, options = {})
+      call :delete, path, options
+    end
+
+    ##
+    # Send a request using HTTP HEAD
+    #
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def head(path, options = {})
+      call :head, path, options
+    end
+
+    ##
+    # Execute an HTTP request against S3.
+    #
+    # @param method [Symbol, String] The HTTP method to use
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Response] The results of the request
+    #
+    # @raise [AWS::UnsuccessfulResponse, AWS::UnknownErrorResponse] on response errors
+    ##
+    def call(method, path, options = {})
+      request = self.build_request method, path, options
+
+      connection = AWS::Connection.new
+      connection.call finish_and_sign_request(request)
+    end
+
+    ##
+    # Build a request but do not send it. Helpful for debugging.
+    #
+    # @param method [Symbol, String] The HTTP method to use
+    # @param path [String] The path of the resource at hand
+    # @param options [Hash] Options as defined above
+    #
+    # @return [AWS::Request] Completed but not yet signed request object
+    ##
     def build_request(method, path, options = {})
       if options[:bucket]
-        path = path.gsub(/^\//, "/#{options[:bucket]}/")
+        path = "/#{options[:bucket]}/#{path}".gsub("//", "/")
       end
 
       request = AWS::Request.new method, self.uri, path
@@ -156,9 +243,6 @@ module AWS
       request
     end
 
-    ##
-    # S3 handles region endpoints a little differently
-    ##
     def uri
       return @uri if @uri
 

data/samples/s3.rb

@@ -45,15 +45,15 @@ puts "", "Uploading #{file_name} to #{bucket_name}:", ""
 bad_usage unless file_name
 uploaded_file_name = File.basename file_name
 
-p s3.put("/#{uploaded_file_name}", :bucket => bucket_name, :file => File.open(file_name))
+p s3.put(uploaded_file_name, :bucket => bucket_name, :file => File.open(file_name))
 
 puts "", "Checking that the file now exists...", ""
 
-p s3.head("/#{uploaded_file_name}", :bucket => bucket_name)
+p s3.head(uploaded_file_name, :bucket => bucket_name)
 
 puts "", "Getting file again", ""
 
-p s3.get("/#{uploaded_file_name}", :bucket => bucket_name,
+p s3.get(uploaded_file_name, :bucket => bucket_name,
          :params => {
            "response-content-disposition" => "attachment",
            "response-content-type" => "text/ruby",
@@ -61,18 +61,18 @@ p s3.get("/#{uploaded_file_name}", :bucket => bucket_name,
 
 puts "", "Signed, expiring URL for this resource: ", ""
 
-puts s3.url_for("/#{uploaded_file_name}", :bucket => bucket_name,
+puts s3.url_for(uploaded_file_name, :bucket => bucket_name,
              :expires => Time.now.to_i + 120,
              :params => { "response-content-disposition" => "attachment" })
 
 puts "", "Deleting the file from S3", ""
 
-p s3.delete("/#{uploaded_file_name}", :bucket => bucket_name)
+p s3.delete(uploaded_file_name, :bucket => bucket_name)
 
 puts "", "Checking that file is no longer in S3...", ""
 
 begin
-  p s3.head("/#{uploaded_file_name}", :bucket => bucket_name)
+  p s3.head(uploaded_file_name, :bucket => bucket_name)
 rescue => ex
   puts "Not found: #{ex.message}"
 end

data/samples/s3_batch_download.rb

@@ -0,0 +1,53 @@
+$: << File.expand_path("../../lib", __FILE__)
+
+require 'aws/s3'
+
+##
+# Expects your Amazon keys to be in the environment, something like
+#
+# export AWS_KEY="KEY"
+# export AWS_SECRET="SECRET"
+#
+# Usage:
+#   ruby sample/s3.rb [resource path] [file to save to]
+#
+# This script shows one way to work with batch downloading big files from S3.
+# Ensure the resource path starts with the bucket in question.
+##
+
+def bad_usage
+  puts ""
+  puts "Usage:"
+  puts "  ruby sample/s3_batch_download.rb [bucket name] [file to use]"
+  exit 1
+end
+
+s3 = AWS::S3.new ENV["AWS_KEY"], ENV["AWS_SECRET"]
+
+s3_resource = ARGV[0]
+save_to = ARGV[1]
+batch_size = 1_000_000
+
+
+# First, get the full size of the file in question
+file_size = s3.head(s3_resource).headers["content-length"].to_i
+
+puts "File is #{file_size} bytes"
+puts "Now downloading in batches of #{batch_size} bytes"
+
+chunk_count = 0
+bytes_start = 0
+bytes_to = batch_size
+
+File.open save_to, "w+" do |file|
+  while bytes_start < file_size
+    puts "Chunk #{chunk_count} : #{bytes_start} - #{bytes_to}"
+    file.write s3.get(s3_resource, :headers => {"Range" => "bytes=#{bytes_start}-#{bytes_to}" }).body
+
+    bytes_start = bytes_to + 1
+    bytes_to += batch_size
+    chunk_count += 1
+  end
+end
+
+puts "Done"

data/simple_aws.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name     = "simple_aws"
-  s.version  = "1.0.0.pre1"
+  s.version  = "1.0.0.pre2"
   s.platform = Gem::Platform::RUBY
   s.authors  = ["Jason Roelofs"]
   s.email    = ["jameskilton@gmail.com"]

data/test/aws/s3_test.rb

@@ -71,16 +71,24 @@ describe AWS::S3 do
       end
     end
 
-    it "rebuilds the host if :bucket given" do
+    it "rebuilds the path if :bucket given" do
       AWS::Connection.any_instance.expects(:call).with do |request|
         request.path.must_equal "/bucket-name/"
-        request.host.must_equal "https://s3.amazonaws.com"
         true
       end
 
       @api.get "/", :bucket => "bucket-name"
     end
 
+    it "auto-fixes path names if not preceeded by a /" do
+      AWS::Connection.any_instance.expects(:call).with do |request|
+        request.path.must_equal "/bucket-name/object_name/that_thing.jpg"
+        true
+      end
+
+      @api.get "object_name/that_thing.jpg", :bucket => "bucket-name"
+    end
+
     it "takes parameters" do
       AWS::Connection.any_instance.expects(:call).with do |request|
         request.params["Parameter1"].must_equal "Value2"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: simple_aws
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre1
+  version: 1.0.0.pre2
   prerelease: 6
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-01-20 00:00:00.000000000 Z
+date: 2012-01-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
-  requirement: &70160933707720 !ruby/object:Gem::Requirement
+  requirement: &70269561987000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -21,10 +21,10 @@ dependencies:
         version: 1.5.0
   type: :runtime
   prerelease: false
-  version_requirements: *70160933707720
+  version_requirements: *70269561987000
 - !ruby/object:Gem::Dependency
   name: httparty
-  requirement: &70160933701880 !ruby/object:Gem::Requirement
+  requirement: &70269561981040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,7 +32,7 @@ dependencies:
         version: 0.8.0
   type: :runtime
   prerelease: false
-  version_requirements: *70160933701880
+  version_requirements: *70269561981040
 description: SimpleAWS is a clean, simple, and forward compatible library for talking
   to Amazon's AWS APIs.
 email:
@@ -77,6 +77,7 @@ files:
 - samples/elb.rb
 - samples/iam.rb
 - samples/s3.rb
+- samples/s3_batch_download.rb
 - samples/sqs.rb
 - samples/turk.rb
 - simple_aws.gemspec