s3cp 1.0.6 → 1.1.0.pre.1

data/History.txt

@@ -1,7 +1,19 @@
+=== 1.1.0 / (Pending)
+
+* Changed:  Underlying AWS library has been changed from 'right-aws' to the official
+	    AWS SDK ('aws-sdk').   The main driver for this is to stay current on the
+	    latest features, including instance-specific roles, temporary credentials,
+	    etc.
+
+* Added:    New 's3buckets' command displays all S3 buckets associated with the
+	    account.
+
+* Added:    New '--acl PERMISSION' option to 's3cp' and 's3up' commands.
+
 === 1.0.6 / (2012-07-23)
 
 * Fixed:  s3cp would exit with errorlevel 0 despite errors when running in
-          interactive mode.
+	  interactive mode.
 
 === 1.0.5 / (2012-06-25)
 
@@ -15,15 +27,15 @@
 
 * Added:    Exponential backoff for `s3cp` command based on the formula:
 
-              delay = initial_retry_delay * (factor ^ retries)
+	      delay = initial_retry_delay * (factor ^ retries)
 
-            where:
-                    - `initial_retry_delay` is 1 second by default.
-                    - `factor` is 1.4142 by default
-                      (retry delay doubles every two retries)
-                    - max. number of retries is now 20 by default.
+	    where:
+		    - `initial_retry_delay` is 1 second by default.
+		    - `factor` is 1.4142 by default
+		      (retry delay doubles every two retries)
+		    - max. number of retries is now 20 by default.
 
-            (which means s3cp retries for roughly 58 minutes by default)
+	    (which means s3cp retries for roughly 58 minutes by default)
 
 * Fixed:  `s3ls` and `s3du` now properly handle Errno::EPIPE
 
@@ -33,18 +45,18 @@
 
 * Added:  `s3up` now outputs uploaded file size to STDERR, e.g.,
 
-          % ls | s3up s3://bucket/path/to/file
-          s3://bucket/path/to/file => 214B
+	  % ls | s3up s3://bucket/path/to/file
+	  s3://bucket/path/to/file => 214B
 
 === 1.0.1 / 2012-03-9
 
 * Added:  New command `s3up` to upload STDIN to S3
-          e.g. some_cmd | s3up s3://bucket/path/to/destination
+	  e.g. some_cmd | s3up s3://bucket/path/to/destination
 
   Note: `s3up` does not yet directly stream data to S3 since right_aws gem
-        requires the upload size to be known in advance. `s3up` currently
-        persists STDIN into a temp file although this may change in the
-        future.
+	requires the upload size to be known in advance. `s3up` currently
+	persists STDIN into a temp file although this may change in the
+	future.
 
 === 1.0.0 / 2012-03-9
 
@@ -58,27 +70,27 @@ bugs -- but at least all known bugs have been addressed.
 Probably worth mentioning: there are two outstanding experimental features,
 
 1) s3cp --sync:  Does not handle missing files.  It's basically
-                 a faster copy; not a true "sync" at this point.
+		 a faster copy; not a true "sync" at this point.
 
 2) Bash command-line completion is still very rough.
 
 === 0.2.7 / 2012-03-09
 
 * Fixed #3: s3cp adds extra slash when copying to the root of a bucket.
-            e.g. s3://myBucket//someFile or s3://myBucket//someFolderToCopy/someFile
+	    e.g. s3://myBucket//someFile or s3://myBucket//someFolderToCopy/someFile
 
 === 0.2.6 / 2012-03-06
 
 * Fixed:  Possible division-by-zero error in s3du if it encounters
-          zero-length files.
+	  zero-length files.
 
 === 0.2.5 / 2012-03-02
 
 * Added: "s3du" command to calculate disk usage.
-         Supports --depth, --regex, --unit parameters and more!
+	 Supports --depth, --regex, --unit parameters and more!
 
 * Changed: "s3ls -l" command now accepts --unit and --precision to configure
-           file size display.  Uses "smart" unit by default.
+	   file size display.  Uses "smart" unit by default.
 
 * Changed: "s3ls -l" will now use S3CP_DATE_FORMAT environment if set.
 
@@ -120,12 +132,12 @@ Probably worth mentioning: there are two outstanding experimental features,
 * Added: Progress bars during upload/download if $stdout.isatty
 
 * Fixed: s3cat now handles broken pipes properly
-         e.g. "s3cat bucket:some/file | head" will now terminate early.
+	 e.g. "s3cat bucket:some/file | head" will now terminate early.
 
 === 0.2.1 / (2012-02-20)
 
 * Added: Bash completion now supports exclusions through S3CP_EXCLUDES
-         and defaults to excluding keys containing "_$folder$".
+	 and defaults to excluding keys containing "_$folder$".
 
 * Changed: s3dir and s3ls --delimiter now display both directories and files
 
@@ -134,15 +146,15 @@ Probably worth mentioning: there are two outstanding experimental features,
 * Added: s3stat command to display S3 object properties
 
 * Added: s3dir as a shortcut for "s3ls --delimiter / ..."
-         (default to character in S3CP_DELIMITER environment variable or "/" if not defined)
+	 (default to character in S3CP_DELIMITER environment variable or "/" if not defined)
 
 * Added: s3cp defaults can now be set using environment variables
-         S3CP_OVERWRITE, S3CP_CHECKSUM, S3CP_RETRIES, S3CP_RETRY_DELAY
+	 S3CP_OVERWRITE, S3CP_CHECKSUM, S3CP_RETRIES, S3CP_RETRY_DELAY
 
 * Added: Support for Bash command-line completion of S3 URLs (see below).
 
 * Fixed: Skip checksum verification for S3 objects with invalid MD5's
-         (details @ https://forums.aws.amazon.com/message.jspa?messageID=234538)
+	 (details @ https://forums.aws.amazon.com/message.jspa?messageID=234538)
 
 To install Bash completion for S3 URLs, add the following to ~/.bashrc:
 
@@ -153,10 +165,10 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
 === 0.1.15 / (2012-02-17)
 
 * Added: s3cp now automatically checks MD5 checksums during download/upload
-         and retries up to 5 times by default if the checksum fails.
-         The number of attempts may be configured using --max-attempts,
-         the retry delay may be changed with --retry-delay and the check
-         may be disabled completely using --no-checksum.
+	 and retries up to 5 times by default if the checksum fails.
+	 The number of attempts may be configured using --max-attempts,
+	 the retry delay may be changed with --retry-delay and the check
+	 may be disabled completely using --no-checksum.
 
 === 0.1.14 / (2012-02-09)
 
@@ -178,7 +190,7 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
 
 * Fixed: --max-keys now works correctly with --delimiter
 * Fixed: do not display any keys if there are no common-prefix
-         delimiter-matching keys
+	 delimiter-matching keys
 
 === 0.1.9 / (2012-01-23)
 
@@ -187,18 +199,18 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
 === 0.1.8 / (2011-12-29)
 
 * Fixed:  Apparently, File.new(path, File::CREAT|File::WRONLY) does not
-          truncate existing files; use File.new(path, "wb") instead.
+	  truncate existing files; use File.new(path, "wb") instead.
 
 === 0.1.7 / (2011-12-29)
 
 * Fixed:  s3cp would not truncate existing files when overwriting, possibly
-          resulting in corrupted files.
+	  resulting in corrupted files.
 
 === 0.1.6 / (2011-12-16)
 
 * Changed: s3rm now uses multi-object delete operation for faster deletes
 * Changed: dependency on 'aboisvert-aws' instead of 'right-aws' until
-           it supports multi-object delete.
+	   it supports multi-object delete.
 
 === 0.1.5 / 2011-10-17
 
@@ -211,7 +223,7 @@ To install Bash completion for S3 URLs, add the following to ~/.bashrc:
 === 0.1.3 / 2011-09-29
 
 * Fixed:  s3cp --headers names are now converted to lowercase since underlying
-          RightAWS gem expects lowercase header names.
+	  RightAWS gem expects lowercase header names.
 
 === 0.1.2 / 2011-09-29
 

data/bin/s3buckets

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+require 's3cp/s3buckets'
+

data/lib/s3cp/#Untitled-1#

@@ -0,0 +1,35 @@
+  def set_header_options(options, headers)
+    return options unless headers
+
+    # legacy options that were previously passed as headers
+    # are now passed explicitly as options/metadata.
+    mappings = {
+      "Content-Type"  =>  :content_type,
+      "x-amz-acl"     =>  :acl,
+      "Cache-Control" =>  :cache_control,
+      "x-amz-storage-class" => :reduced_redundancy
+    }
+
+    lambdas = {
+      "x-amz-storage-class" => lambda { |v| (v =~ /^REDUCED_REDUNDANCY$/i) ? true : false }
+    }
+
+    remaining = headers.dup
+    headers.each do |hk, hv|
+      mappings.each do |mk, mv|
+        if hk.to_s =~ /^#{mk}$/i
+          lambda = lambdas[mk]
+          options[mv] = lambda ? lambda.call(hv) : hv
+          remaining.delete(hk)
+        end
+      end
+    end
+
+
+    options[:metadata] = remaining unless remaining.empty?
+
+    options
+  end
+
+  
+  set_header_options({}, { "Content-type" => "foo ", 'x-amz-acl' => 'private', 'foo' => 'bar', 'x-amz-storage-class' => 'REDUCED_REDUNDANCY'})

data/lib/s3cp/s3buckets.rb

@@ -0,0 +1,41 @@
+# Copyright (C) 2010-2012 Alex Boisvert and Bizo Inc. / All rights reserved.
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor
+# license agreements.  See the NOTICE file  distributed with this work for
+# additional information regarding copyright ownership.  The ASF licenses this
+# file to you under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License.  You may obtain a copy of
+# the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
+require 's3cp/utils'
+
+# Parse arguments
+options = {}
+
+op = OptionParser.new do |opts|
+  opts.banner = "s3buckets"
+
+  opts.on("--verbose", "Verbose mode") do
+    options[:verbose] = true
+  end
+
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts op
+    exit
+  end
+end
+op.parse!(ARGV)
+
+s3 = S3CP.connect()
+s3.buckets.each do |bucket|
+  puts bucket.name
+end
+

data/lib/s3cp/s3cat.rb

@@ -15,16 +15,9 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
-require 'date'
-require 'highline/import'
-require 'tempfile'
-require 'progressbar'
-
 require 's3cp/utils'
+require 'progressbar'
+require 'tempfile'
 
 # Parse arguments
 options = {}
@@ -64,12 +57,11 @@ end
 @bucket, @prefix = S3CP.bucket_and_key(url)
 fail "Your URL looks funny, doesn't it?" unless @bucket
 
-@s3 = S3CP.connect().interface
+@s3 = S3CP.connect().buckets[@bucket]
 
 if options[:tty]
   # store contents to file to display with PAGER
-  metadata = @s3.head(@bucket, @prefix)
-  size = metadata["content-length"].to_i
+  size = @s3.objects[@prefix].content_length
 
   progress_bar = ProgressBar.new(File.basename(@prefix), size).tap do |p|
     p.file_transfer_mode
@@ -78,7 +70,7 @@ if options[:tty]
   file = Tempfile.new('s3cat')
   out = File.new(file.path, "wb")
   begin
-    @s3.get(@bucket, @prefix) do |chunk|
+    @s3.objects[@prefix].read_as_stream do |chunk|
       out.write(chunk)
       progress_bar.inc chunk.size
     end
@@ -89,7 +81,7 @@ if options[:tty]
   exec "#{ENV['PAGER'] || 'less'} #{file.path}"
   file.delete()
 else
-  @s3.get(@bucket, @prefix) do |chunk|
+  @s3.objects[@prefix].read_as_stream do |chunk|
     begin
       STDOUT.print(chunk)
     rescue Errno::EPIPE

data/lib/s3cp/s3cp.rb

@@ -15,17 +15,8 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
-require 'date'
-require 'highline/import'
-require 'fileutils'
-require 'digest'
-require 'progressbar'
-
 require 's3cp/utils'
+require 'progressbar'
 
 # Parse arguments
 options = {}
@@ -103,6 +94,10 @@ op = OptionParser.new do |opts|
     options[:headers] = h
   end
 
+  opts.on("--acl PERMISSION", "One of 'private', 'authenticated-read', 'public-read', 'public-read-write'") do |permission|
+    options[:acl] = S3CP.validate_acl(permission)
+  end
+
   opts.separator "        e.g.,"
   opts.separator "              HTTP headers: \'Content-Type: image/jpg\'"
   opts.separator "               AMZ headers: \'x-amz-acl: public-read\'"
@@ -243,12 +238,16 @@ end
 
 def s3_to_s3(bucket_from, key, bucket_to, dest, options = {})
   log(with_headers("Copy s3://#{bucket_from}/#{key} to s3://#{bucket_to}/#{dest}"))
-  if @headers.empty?
-    @s3.interface.copy(bucket_from, key, bucket_to, dest)
+  s3_source = @s3.buckets[bucket_from].objects[key]
+  s3_dest = @s3.buckets[bucket_to].objects[dest]
+  s3_options = {}
+  S3CP.set_header_options(s3_, @headers)
+  s3_options[:acl] = options[:acl]
+  unless options[:move]
+    s3_source.copy_to(s3_dest, s3_options)
   else
-    @s3.interface.copy(bucket_from, key, bucket_to, dest, :copy, @headers)
+    s3_source.move_to(s3_dest, s3_options)
   end
-  @s3.interface.delete(bucket_from, key) if options[:move]
 end
 
 def local_to_s3(bucket_to, key, file, options = {})
@@ -279,37 +278,33 @@ def local_to_s3(bucket_to, key, file, options = {})
       end
       if retries > 0
         delay = options[:retry_delay] * (options[:retry_backoff] ** retries)
-        STDERR.puts "Sleeping #{delay} seconds.  Will retry #{options[:retries] - retries} more time(s)."
+        STDERR.puts "Sleeping #{"%0.2f" % delay} seconds.  Will retry #{options[:retries] - retries} more time(s)."
         sleep delay
       end
 
-      f = File.open(file)
       begin
-        if $stdout.isatty
-          f = Proxy.new(f)
-          progress_bar = ProgressBar.new(File.basename(file), File.size(file)).tap do |p|
-            p.file_transfer_mode
-          end
-          class << f
-            attr_accessor :progress_bar
-
-            def read(length, buffer=nil)
-              begin
-                result = @target.read(length, buffer)
-                @progress_bar.inc result.length if result
-                result
-              rescue => e
-                STDERR.puts e
-                raise e
-              end
+        s3_options = {
+          :bucket_name => bucket_to,
+          :key => key
+        }
+        S3CP.set_header_options(s3_options, @headers)
+        s3_options[:acl] = options[:acl]
+        s3_options[:content_length] = File.size(file)
+
+        progress_bar = ProgressBar.new(File.basename(file), File.size(file)).tap do |p|
+          p.file_transfer_mode
+        end
+
+        meta = @s3.client.put_object(s3_options) do |buffer|
+          File.open(file) do |io|
+            while !io.eof?
+              result = io.read(32 * 1024)
+              progress_bar.inc result.length if result
+              buffer.write(result)
             end
           end
-          f.progress_bar = progress_bar
-        else
-          progress_bar = nil
         end
 
-        meta = @s3.interface.put(bucket_to, key, f, @headers)
         progress_bar.finish if progress_bar
 
         if options[:checksum]
@@ -329,10 +324,8 @@ def local_to_s3(bucket_to, key, file, options = {})
           progress_bar.clear
           puts "Error copying #{file} to s3://#{bucket_to}/#{key}"
         end
-        raise e if !options[:checksum] || e.to_s =~ /Denied/
+        raise e if !options[:checksum] || e.is_a?(AWS::S3::Errors::AccessDenied)
         STDERR.puts e
-      ensure
-        f.close()
       end
       retries += 1
     end until options[:checksum] == false || actual_md5.nil? || expected_md5 == actual_md5
@@ -353,7 +346,8 @@ def s3_to_local(bucket_from, key_from, dest, options = {})
     end
     if retries > 0
       delay = options[:retry_delay] * (options[:retry_backoff] ** retries)
-      STDERR.puts "Sleeping #{delay} seconds.  Will retry #{options[:retries] - retries} more time(s)."
+      delay = delay.to_i
+      STDERR.puts "Sleeping #{"%0.2f" % delay} seconds.  Will retry #{options[:retries] - retries} more time(s)."
       sleep delay
     end
     begin
@@ -383,7 +377,7 @@ def s3_to_local(bucket_from, key_from, dest, options = {})
               p.file_transfer_mode
             end
           end
-          @s3.interface.get(bucket_from, key_from) do |chunk|
+          @s3.buckets[bucket_from].objects[key_from].read_as_stream do |chunk|
             f.write(chunk)
             progress_bar.inc chunk.size if progress_bar
           end
@@ -413,25 +407,23 @@ def s3_to_local(bucket_from, key_from, dest, options = {})
     retries += 1
   end until options[:checksum] == false || expected_md5.nil? || md5(dest) == expected_md5
 
-  @s3.interface.delete(bucket_from, key_from) if options[:move]
+  @s3.buckets[bucket_from].objects[key_from].delete() if options[:move]
 end
 
 def s3_exist?(bucket, key)
-  metadata = @s3.interface.head(bucket, key)
-  #puts "exist? #{bucket} #{key} => #{metadata != nil}"
-  (metadata != nil)
+  @s3.buckets[bucket].objects[key].exist?
 end
 
 def s3_checksum(bucket, key)
   begin
-    metadata = @s3.interface.head(bucket, key)
+    metadata = @s3.buckets[bucket].objects[key].head()
     return :not_found unless metadata
   rescue => e
-    return :not_found if e.is_a?(RightAws::AwsError) && e.http_code == "404"
+    return :not_found if e.is_a?(AWS::S3::Errors::NoSuchKey)
     raise e
   end
 
-  md5 = metadata["etag"] or fail "Unable to get etag/md5 for #{bucket_to}:#{key}"
+  md5 = metadata[:etag] or fail "Unable to get etag/md5 for #{bucket_to}:#{key}"
   return :invalid unless md5
 
   md5 = md5.sub(/^"/, "").sub(/"$/, "") # strip beginning and trailing quotes
@@ -449,8 +441,7 @@ def key_path(prefix, key)
 end
 
 def s3_size(bucket, key)
-  metadata = @s3.interface.head(bucket, key)
-  metadata["content-length"].to_i
+  @s3.buckets[bucket].objects[key].content_length
 end
 
 def copy(from, to, options)

data/lib/s3cp/s3du.rb

@@ -15,13 +15,6 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
-require 'date'
-require 'highline/import'
-
 require 's3cp/utils'
 
 # Parse arguments
@@ -72,9 +65,6 @@ fail "Your URL looks funny, doesn't it?" unless @bucket
 
 @s3 = S3CP.connect()
 
-s3_options = Hash.new
-s3_options[:prefix] = @prefix
-
 def depth(path)
   path.count("/")
 end
@@ -103,29 +93,25 @@ def print(key, size)
 end
 
 begin
-  @s3.interface.incrementally_list_bucket(@bucket, s3_options) do |page|
-
-    entries = page[:contents]
-    entries.each do |entry|
-      key  = entry[:key]
-      size = entry[:size]
-
-      if options[:regex].nil? || options[:regex].match(key)
-        current_key = if actual_depth
-          pos = nth_occurrence(key, "/", actual_depth)
-          (pos != -1) ? key[0..pos-1] : key
-        end
-
-        if (last_key && last_key != current_key)
-          print(last_key, subtotal_size)
-          subtotal_size = size
-        else
-          subtotal_size += size
-        end
-
-        last_key = current_key
-        total_size += size
+  @s3.buckets[@bucket].objects.with_prefix(@prefix).each do |entry|
+    key  = entry.key
+    size = entry.content_length
+
+    if options[:regex].nil? || options[:regex].match(key)
+      current_key = if actual_depth
+        pos = nth_occurrence(key, "/", actual_depth)
+        (pos != -1) ? key[0..pos-1] : key
       end
+
+      if (last_key && last_key != current_key)
+        print(last_key, subtotal_size)
+        subtotal_size = size
+      else
+        subtotal_size += size
+      end
+
+      last_key = current_key
+      total_size += size
     end
   end
 

data/lib/s3cp/s3ls.rb

@@ -15,19 +15,12 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
-require 'date'
-require 'highline/import'
-
 require 's3cp/utils'
 
 # Parse arguments
 options = {}
 options[:date_format] = ENV['S3CP_DATE_FORMAT'] || '%x %X'
-options[:rows_per_page] = $terminal.output_rows if $stdout.isatty
+options[:rows_per_page] = ($terminal.output_rows - 1) if $stdout.isatty
 options[:precision] = 0
 
 op = OptionParser.new do |opts|
@@ -93,53 +86,49 @@ if options[:verbose]
   puts "key #{@key}"
 end
 
-@s3 = S3CP.connect()
+@s3 = S3CP.connect().buckets[@bucket]
 
 keys = 0
 rows = 0
 
-s3_options = Hash.new
-s3_options[:prefix] = @key
-s3_options["max-keys"] = options[:max_keys] if options[:max_keys] && !options[:delimiter]
-s3_options[:delimiter] = options[:delimiter] if options[:delimiter]
-
 begin
-  @s3.interface.incrementally_list_bucket(@bucket, s3_options) do |page|
-    entries = []
-    if options[:delimiter]
-      entries << { :key => page[:contents][0][:key] } if page[:contents].length > 0 && entries.length > 0
-      page[:common_prefixes].each do |entry|
-        entries << { :key => entry }
-      end
-      entries << { :key => nil }
+  display = lambda do |entry|
+    key = entry.key ? "s3://#{@bucket}/#{entry.key}" : "---"
+    if options[:long_format] && entry.last_modified && entry.content_length
+      size = entry.content_length
+      size = S3CP.format_filesize(size, :unit => options[:unit], :precision => options[:precision])
+      size = ("%#{7 + options[:precision]}s " % size)
+      puts "#{entry.last_modified.strftime(options[:date_format])} #{size} #{key}"
+    else
+      puts key
     end
-    entries += page[:contents]
-    entries.each do |entry|
-      key = entry[:key] ? "s3://#{@bucket}/#{entry[:key]}" : "---"
-      if options[:long_format] && entry[:last_modified] && entry[:size]
-        last_modified = DateTime.parse(entry[:last_modified])
-        size = entry[:size]
-        size = S3CP.format_filesize(size, :unit => options[:unit], :precision => options[:precision])
-        size = ("%#{7 + options[:precision]}s " % size)
-        puts "#{last_modified.strftime(options[:date_format])} #{size} #{key}"
-      else
-        puts key
-      end
-      rows += 1
-      keys += 1
-      if options[:max_keys] && keys >= options[:max_keys]
-        exit
-      end
-      if options[:rows_per_page] && (rows % options[:rows_per_page] == 0)
-        begin
-          print "Continue? (Y/n) "
-          response = STDIN.gets.chomp.downcase
-        end until response == 'n' || response == 'y' || response == ''
-        exit if response == 'n'
+    rows += 1
+    keys += 1
+    if options[:rows_per_page] && (rows % options[:rows_per_page] == 0)
+      begin
+        print "Continue? (Y/n) "
+        response = STDIN.gets.chomp.downcase
+      end until response == 'n' || response == 'y' || response == ''
+      exit if response == 'n'
+    end
+  end
+
+  if options[:delimiter]
+    @s3.objects.with_prefix(@key).as_tree(:delimier => options[:delimiter], :append => false).children.each do |entry|
+      if entry.leaf?
+        entry = @s3.objects[entry.key]
+        break if display.call(entry)
       end
     end
+  else
+    s3_options = Hash.new
+    s3_options[:limit] = options[:max_keys] if options[:max_keys]
+    @s3.objects.with_prefix(@key).each(s3_options) do |entry|
+      break if display.call(entry)
+    end
   end
 rescue Errno::EPIPE
   # ignore
+  break
 end
 

data/lib/s3cp/s3mod.rb

@@ -15,10 +15,6 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
 require 's3cp/utils'
 
 op = OptionParser.new do |opts|
@@ -48,14 +44,11 @@ end
 
 def update_permissions(s3, bucket, key, permission)
   puts "Setting #{permission} on s3://#{bucket}/#{key}"
-  s3.interface.copy(bucket, key, bucket, key, :replace, {"x-amz-acl" => permission })
+  s3.buckets[bucket].objects[key].acl = permission
 end
 
 source  = ARGV[0]
-permission = ARGV.last
-
-LEGAL_MODS = %w{private authenticated-read public-read public-read-write}
-raise "Permissions must be one of the following values: #{LEGAL_MODS}" unless LEGAL_MODS.include?(permission)
+permission = S3CP.validate_acl(ARGV.last)
 
 @s3 = S3CP.connect()
 bucket,key = S3CP.bucket_and_key(source)

data/lib/s3cp/s3rm.rb

@@ -15,13 +15,6 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
-require 'date'
-require 'highline/import'
-
 require 's3cp/utils'
 
 # Parse arguments
@@ -110,20 +103,18 @@ exclude_regex = options[:exclude_regex] ? Regexp.new(options[:exclude_regex]) :
 if options[:recursive]
   matching_keys = []
 
-  @s3.interface.incrementally_list_bucket(@bucket, :prefix => @key) do |page|
-    page[:contents].each do |entry|
-      key = "s3://#{@bucket}/#{entry[:key]}"
+  @s3.buckets[@bucket].objects.with_prefix(@key).each do |entry|
+    key = "s3://#{@bucket}/#{entry.key}"
 
-      matching = true
-      matching = false if include_regex && !include_regex.match(entry[:key])
-      matching = false if exclude_regex && exclude_regex.match(entry[:key])
+    matching = true
+    matching = false if include_regex && !include_regex.match(entry.key)
+    matching = false if exclude_regex && exclude_regex.match(entry.key)
 
-      puts "#{key} => #{matching}" if options[:verbose]
+    puts "#{key} => #{matching}" if options[:verbose]
 
-      if matching
-        matching_keys << entry[:key]
-        puts key unless options[:silent] || options[:verbose]
-      end
+    if matching
+      matching_keys << entry.key
+      puts key unless options[:silent] || options[:verbose]
     end
   end
 
@@ -132,28 +123,20 @@ if options[:recursive]
     exit(1)
   end
 
-  errors = []
-  errors = @s3.interface.delete_multiple(@bucket, matching_keys) unless options[:test]
-
-  if errors.length > 0
-    puts "Errors during deletion:"
-    errors.each do |error|
-      puts "#{error[:key]} #{error[:code]} #{error[:message]}"
-    end
-    exit(1)
-  end
+  # if any of the objects failed to delete, a BatchDeleteError will be raised with a summary of the errors
+  @s3.buckets[@bucket].objects.delete(matching_keys) unless options[:test]
 else
   # delete a single file; check if it exists
-  if options[:fail_if_not_exist] && @s3.interface.head(@bucket, @key) == nil
+  if options[:fail_if_not_exist] && !@s3.buckets[@bucket].objects[@key].exist?
     key = "s3://#{@bucket}/#{@key}"
     puts "#{key} does not exist."
     exit(1)
   end
 
   begin
-    @s3.interface.delete(@bucket, @key) unless options[:test]
+    @s3.buckets[@bucket].objects[@key].delete() unless options[:test]
   rescue => e
     puts e.to_s
-    raise e unless e.to_s =~ /Not Found/
+    raise e unless e.is_a? AWS::S3::Errors::NoSuchKey
   end
 end

data/lib/s3cp/s3stat.rb

@@ -15,10 +15,6 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
 require 's3cp/utils'
 
 op = OptionParser.new do |opts|
@@ -40,15 +36,11 @@ end
 source  = ARGV[0]
 permission = ARGV.last
 
-@s3 = S3CP.connect()
+@bucket, @key = S3CP.bucket_and_key(source)
+@s3 = S3CP.connect().buckets[@bucket]
 
-def get_metadata(bucket, key)
-  metadata = @s3.interface.head(bucket, key)
-  metadata.sort.each do |k,v|
-    puts "#{"%20s" % k} #{v}"
-  end
+metadata = @s3.objects[@key].head
+metadata.keys.sort { |k1, k2| k1.to_s <=> k2.to_s}.each do |k|
+  puts "#{"%30s" % k} #{metadata[k]}"
 end
 
-bucket,key = S3CP.bucket_and_key(source)
-get_metadata(bucket, key)
-

data/lib/s3cp/s3up.rb

@@ -15,13 +15,8 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
-require 'rubygems'
-require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
-require 'right_aws'
-require 'optparse'
-require 'tempfile'
-
 require 's3cp/utils'
+require 'tempfile'
 
 # Parse arguments
 options = {}
@@ -37,6 +32,10 @@ op = OptionParser.new do |opts|
     options[:headers] = h
   end
 
+  opts.on("--acl PERMISSION", "One of 'private', 'authenticated-read', 'public-read', 'public-read-write'") do |permission|
+    options[:acl] = S3CP.validate_acl(permission)
+  end
+
   opts.separator "        e.g.,"
   opts.separator "              HTTP headers: \'Content-Type: image/jpg\'"
   opts.separator "               AMZ headers: \'x-amz-acl: public-read\'"
@@ -78,7 +77,10 @@ temp.open
 
 # upload temp file
 begin
-  @s3.interface.put(bucket, key, temp, @headers)
+  s3_options = {}
+  S3CP.set_header_options(s3_options, @headers)
+  s3_options[:acl] = options[:acl]
+  @s3.buckets[bucket].objects[key].write(temp, s3_options)
   STDERR.puts "s3://#{bucket}/#{key} => #{S3CP.format_filesize(temp.size)} "
 ensure
   # cleanup

data/lib/s3cp/utils.rb

@@ -15,19 +15,37 @@
 # License for the specific language governing permissions and limitations under
 # the License.
 
+require 'rubygems'
+require 'extensions/kernel' if RUBY_VERSION =~ /1.8/
+require 'aws/s3'
+require 'optparse'
+require 'date'
+require 'highline/import'
+
 module S3CP
   extend self
 
   # Valid units for file size formatting
   UNITS = %w{B KB MB GB TB EB ZB YB BB}
 
+  LEGAL_MODS = %w{
+    private
+    public-read
+    public-read-write
+    authenticated-read
+    bucket_owner_read
+    bucket_owner_full_control
+  }
+
   # Connect to AWS S3
   def connect()
     access_key = ENV["AWS_ACCESS_KEY_ID"]     || raise("Missing environment variable AWS_ACCESS_KEY_ID")
     secret_key = ENV["AWS_SECRET_ACCESS_KEY"] || raise("Missing environment variable AWS_SECRET_ACCESS_KEY")
 
-    logger = Logger.new('/dev/null')
-    RightAws::S3.new(access_key, secret_key, :logger => logger)
+    ::AWS::S3.new(
+      :access_key_id     => access_key,
+      :secret_access_key => secret_key
+    )
   end
 
   # Parse URL and return bucket and key.
@@ -100,5 +118,67 @@ module S3CP
     end
   end
 
+  def set_header_options(options, headers)
+    return options unless headers
+
+    # legacy options that were previously passed as headers
+    # are now passed explicitly as options/metadata.
+    mappings = {
+      "Content-Type"  =>  :content_type,
+      "x-amz-acl"     =>  :acl,
+      "Cache-Control" =>  :cache_control,
+      "x-amz-storage-class" => :reduced_redundancy
+    }
+
+    lambdas = {
+      "x-amz-storage-class" => lambda { |v| (v =~ /^REDUCED_REDUNDANCY$/i) ? true : false }
+    }
+
+    remaining = headers.dup
+    headers.each do |hk, hv|
+      mappings.each do |mk, mv|
+        if hk.to_s =~ /^#{mk}$/i
+          lambda = lambdas[mk]
+          options[mv] = lambda ? lambda.call(hv) : hv
+          remaining.delete(hk)
+        end
+      end
+    end
+
+
+    options[:metadata] = remaining unless remaining.empty?
+
+    options
+  end
+
+  def validate_acl(permission)
+    if !LEGAL_MODS.include?(permission)
+      raise "Permissions must be one of the following values: #{LEGAL_MODS}"
+    end
+    permission
+  end
+end
+
+# Monkey-patch S3 object for download streaming
+# https://forums.aws.amazon.com/thread.jspa?messageID=295587
+module AWS
+
+  DEFAULT_STREAMING_CHUNK_SIZE = ENV["S3CP_STREAMING_CHUNK_SIZE"] ? ENV["S3CP_STREAMING_CHUNK_SIZE"].to_i : (512 * 1024)
+
+  class S3
+    class S3Object
+      def read_as_stream(options = nil, &blk)
+        options ||= {}
+        chunk_size = options[:chunk] || DEFAULT_STREAMING_CHUNK_SIZE
+        size = content_length
+        byte_offset = 0
+        while byte_offset < size
+          range = "bytes=#{byte_offset}-#{byte_offset + chunk_size - 1}"
+          yield read(:range => range)
+          byte_offset += chunk_size
+        end
+      end
+    end
+  end
 end
 

data/lib/s3cp/version.rb

@@ -16,5 +16,5 @@
 # the License.
 
 module S3CP
-  VERSION = "1.0.6"
+  VERSION = "1.1.0.pre.1"
 end

metadata

@@ -1,13 +1,15 @@
 --- !ruby/object:Gem::Specification 
 name: s3cp
 version: !ruby/object:Gem::Version 
-  hash: 27
-  prerelease: 
+  hash: -3842078234
+  prerelease: 6
   segments: 
   - 1
+  - 1
   - 0
-  - 6
-  version: 1.0.6
+  - pre
+  - 1
+  version: 1.1.0.pre.1
 platform: ruby
 authors: 
 - Alex Boisvert
@@ -15,7 +17,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2012-07-23 00:00:00 Z
+date: 2012-07-24 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   prerelease: false
@@ -50,40 +52,24 @@ dependencies:
   type: :runtime
 - !ruby/object:Gem::Dependency 
   prerelease: false
-  name: aboisvert_aws
+  name: aws-sdk
   version_requirements: &id003 !ruby/object:Gem::Requirement 
     none: false
     requirements: 
     - - ~>
       - !ruby/object:Gem::Version 
-        hash: 7
-        segments: 
-        - 3
-        - 0
-        - 0
-        version: 3.0.0
-  requirement: *id003
-  type: :runtime
-- !ruby/object:Gem::Dependency 
-  prerelease: false
-  name: right_http_connection
-  version_requirements: &id004 !ruby/object:Gem::Requirement 
-    none: false
-    requirements: 
-    - - ~>
-      - !ruby/object:Gem::Version 
-        hash: 27
+        hash: 13
         segments: 
         - 1
-        - 3
-        - 0
-        version: 1.3.0
-  requirement: *id004
+        - 5
+        - 7
+        version: 1.5.7
+  requirement: *id003
   type: :runtime
 - !ruby/object:Gem::Dependency 
   prerelease: false
   name: progressbar
-  version_requirements: &id005 !ruby/object:Gem::Requirement 
+  version_requirements: &id004 !ruby/object:Gem::Requirement 
     none: false
     requirements: 
     - - ~>
@@ -94,12 +80,12 @@ dependencies:
         - 10
         - 0
         version: 0.10.0
-  requirement: *id005
+  requirement: *id004
   type: :runtime
 - !ruby/object:Gem::Dependency 
   prerelease: false
   name: rspec
-  version_requirements: &id006 !ruby/object:Gem::Requirement 
+  version_requirements: &id005 !ruby/object:Gem::Requirement 
     none: false
     requirements: 
     - - ~>
@@ -110,12 +96,12 @@ dependencies:
         - 5
         - 0
         version: 2.5.0
-  requirement: *id006
+  requirement: *id005
   type: :development
 - !ruby/object:Gem::Dependency 
   prerelease: false
   name: rake
-  version_requirements: &id007 !ruby/object:Gem::Requirement 
+  version_requirements: &id006 !ruby/object:Gem::Requirement 
     none: false
     requirements: 
     - - ~>
@@ -126,7 +112,7 @@ dependencies:
         - 8
         - 7
         version: 0.8.7
-  requirement: *id007
+  requirement: *id006
   type: :development
 description: 
 email: 
@@ -157,7 +143,9 @@ files:
 - lib/s3cp/s3mod.rb
 - lib/s3cp/utils.rb
 - lib/s3cp/s3cp.rb
+- lib/s3cp/#Untitled-1#
 - lib/s3cp/s3cat.rb
+- lib/s3cp/s3buckets.rb
 - lib/s3cp/s3rm.rb
 - lib/s3cp/s3stat.rb
 - lib/s3cp/s3up.rb
@@ -167,6 +155,7 @@ files:
 - bin/s3mv
 - bin/s3rm
 - bin/s3dir
+- bin/s3buckets
 - bin/s3mod
 - bin/s3du
 - bin/s3cat