sluice 0.0.8 → 0.0.9

data/CHANGELOG

@@ -1,3 +1,7 @@
+Version 0.0.9 (2013-08-14)
+--------------------------
+Added ability to move and copy files between two different AWS accounts
+
 Version 0.0.8 (2013-08-14)
 --------------------------
 Added in ability to process a list of files rather than an S3 location

data/README.md

@@ -2,7 +2,13 @@
 
 Sluice is a Ruby gem (built with [Bundler] [bundler]) to help you build cloud-friendly ETL (extract, transform, load) processes.
 
-**Currently it does one thing: supports very robust, very parallel copy/delete/move of S3 files from one bucket to another.**
+Currently Sluice provides the following very robust, very parallel S3 operations:
+
+* File upload to S3
+* File download from S3
+* File delete from S3
+* File move within S3 (from/to the same or different AWS accounts)
+* File copy within S3 (from/to the same or different AWS accounts)
 
 Sluice has been extracted from a pair of Ruby ETL applications built by the [SnowPlow Analytics] [snowplow-analytics] team, specifically:
 
@@ -15,7 +21,7 @@ Sluice has been extracted from a pair of Ruby ETL applications built by the [Sno
 
 Or in your Gemfile:
 
-    gem 'sluice', '~> 0.0.6'
+    gem 'sluice', '~> 0.0.9'
 
 ## Usage
 
@@ -26,7 +32,7 @@ Rubydoc and usage examples to come.
 To hack on Sluice locally:
 
     $ gem build sluice.gemspec
-    $ sudo gem install sluice-0.0.6.gem
+    $ sudo gem install sluice-0.0.9.gem
 
 To contribute:
 

data/lib/sluice/storage/s3.rb

@@ -13,6 +13,7 @@
 # Copyright:: Copyright (c) 2012 SnowPlow Analytics Ltd
 # License::   Apache License Version 2.0
 
+require 'tmpdir'
 require 'fog'
 require 'thread'
 
@@ -180,6 +181,37 @@ module Sluice
       end
       module_function :delete_files
 
+      # Copies files between S3 locations in two different accounts
+      #
+      # Implementation is as follows:
+      # 1. Concurrent download of all files from S3 source to local tmpdir
+      # 2. Concurrent upload of all files from local tmpdir to S3 target
+      #
+      # In other words, the download and upload are not interleaved (which is
+      # inefficient because upload speeds are much lower than download speeds)
+      #
+      # In other words, the download and upload are not interleaved (which
+      # is inefficient because upload speeds are much lower than download speeds)
+      #
+      # +from_s3+:: A Fog::Storage s3 connection for accessing the from S3Location
+      # +to_s3+:: A Fog::Storage s3 connection for accessing the to S3Location
+      # +from_location+:: S3Location to copy files from
+      # +to_location+:: S3Location to copy files to
+      # +match_regex+:: a regex string to match the files to move
+      # +alter_filename_lambda+:: lambda to alter the written filename
+      # +flatten+:: strips off any sub-folders below the from_location
+      def copy_files_inter(from_s3, to_s3, from_location, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)            
+
+        puts "  copying inter-account #{describe_from(from_location)} to #{to_location}"
+        Dir.mktmpdir do |t|
+          tmp = Sluice::Storage.trail_slash(t)
+          download_files(from_s3, from_location, tmp, match_regex)
+          upload_files(to_s3, tmp, to_location, '**/*') # Upload all files we downloaded
+        end
+
+      end
+      module_function :copy_files_inter
+
       # Copies files between S3 locations concurrently
       #
       # Parameters:
@@ -196,6 +228,36 @@ module Sluice
       end
       module_function :copy_files
 
+      # Moves files between S3 locations in two different accounts
+      #
+      # Implementation is as follows:
+      # 1. Concurrent download of all files from S3 source to local tmpdir
+      # 2. Concurrent upload of all files from local tmpdir to S3 target
+      # 3. Concurrent deletion of all files from S3 source
+      #
+      # In other words, the three operations are not interleaved (which is
+      # inefficient because upload speeds are much lower than download speeds)
+      #
+      # +from_s3+:: A Fog::Storage s3 connection for accessing the from S3Location
+      # +to_s3+:: A Fog::Storage s3 connection for accessing the to S3Location      
+      # +from_location+:: S3Location to move files from
+      # +to_location+:: S3Location to move files to
+      # +match_regex+:: a regex string to match the files to move
+      # +alter_filename_lambda+:: lambda to alter the written filename
+      # +flatten+:: strips off any sub-folders below the from_location
+      def move_files_inter(from_s3, to_s3, from_location, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)
+
+        puts "  moving inter-account #{describe_from(from_location)} to #{to_location}"
+        Dir.mktmpdir do |t|
+          tmp = Sluice::Storage.trail_slash(t)
+          download_files(from_s3, from_location, tmp, match_regex)
+          upload_files(to_s3, tmp, to_location, '**/*') # Upload all files we downloaded
+          delete_files(from_s3, from_location, '.+') # Delete all files we downloaded
+        end
+
+      end
+      module_function :move_files_inter
+
       # Moves files between S3 locations concurrently
       #
       # Parameters:
@@ -326,7 +388,7 @@ module Sluice
           globbed = true
         # Otherwise if it's an upload, we can glob now
         elsif operation == :upload
-          files_to_process = Dir.glob(File.join(from_files_or_dir_or_loc, match_regex_or_glob))
+          files_to_process = glob_files(from_files_or_dir_or_loc, match_regex_or_glob)
           globbed = true
         # Otherwise we'll do threaded globbing later...
         else
@@ -503,6 +565,22 @@ module Sluice
       end
       module_function :process_files
 
+      # A helper function to list all files
+      # recursively in a folder
+      #
+      # Parameters:
+      # +dir+:: Directory to list files recursively
+      # +match_regex+:: a regex string to match the files to copy      
+      #
+      # Returns array of files (no sub-directories)
+      def glob_files(dir, glob)
+        Dir.glob(File.join(dir, glob)).select { |f|
+          File.file?(f) # Drop sub-directories
+        }
+      end
+      module_function :glob_files
+
+
       # A helper function to attempt to run a
       # function retries times
       #

data/lib/sluice.rb

@@ -19,5 +19,5 @@ require 'sluice/storage/s3'
 
 module Sluice
   NAME    = "sluice"
-  VERSION = "0.0.8"
+  VERSION = "0.0.9"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sluice
 version: !ruby/object:Gem::Version
-  version: 0.0.8
+  version: 0.0.9
   prerelease: 
 platform: ruby
 authors: