sluice 0.0.9 → 0.1.0

data/CHANGELOG

@@ -6,7 +6,7 @@ Version 0.0.8 (2013-08-14)
 --------------------------
 Added in ability to process a list of files rather than an S3 location
 
-Version 0.0.7 (2013-XX-XX)
+Version 0.0.7 (2013-07-12)
 --------------------------
 Added in upload capability
 

data/README.md

@@ -8,12 +8,12 @@ Currently Sluice provides the following very robust, very parallel S3 operations
 * File download from S3
 * File delete from S3
 * File move within S3 (from/to the same or different AWS accounts)
-* File copy within S3 (from/to the same or different AWS accounts)
+* File copy within S3 (from/to the same or different AWS accounts; optionally using a manifest)
 
-Sluice has been extracted from a pair of Ruby ETL applications built by the [SnowPlow Analytics] [snowplow-analytics] team, specifically:
+Sluice has been extracted from a pair of Ruby ETL applications built by the [Snowplow Analytics] [snowplow-analytics] team, specifically:
 
-1. [EmrEtlRunner] [emr-etl-runner], a Ruby application to run the SnowPlow ETL process on Elastic MapReduce
-2. [StorageLoader] [storage-loader], a Ruby application to load SnowPlow event files from Amazon S3 into databases such as Infobright
+1. [EmrEtlRunner] [emr-etl-runner], a Ruby application to run the Snowplow ETL process on Elastic MapReduce
+2. [StorageLoader] [storage-loader], a Ruby application to load Snowplow event files from Amazon S3 into databases such as Redshift and Postgres
 
 ## Installation 
 
@@ -21,7 +21,7 @@ Sluice has been extracted from a pair of Ruby ETL applications built by the [Sno
 
 Or in your Gemfile:
 
-    gem 'sluice', '~> 0.0.9'
+    gem 'sluice', '~> 0.1.0'
 
 ## Usage
 
@@ -32,7 +32,7 @@ Rubydoc and usage examples to come.
 To hack on Sluice locally:
 
     $ gem build sluice.gemspec
-    $ sudo gem install sluice-0.0.9.gem
+    $ sudo gem install sluice-0.1.0.gem
 
 To contribute:
 
@@ -44,11 +44,11 @@ To contribute:
 
 ## Credits and thanks
 
-Sluice was developed by [Alex Dean] [alexanderdean] ([SnowPlow Analytics] [snowplow-analytics]) and [Michael Tibben] [mtibben] ([99designs] [99designs]).
+Sluice was developed by [Alex Dean] [alexanderdean] ([Snowplow Analytics] [snowplow-analytics]) and [Michael Tibben] [mtibben] ([99designs] [99designs]).
 
 ## Copyright and license
 
-Sluice is copyright 2012 SnowPlow Analytics Ltd.
+Sluice is copyright 2012-2013 Snowplow Analytics Ltd.
 
 Licensed under the [Apache License, Version 2.0] [license] (the "License");
 you may not use this software except in compliance with the License.
@@ -66,7 +66,7 @@ limitations under the License.
 [mtibben]: https://github.com/mtibben
 [99designs]: http://99designs.com
 
-[emr-etl-runner]: https://github.com/snowplow/snowplow/tree/master/3-etl/emr-etl-runner
+[emr-etl-runner]: https://github.com/snowplow/snowplow/tree/master/3-enrich/emr-etl-runner
 [storage-loader]: https://github.com/snowplow/snowplow/tree/master/4-storage/storage-loader
 
 [license]: http://www.apache.org/licenses/LICENSE-2.0

data/lib/sluice.rb

@@ -19,5 +19,5 @@ require 'sluice/storage/s3'
 
 module Sluice
   NAME    = "sluice"
-  VERSION = "0.0.9"
+  VERSION = "0.1.0"
 end

data/lib/sluice/storage/s3.rb

@@ -13,10 +13,14 @@
 # Copyright:: Copyright (c) 2012 SnowPlow Analytics Ltd
 # License::   Apache License Version 2.0
 
+require 'set'
 require 'tmpdir'
 require 'fog'
 require 'thread'
 
+require 'contracts'
+include Contracts
+
 module Sluice
   module Storage
     module S3
@@ -29,12 +33,18 @@ module Sluice
       RETRIES = 3      # Attempts
       RETRY_WAIT = 10  # Seconds
 
+      # Aliases for Contracts
+      FogStorage = Fog::Storage::AWS::Real
+      FogFile = Fog::Storage::AWS::File
+
       # Class to describe an S3 location
       # TODO: if we are going to impose trailing line-breaks on
       # buckets, maybe we should make that clearer?
       class Location
         attr_reader :bucket, :dir, :s3location
 
+        # Location constructor
+        #
         # Parameters:
         # +s3location+:: the s3 location config string e.g. "bucket/directory"
         def initialize(s3_location)
@@ -60,6 +70,108 @@ module Sluice
         end
       end
 
+      # Legitimate manifest scopes:
+      # 1. :filename - store only the filename
+      #                in the manifest
+      # 2. :relpath  - store the relative path
+      #                to the file in the manifest
+      # 3. :abspath  - store the absolute path
+      #                to the file in the manifest
+      # 4. :bucket   - store bucket PLUS absolute
+      #                path to the file in the manifest
+      #
+      # TODO: add support for 2-4. Currently only 1 supported 
+      class ManifestScope
+
+        @@scopes = Set::[](:filename) # TODO add :relpath, :abspath, :bucket
+
+        def self.valid?(val)
+          val.is_a?(Symbol) &&
+            @@scopes.include?(val)
+        end
+      end
+
+      # Class to read and maintain a manifest.
+      class Manifest
+        attr_reader :s3_location, :scope, :manifest_file
+
+        # Manifest constructor
+        #
+        # Parameters:
+        # +path+:: full path to the manifest file
+        # +scope+:: whether file entries in the
+        #           manifest should be scoped to
+        #           filename, relative path, absolute
+        #           path, or absolute path and bucket
+        Contract Location, ManifestScope => nil
+        def initialize(s3_location, scope)
+          @s3_location = s3_location
+          @scope = scope
+          @manifest_file = "%ssluice-%s-manifest" % [s3_location.dir_as_path, scope.to_s]
+          nil
+        end
+
+        # Get the current file entries in the manifest
+        #
+        # Parameters:
+        # +s3+:: A Fog::Storage s3 connection
+        #
+        # Returns an Array of filenames as Strings
+        Contract FogStorage => ArrayOf[String]
+        def get_entries(s3)
+
+          manifest = self.class.get_manifest(s3, @s3_location, @manifest_file)
+          if manifest.nil?
+            return []
+          end
+
+          manifest.body.split("\n").reject(&:empty?)
+        end
+
+        # Add (i.e. append) the following file entries
+        # to the manifest
+        # Files listed previously in the manifest will
+        # be kept in the new manifest file.
+        #
+        # Parameters:
+        # +s3+:: A Fog::Storage s3 connection
+        # +entries+:: an Array of filenames as Strings
+        #
+        # Returns all entries now in the manifest
+        Contract FogStorage, ArrayOf[String] => ArrayOf[String]
+        def add_entries(s3, entries)
+
+          existing = get_entries(s3)
+          filenames = entries.map { |filepath|
+            File.basename(filepath)
+          } # TODO: update when non-filename-based manifests supported
+          all = (existing + filenames)
+
+          manifest = self.class.get_manifest(s3, @s3_location, @manifest_file)
+          body = all.join("\n")
+          if manifest.nil?
+            bucket = s3.directories.get(s3_location.bucket).files.create(
+              :key  => @manifest_file,
+              :body => body
+            )
+          else
+            manifest.body = body
+            manifest.save
+          end
+
+          all
+        end
+
+        private
+
+        # Helper to get the manifest file
+        # Contract FogStorage, Location, String => Or[FogFile, nil] TODO: fix this. Expected: File, Actual: <Fog::Storage::AWS::File>
+        def self.get_manifest(s3, s3_location, filename)
+          s3.directories.get(s3_location.bucket, prefix: s3_location.dir).files.get(filename) # TODO: break out into new generic get_file() procedure
+        end
+
+      end
+
       # Helper function to instantiate a new Fog::Storage
       # for S3 based on our config options
       #
@@ -67,6 +179,7 @@ module Sluice
       # +region+:: Amazon S3 region we will be working with
       # +access_key_id+:: AWS access key ID
       # +secret_access_key+:: AWS secret access key
+      Contract String, String, String => FogStorage
       def new_fog_s3_from(region, access_key_id, secret_access_key)
         fog = Fog::Storage.new({
           :provider => 'AWS',
@@ -164,7 +277,7 @@ module Sluice
       def download_files(s3, from_files_or_loc, to_directory, match_regex='.+')
 
         puts "  downloading #{describe_from(from_files_or_loc)} to #{to_directory}"
-        process_files(:download, s3, from_files_or_loc, match_regex, to_directory)
+        process_files(:download, s3, from_files_or_loc, [], match_regex, to_directory)
       end
       module_function :download_files    
 
@@ -177,7 +290,7 @@ module Sluice
       def delete_files(s3, from_files_or_loc, match_regex='.+')
 
         puts "  deleting #{describe_from(from_files_or_loc)}"
-        process_files(:delete, s3, from_files_or_loc, match_regex)
+        process_files(:delete, s3, from_files_or_loc, [], match_regex)
       end
       module_function :delete_files
 
@@ -203,12 +316,14 @@ module Sluice
       def copy_files_inter(from_s3, to_s3, from_location, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)            
 
         puts "  copying inter-account #{describe_from(from_location)} to #{to_location}"
+        processed = []
         Dir.mktmpdir do |t|
           tmp = Sluice::Storage.trail_slash(t)
-          download_files(from_s3, from_location, tmp, match_regex)
+          processed = download_files(from_s3, from_location, tmp, match_regex)
           upload_files(to_s3, tmp, to_location, '**/*') # Upload all files we downloaded
         end
 
+        processed
       end
       module_function :copy_files_inter
 
@@ -224,10 +339,37 @@ module Sluice
       def copy_files(s3, from_files_or_loc, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)
 
         puts "  copying #{describe_from(from_files_or_loc)} to #{to_location}"
-        process_files(:copy, s3, from_files_or_loc, match_regex, to_location, alter_filename_lambda, flatten)
+        process_files(:copy, s3, from_files_or_loc, [], match_regex, to_location, alter_filename_lambda, flatten)
       end
       module_function :copy_files
 
+      # Copies files between S3 locations maintaining a manifest to
+      # avoid copying a file which was copied previously.
+      #
+      # Useful in scenarios such as:
+      # 1. You would like to do a move but only have read permission
+      #    on the source bucket
+      # 2. You would like to do a move but some other process needs
+      #    to use the files after you
+      #
+      # +s3+:: A Fog::Storage s3 connection
+      # +manifest+:: A Sluice::Storage::S3::Manifest object
+      # +from_files_or_loc+:: Array of filepaths or Fog::Storage::AWS::File objects, or S3Location to copy files from
+      # +to_location+:: S3Location to copy files to
+      # +match_regex+:: a regex string to match the files to copy
+      # +alter_filename_lambda+:: lambda to alter the written filename
+      # +flatten+:: strips off any sub-folders below the from_location
+      def copy_files_manifest(s3, manifest, from_files_or_loc, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)
+
+        puts "  copying with manifest #{describe_from(from_files_or_loc)} to #{to_location}"
+        ignore = manifest.get_entries(s3) # Files to leave untouched
+        processed = process_files(:copy, s3, from_files_or_loc, ignore, match_regex, to_location, alter_filename_lambda, flatten)
+        manifest.add_entries(s3, processed)
+
+        processed
+      end
+      module_function :copy_files_manifest
+
       # Moves files between S3 locations in two different accounts
       #
       # Implementation is as follows:
@@ -248,13 +390,15 @@ module Sluice
       def move_files_inter(from_s3, to_s3, from_location, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)
 
         puts "  moving inter-account #{describe_from(from_location)} to #{to_location}"
+        processed = []
         Dir.mktmpdir do |t|
           tmp = Sluice::Storage.trail_slash(t)
-          download_files(from_s3, from_location, tmp, match_regex)
+          processed = download_files(from_s3, from_location, tmp, match_regex)
           upload_files(to_s3, tmp, to_location, '**/*') # Upload all files we downloaded
           delete_files(from_s3, from_location, '.+') # Delete all files we downloaded
         end
 
+        processed
       end
       module_function :move_files_inter
 
@@ -270,7 +414,7 @@ module Sluice
       def move_files(s3, from_files_or_loc, to_location, match_regex='.+', alter_filename_lambda=false, flatten=false)
 
         puts "  moving #{describe_from(from_files_or_loc)} to #{to_location}"
-        process_files(:move, s3, from_files_or_loc, match_regex, to_location, alter_filename_lambda, flatten)
+        process_files(:move, s3, from_files_or_loc, [], match_regex, to_location, alter_filename_lambda, flatten)
       end
       module_function :move_files
 
@@ -284,7 +428,7 @@ module Sluice
       def upload_files(s3, from_files_or_dir, to_location, match_glob='*')
 
         puts "  uploading #{describe_from(from_files_or_dir)} to #{to_location}"
-        process_files(:upload, s3, from_files_or_dir, match_glob, to_location)
+        process_files(:upload, s3, from_files_or_dir, [], match_glob, to_location)
       end
       module_function :upload_files
 
@@ -357,13 +501,14 @@ module Sluice
       #
       # Parameters:
       # +operation+:: Operation to perform. :download, :upload, :copy, :delete, :move supported
+      # +ignore+:: Array of filenames to ignore (used by manifest code)
       # +s3+:: A Fog::Storage s3 connection
       # +from_files_or_dir_or_loc+:: Array of filepaths or Fog::Storage::AWS::File objects, local directory or S3Location to process files from
       # +match_regex_or_glob+:: a regex or glob string to match the files to process
       # +to_loc_or_dir+:: S3Location or local directory to process files to
       # +alter_filename_lambda+:: lambda to alter the written filename
       # +flatten+:: strips off any sub-folders below the from_loc_or_dir
-      def process_files(operation, s3, from_files_or_dir_or_loc, match_regex_or_glob='.+', to_loc_or_dir=nil, alter_filename_lambda=false, flatten=false)
+      def process_files(operation, s3, from_files_or_dir_or_loc, ignore=[], match_regex_or_glob='.+', to_loc_or_dir=nil, alter_filename_lambda=false, flatten=false)
 
         # Validate that the file operation makes sense
         case operation
@@ -401,6 +546,7 @@ module Sluice
         mutex = Mutex.new
         complete = false
         marker_opts = {}
+        processed_files = [] # For manifest updating, determining if any files were moved etc
 
         # If an exception is thrown in a thread that isn't handled, die quickly
         Thread.abort_on_exception = true
@@ -410,7 +556,8 @@ module Sluice
 
           # Each thread pops a file off the files_to_process array, and moves it.
           # We loop until there are no more files
-          threads << Thread.new do
+          threads << Thread.new(i) do |thread_idx|
+
             loop do
               file = false
               filepath = false
@@ -418,7 +565,7 @@ module Sluice
               from_path = false
               match = false
 
-              # Critical section:
+              # First critical section:
               # only allow one thread to modify the array at any time
               mutex.synchronize do
 
@@ -476,6 +623,7 @@ module Sluice
                             else
                               filepath.match(match_regex_or_glob)
                             end
+
                   end
                 end
               end
@@ -485,6 +633,8 @@ module Sluice
 
               # Name file
               basename = get_basename(filepath)
+              break if ignore.include?(basename) # Don't process if in our leave list
+
               if alter_filename_lambda.class == Proc
                 filename = alter_filename_lambda.call(basename)
               else
@@ -497,23 +647,23 @@ module Sluice
               when :upload
                 source = "#{filepath}"
                 target = name_file(filepath, filename, from_path, to_loc_or_dir.dir_as_path, flatten)
-                puts "    UPLOAD #{source} +-> #{to_loc_or_dir.bucket}/#{target}"                
+                puts "(t#{thread_idx})    UPLOAD #{source} +-> #{to_loc_or_dir.bucket}/#{target}"                
               when :download
                 source = "#{from_bucket}/#{filepath}"
                 target = name_file(filepath, filename, from_path, to_loc_or_dir, flatten)
-                puts "    DOWNLOAD #{source} +-> #{target}"
+                puts "(t#{thread_idx})    DOWNLOAD #{source} +-> #{target}"
               when :move
                 source = "#{from_bucket}/#{filepath}"
                 target = name_file(filepath, filename, from_path, to_loc_or_dir.dir_as_path, flatten)
-                puts "    MOVE #{source} -> #{to_loc_or_dir.bucket}/#{target}"
+                puts "(t#{thread_idx})    MOVE #{source} -> #{to_loc_or_dir.bucket}/#{target}"
               when :copy
                 source = "#{from_bucket}/#{filepath}"
                 target = name_file(filepath, filename, from_path, to_loc_or_dir.dir_as_path, flatten)
-                puts "    COPY #{source} +-> #{to_loc_or_dir.bucket}/#{target}"
+                puts "(t#{thread_idx})    COPY #{source} +-> #{to_loc_or_dir.bucket}/#{target}"
               when :delete
                 source = "#{from_bucket}/#{filepath}"
                 # No target
-                puts "    DELETE x #{source}" 
+                puts "(t#{thread_idx})    DELETE x #{source}" 
               end
 
               # Upload is a standalone operation vs move/copy/delete
@@ -555,6 +705,12 @@ module Sluice
                   "      x #{source}",
                   "Problem destroying #{filepath}. Retrying.")
               end
+
+              # Second critical section: we need to update
+              # processed_files in a thread-safe way
+              mutex.synchronize do
+                processed_files << filepath
+              end
             end
           end
         end
@@ -562,6 +718,7 @@ module Sluice
         # Wait for threads to finish
         threads.each { |aThread|  aThread.join }
 
+        processed_files # Return the processed files
       end
       module_function :process_files
 
@@ -600,7 +757,7 @@ module Sluice
           sleep(RETRY_WAIT)  # Give us a bit of time before retrying
           i += 1
           retry
-        end        
+        end
       end
       module_function :retry_x
 

data/sluice.gemspec

@@ -35,4 +35,5 @@ Gem::Specification.new do |gem|
 
   # Dependencies
   gem.add_dependency 'fog', '~> 1.14.0'
+  gem.add_dependency 'contracts', '~> 0.2.3'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sluice
 version: !ruby/object:Gem::Version
-  version: 0.0.9
+  version: 0.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-14 00:00:00.000000000 Z
+date: 2013-09-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog
@@ -28,6 +28,22 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 1.14.0
+- !ruby/object:Gem::Dependency
+  name: contracts
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.2.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.2.3
 description: A Ruby gem to help you build ETL processes involving Amazon S3. Uses
   Fog
 email:
@@ -67,7 +83,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: Ruby toolkit for cloud-friendly ETL