smooth_s3 0.1.0

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+gem "s3", "~> 0.3.9"
+gem "jeweler"

data/Gemfile.lock

@@ -0,0 +1,19 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    git (1.2.5)
+    jeweler (1.6.4)
+      bundler (~> 1.0)
+      git (>= 1.2.5)
+      rake
+    proxies (0.2.1)
+    rake (0.9.2.2)
+    s3 (0.3.9)
+      proxies (~> 0.2.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  jeweler
+  s3 (~> 0.3.9)

data/README.rdoc

@@ -0,0 +1,19 @@
+= smooth_s3
+
+Description goes here.
+
+== Contributing to smooth_s3
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+== Copyright
+
+Copyright (c) 2011 Nicholas Brochu. See LICENSE.txt for
+further details.
+

data/README.textile

@@ -0,0 +1,111 @@
+h1. Smooth S3
+
+"http://github.com/nbrochu/smooth_s3":http://github.com/nbrochu/smooth_s3
+
+h2. Summary
+
+A user-friendly superset of the S3 gem geared towards file system backup operations.
+
+h2. Description
+
+Smooth S3 is a user-friendly superset of the S3 gem geared towards file system backup operations. It greatly simplifies regular file uploads to S3 by using Convention over Configuration™. The library also adds new features such as directory syncronization and timestamped uploads, which should come in real handy to anyone doing backup scripts on a regular basis. A decent amount of control is left to the developer: You can specify a prefix path to use with any upload and provide your own timestamp formats if desired.
+
+The goal with Smooth S3 is to facilitate and simplify your S3 uploads. It is a library focused on the file system, so no integration with MySQL, third-party services or anything like that. Nothing prevents you from doing a _mysqldump_ and uploading the results in the same script using Smooth S3 though ;)
+
+h2. Installation
+
+This library has been developed against and designed for MRI 1.9.2 in a UNIX environment. It should be compatible with 1.8.6+, as well as with other Ruby implementations, but no guarantees. Probably not compatible with Windows environments.
+
+<pre>
+Using RubyGems - gem install smooth_s3
+Using Bundler - gem "smooth_s3" in Gemfile, then bundle install
+</pre>
+
+h2. Overview
+
+Before running upload operations using Smooth S3, you will need to initialize the service using your AWS credentials.
+
+Once that is done you can use the following methods:
+
+* upload() - Regular file upload. Can specify multiple files at once. Directory structure is not preserved, only the file name.
+
+* sync_directory() - Uploads the entire content of a directory and its subdirectories. Preserves folder directory structure inside S3.
+ 
+* timestamped_upload() - Like a regular upload. Files uploaded this way have a timestamp added in front of their names. Default timestamp format: YYYYmmddHHMMSS  
+
+* timestamped_directory_sync() Like a regular directory sync. Provided directory has a timestamp added in front of its name. Default timestamp format: YYYYmmddHHMMS
+
+Each of the 4 methods above also has a bang(!) version available. The difference between the regular and bang version is that the former won't overwrite existing files in the selected bucket, while the latter will.
+
+You can also specify a prefix as an option to the above methods. This will be inserted in front of what would have been the normal path on S3. For example, you upload _'test.rb'_ with the prefix _'path/to'_. It will show up on S3 as _'path/to/test.rb'_
+
+That's it! Code examples in the next section.
+
+h2. How to use
+
+<pre>
+require 'rubygems' # if using 1.8
+require 'smooth_s3'
+
+# Initialize the service
+# Params: :aws_key, :aws_secret(, :ssl => false)
+ 
+@service = SmoothS3::Service.new(:aws_key => "MY_AWS_ACCESS_KEY", :aws_secret => "MY_AWS_SECRET_KEY")
+
+
+# upload()
+# Params: bucket_name, files(, :prefix)
+
+@service.upload("my_test_bucket", "test.rb")
+@service.upload!("my_test_bucket", ["here.rb", "../parent.rb"], :prefix => "prefix/to/files")
+
+
+# sync_directory()
+# Params: bucket_name, directory(, :prefix)
+
+@service.sync_directory("my_test_bucket", "../reports")
+@service.sync_directory!("my_test_bucket", "data", :prefix => "prefix/to/dir")
+
+
+# timestamped_upload()
+# Params: bucket_name, files(, :prefix, :timestamp_type => :epoch/:strftime(default), :timestamp_format => "%Y%m%d%H%M%S" )
+
+@service.timestamped_upload("my_test_bucket", "test.rb")
+@service.timestamped_upload!("my_test_bucket", "test.rb", :timestamp_type => :epoch)
+@service.timestamped_upload!("my_test_bucket", "test.rb", :timestamp_type => :strftime, :timestamp_format => "%Y%m%d")
+
+
+# timestamped_directory_sync()
+# Params: bucket_name, directory(, :prefix, :timestamp_type => :epoch/:strftime(default), :timestamp_format => "%Y%m%d%H%M%S" )
+
+@service.timestamped_directory_sync("my_test_bucket", "data")
+@service.timestamped_directory_sync!("my_test_bucket", "data", :timestamp_type => :epoch)
+@service.timestamped_directory_sync!("my_test_bucket", "data", :timestamp_type => :strftime, :timestamp_format => "%Y%m%d")
+</pre>
+
+h2. Credits
+
+* Jakub Kuźma for his great S3 library, often overshadowed by the popular but old AWS-S3.
+
+h2. LICENSE
+
+Copyright (c) 2011 - Nicholas Brochu
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/smooth_s3/bucket.rb

@@ -0,0 +1,58 @@
+module SmoothS3
+  class Bucket
+    
+    def self.exists?(bucket, service)
+      service.buckets.include? bucket
+    end
+
+    def self.file_exists?(file, bucket, service)
+      b = service.proxy_service.buckets.find_first(bucket)
+
+      begin
+        b.objects.find_first(file)
+        return true
+      rescue S3::Error::NoSuchKey
+        return false
+      end
+    end
+
+    def self.select(bucket, service)
+      Bucket.create(bucket, service) unless Bucket.exists?(bucket, service)
+    end
+
+    def self.create(bucket_name, service)
+      begin
+        new_bucket = service.proxy_service.buckets.build(bucket_name)
+        new_bucket.save
+      rescue S3::Error::BucketAlreadyExists
+        raise SmoothS3::Error, "A bucket named '#{bucket_name}' already exists in the Global S3 Namespace. Please select one of you existing buckets or try a new name."
+      end
+    end
+
+    def self.store_file(file, remote_file, bucket, service, prefix, overwrite)
+      b = service.proxy_service.buckets.find_first(bucket)
+
+      if prefix
+        remote_file = prefix + remote_file  if prefix =~ /\/$/
+        remote_file = prefix + "/" + remote_file unless prefix =~ /\/$/
+      end
+
+      unless overwrite == true
+        if Bucket.file_exists?(remote_file, bucket, service)
+          puts "'#{remote_file}' already exists on S3 bucket named '#{bucket}'. Use the bang(!) version of the method to overwrite."
+          return
+        end
+      end
+
+      bo = b.objects.build(remote_file)
+      bo.content = open(file)
+      
+      if bo.save
+        puts "'#{file}' was uploaded to S3 bucket '#{bucket}' under the name '#{remote_file}'."
+      else
+        puts "There was a problem trying to upload '#{file}' to S3"
+      end
+    end
+
+  end
+end

data/lib/smooth_s3/error.rb

@@ -0,0 +1,3 @@
+module SmoothS3
+  class Error < StandardError ; end
+end

data/lib/smooth_s3/service.rb

@@ -0,0 +1,63 @@
+module SmoothS3
+  class Service
+    attr_reader :aws_key, :aws_secret, :ssl, :proxy_service
+
+    def initialize(opts={})
+      @aws_key = opts.delete(:aws_key)
+      @aws_secret = opts.delete(:aws_secret)
+
+      @ssl = opts.delete(:ssl) == true ? true : false
+      @proxy_service = S3::Service.new(:access_key_id => @aws_key, :secret_access_key => @aws_secret, :use_ssl => @ssl)
+
+      test_connection
+    end
+
+    def upload(bucket, files, options={})
+      Uploader.upload(self, bucket, files, options)
+    end
+
+    def upload!(bucket, files, options={})
+      Uploader.upload!(self, bucket, files, options)
+    end
+
+    def sync_directory(bucket, directory, options={})
+      Uploader.sync_directory(self, bucket, directory, options)
+    end
+
+    def sync_directory!(bucket, directory, options={})
+      Uploader.sync_directory!(self, bucket, directory, options)
+    end
+
+    def timestamped_upload(bucket, files, options={})
+      Uploader.timestamped_upload(self, bucket, files, options)
+    end
+
+    def timestamped_upload!(bucket, files, options={})
+      Uploader.timestamped_upload!(self, bucket, files, options)
+    end
+
+    def timestamped_directory_sync(bucket, directory, options={})
+      Uploader.timestamped_directory_sync(self, bucket, directory, options)
+    end
+
+    def timestamped_directory_sync!(bucket, directory, options={})
+      Uploader.timestamped_directory_sync!(self, bucket, directory, options)
+    end
+
+    # Utility Methods
+    def buckets
+      @proxy_service.buckets.map { |b| b.name }
+    end
+
+    private
+
+      def test_connection
+        begin
+          @proxy_service.send(:service_request, :get)
+        rescue S3::Error::SignatureDoesNotMatch => e
+          raise Error, "Invalid AWS Key and/or Secret provided."
+        end
+      end
+
+  end
+end

data/lib/smooth_s3/uploader.rb

@@ -0,0 +1,129 @@
+module SmoothS3
+  class Uploader
+
+    def self.upload(service, bucket, files, options={})
+      options[:overwrite] = false unless options[:overwrite]
+      Bucket.select(bucket, service)
+      
+      valid_files = Uploader.validate_files(files)
+      valid_files.each do |vf|
+        Bucket.store_file(vf, vf.split("/")[-1], bucket, service, options[:prefix], options[:overwrite])
+      end
+    end
+
+    def self.upload!(service, bucket, files, options={})
+      options.merge!(:overwrite => true)
+      Uploader.upload(service, bucket, files, options)
+    end
+
+    def self.sync_directory(service, bucket, directory, options={})
+      options[:overwrite] = false unless options[:overwrite]
+      Bucket.select(bucket, service)
+
+      valid_files = Uploader.validate_files_in_directory(directory)
+      valid_files.each do |vf|
+        Bucket.store_file(vf[0], vf[1], bucket, service, options[:prefix], options[:overwrite])
+      end
+    end
+
+    def self.sync_directory!(service, bucket, directory, options={})
+      options.merge!(:overwrite => true)
+      Uploader.sync_directory(service, bucket, directory, options)
+    end
+
+    def self.timestamped_upload(service, bucket, files, options={})
+      options[:overwrite] = false unless options[:overwrite]
+      Bucket.select(bucket, service)
+
+      if options[:timestamp_type] == :epoch
+        timestamp = Time.now.strftime("%s")
+      elsif options[:timestamp_type] == :strftime
+        if options[:timestamp_format]
+          timestamp = Time.now.strftime(options[:timestamp_format])
+        else
+          timestamp = Uploader.default_timestamp
+        end
+      else
+        timestamp = Uploader.default_timestamp
+      end
+
+      valid_files = Uploader.validate_files(files)
+      valid_files.each do |vf|
+        Bucket.store_file(vf, timestamp + "_" + vf.split("/")[-1], bucket, service, options[:prefix], options[:overwrite])
+      end
+    end
+
+    def self.timestamped_upload!(service, bucket, files, options={})
+      options.merge!(:overwrite => true)
+      Uploader.timestamped_upload(service, bucket, files, options)
+    end
+
+    def self.timestamped_directory_sync(service, bucket, directory, options={})
+      options[:overwrite] = false unless options[:overwrite]
+      Bucket.select(bucket, service)
+
+      if options[:timestamp_type] == :epoch
+        timestamp = Time.now.strftime("%s")
+      elsif options[:timestamp_type] == :strftime
+        if options[:timestamp_format]
+          timestamp = Time.now.strftime(options[:timestamp_format])
+        else
+          timestamp = Uploader.default_timestamp
+        end
+      else
+        timestamp = Uploader.default_timestamp
+      end
+
+      valid_files = Uploader.validate_files_in_directory(directory)
+      valid_files.each do |vf|
+        Bucket.store_file(vf[0], timestamp + "_" + vf[1], bucket, service, options[:prefix], options[:overwrite])
+      end
+    end
+
+    def self.timestamped_directory_sync!(service, bucket, directory, options={})
+      options.merge!(:overwrite => true)
+      Uploader.timestamped_directory_sync(service, bucket, directory, options)
+    end
+
+    private
+      
+      def self.validate_files(files)
+        valid_files = []
+
+        files = [files] if files.class == String
+        files.each do |f|
+          begin
+            file = File.open(f, "r")
+            valid_files << f
+            file.close
+          rescue Errno::ENOENT
+            puts "'#{f}' is an invalid file. Skipping."
+          end
+        end
+
+        valid_files
+      end
+
+      def self.validate_files_in_directory(directory)
+        valid_files = []
+
+        begin
+          Find.find(directory) do |f|
+            next if File.directory?(f)
+
+            file_name = directory.split("/")[-1] + "/" + f.gsub(directory + "/", "")
+            valid_files << [f, file_name]
+          end
+        rescue Errno::ENOENT
+          raise SmoothS3::Error, "'#{directory}' is not a valid directory."
+        end
+
+        valid_files
+      end
+
+      def self.default_timestamp
+        Time.now.strftime("%Y%m%d%H%M%S")
+      end
+
+  end
+end

data/lib/smooth_s3.rb

@@ -0,0 +1,15 @@
+$LOAD_PATH.unshift(File.dirname(__FILE__)) unless $LOAD_PATH.include?(File.dirname(__FILE__))
+
+require 'rubygems'
+require 's3'
+
+require 'find'
+
+require 'smooth_s3/service.rb'
+require 'smooth_s3/bucket.rb'
+require 'smooth_s3/uploader.rb'
+require 'smooth_s3/error.rb'
+
+module SmoothS3
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,86 @@
+--- !ruby/object:Gem::Specification
+name: smooth_s3
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Nicholas Brochu
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2011-12-08 00:00:00.000000000Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: s3
+  requirement: &2165830360 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.3.9
+  type: :runtime
+  prerelease: false
+  version_requirements: *2165830360
+- !ruby/object:Gem::Dependency
+  name: jeweler
+  requirement: &2165829340 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *2165829340
+description: A user-friendly superset of the S3 gem geared towards file system backup
+  operations. Simplifies standard actions such as basic uploads, for example allowing
+  multiple files to be uploaded in one operation and adds new functionality such as
+  directory syncs and timestamped uploads.
+email: info@nicholasbrochu.com
+executables: []
+extensions: []
+extra_rdoc_files:
+- README.rdoc
+- README.textile
+files:
+- Gemfile
+- Gemfile.lock
+- README.textile
+- VERSION
+- lib/smooth_s3.rb
+- lib/smooth_s3/bucket.rb
+- lib/smooth_s3/error.rb
+- lib/smooth_s3/service.rb
+- lib/smooth_s3/uploader.rb
+- README.rdoc
+homepage: http://github.com/nbrochu/smooth_s3
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+      segments:
+      - 0
+      hash: 3856766489077473240
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.10
+signing_key: 
+specification_version: 3
+summary: A user-friendly superset of the S3 gem geared towards file system backup
+  operations.
+test_files: []