s3snapshot 0.0.11 → 0.0.12

data/Readme.markdown

@@ -6,6 +6,15 @@ This gem is designed to sync an immutable directory to a timestamped prefix on A
 
 		gem install s3snapshot
 
+# Building the gem
+
+From the directory the project is checked out in, run:
+
+	gem build s3snapshot.gemspec
+	# to install the gem you just built, run the following two commands.
+	gem uninstall -x s3snapshot
+	gem install s3snapshot-*.gem
+
 # Operations
 
 * Backup a directory
@@ -14,11 +23,11 @@ This gem is designed to sync an immutable directory to a timestamped prefix on A
 * List all prefixes
 * List all times for prefixes
 * Clean incomplete uploads (Use wisely, can delete a backup in progress)
-* Perform rolling cleanup.  Can keep a user defined number of daily and weekly backups with user specified day.  Ally days and weeks are deltas calculated from the timestamp of the last successful backup to s3.
+* Perform rolling cleanup.  Can keep a user defined number of daily and weekly backups with user specified day.  All days and weeks are deltas calculated from the timestamp of the last successful backup to s3.
 
 # Algorithm
 
-Below is a general outline on how the plugin was designed to work.  No meta data is stored on S3.  Every time this plugin is launched it performs an analysis of S3 to ensure it is always using a correct state of backups.  2 instances should never access the same prefix concurrently, this could cause issues with data consistency.
+Below is a general outline on how the plugin was designed to work. Every time this plugin is launched it performs an analysis of S3 to ensure it is always using a correct state of backups.  2 instances should never access the same prefix concurrently, this could cause issues with data consistency.
 
 ## Snapshot path
 
@@ -32,13 +41,23 @@ Remove all incomplete backups before start time
 
 Analyze all time stamps, if more than one backup is present per day, only keep the latest complete backup for that day
 
-For all backups, if older than the max weekly, remove it , of between oldest weekly and oldest daily, only keep it if it falls on the day specified
+For all backups, if older than the max weekly, remove it, of between oldest weekly and oldest daily, only keep it if it falls on the day specified
 
 
 # Notes
 
 Occasionally a cleanup operation will miss a time stamp.  From my testing this appears to be due to the amazon eventual consistency, and the timestamp not being returned on a delim search.  On the next run it is usually deleted
 
+This should only be run on machines with the *nix split command available. May work on windows with cygwin, but is untested. 
+If a file is over 5GB, the only way to upload it to S3 is to use multipart uploads. To split the file into parts, s3snapshot relies on the split command. The file parts are uploaded sequentially, so there is no speed boost in using multipart upload.
+
+This has only been tested on Ruby 1.8.7, but appears to work on Ruby 1.9.3.
+
+
 #Future features
 
-* Use native s3 copying for unmodified files (similar to s3sync).  If a file exists in the previous completed backup and has not changed, copy it on the new path in s3, this will avoid the need to upload unmodified files.
+* Use native s3 copying for unmodified files (similar to s3sync).  If a file exists in the previous completed backup and has not changed, copy it on the new path in s3, this will avoid the need to upload unmodified files.
+
+# Change Log
+
+See the [change log|changelog.markdown] for changes.

data/changelog.markdown

@@ -0,0 +1,3 @@
+# v0.0.12
+
+backup_manager.read_complete method now checks if the s3snapshot complete marker exists for a particular prefix rather than relying on the common prefixes response from the S3 query. This change was made because sometimes the common_prefixes method was not returning a prefix that was expected, leading the backup manager to handle completed backups incorrectly.

data/lib/s3snapshot/backup_manager.rb

@@ -22,7 +22,7 @@ module S3snapshot
     
     ##
     #Get all prefixes in the bucket
-    #
+    ##
     def prefixes
       bucket.files.all(:delimiter => "/").common_prefixes
     end
@@ -104,9 +104,9 @@ module S3snapshot
       end
     end
     
-    #
+    ##
     #Delete all files from a snapshot.  Will remove the complete file first to avoid other clients using the backup
-    #
+    ##
     def remove(prefix, timestamp)
       complete_marker =  bucket.files.get(complete_path(prefix, timestamp))
       
@@ -137,7 +137,7 @@ module S3snapshot
     
     ##
     #Perform a rolling delete for the given prefix.  Keeps the "newest" daily backup for the given day, and keeps a backup for the day of the week specified by day of week.  day_of_week follows cron style syntax, 0 = sunday and 6 = saturday
-    # 
+    ##
     def roll(prefix, num_days, num_weeks, day_of_week)
       
       start =  TimeFactory.utc_time
@@ -188,19 +188,17 @@ module S3snapshot
     
     private
     
-    
-    #Returns true if the backup is complete, false otherwise. Downloads the file so can be slow
+    ##
+    #Returns true if the backup is complete, false otherwise. A backup is complete if the s3snapshot complete marker exists.
+    ##
     def read_complete?(prefix, time)
-      
-      found_prefixes = bucket.files.all(:prefix => timepath(prefix, time), :delimiter => COMPLETE_EXTENSION).common_prefixes
-      
-      
-      !found_prefixes.nil? && found_prefixes.length > 0
+      marker = complete_path(prefix, time)
+      !bucket.files.head(marker).nil?
     end
     
     ##
-    # Returnes true if the time occurs on the same day_of_week.  day_of_week follows cron style syntax, 0 = sunday and 6 = saturday
-    #
+    # Returns true if the time occurs on the same day_of_week.  day_of_week follows cron style syntax, 0 = sunday and 6 = saturday
+    ##
     def same_day(time, day_of_week)
       
       unless day_of_week > -1 && day_of_week < 7
@@ -242,7 +240,9 @@ module S3snapshot
       end
     end
     
+    ##
     #returns true if the first and second time occur in the same date (assumes utc time)
+    ##
     def samedate?(first, second)
    #   puts "first: #{first};#{first.nil? ? "nil": first.yday}, second #{second};#{second.nil? ? "nil": second.yday}"
       !first.nil? && !second.nil? && first.yday == second.yday

data/lib/s3snapshot/version.rb

@@ -1,3 +1,3 @@
 module S3snapshot
-  VERSION = "0.0.11"
+  VERSION = "0.0.12"
 end

data/s3snapshot.gemspec

@@ -7,7 +7,7 @@ Gem::Specification.new do |s|
   s.version     = S3snapshot::VERSION
   s.platform    = Gem::Platform::RUBY
   s.authors     = ["Todd Nine", "Andrew Esler"]
-  s.email       = ["todd@spidertracks.co.nz"]
+  s.email       = ["todd@spidertracks.co.nz", "andrew.esler@spidertracks.co.nz"]
   s.homepage    = "https://github.com/spidertracks/s3snapshot"
   s.summary     = %q{Uploads to s3}
   s.description = %q{see summary}

metadata

@@ -1,90 +1,82 @@
---- !ruby/object:Gem::Specification 
+--- !ruby/object:Gem::Specification
 name: s3snapshot
-version: !ruby/object:Gem::Version 
-  hash: 9
+version: !ruby/object:Gem::Version
+  version: 0.0.12
   prerelease: 
-  segments: 
-  - 0
-  - 0
-  - 11
-  version: 0.0.11
 platform: ruby
-authors: 
+authors:
 - Todd Nine
 - Andrew Esler
 autorequire: 
 bindir: bin
 cert_chain: []
-
-date: 2012-07-09 00:00:00 Z
-dependencies: 
-- !ruby/object:Gem::Dependency 
+date: 2013-10-20 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
   name: fog
-  prerelease: false
-  requirement: &id001 !ruby/object:Gem::Requirement 
+  requirement: !ruby/object:Gem::Requirement
     none: false
-    requirements: 
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        hash: 25
-        segments: 
-        - 1
-        - 3
-        - 1
+      - !ruby/object:Gem::Version
         version: 1.3.1
   type: :runtime
-  version_requirements: *id001
-- !ruby/object:Gem::Dependency 
-  name: thor
   prerelease: false
-  requirement: &id002 !ruby/object:Gem::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
     none: false
-    requirements: 
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        hash: 43
-        segments: 
-        - 0
-        - 14
-        - 6
+      - !ruby/object:Gem::Version
         version: 0.14.6
   type: :runtime
-  version_requirements: *id002
-- !ruby/object:Gem::Dependency 
-  name: dictionary
   prerelease: false
-  requirement: &id003 !ruby/object:Gem::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement
     none: false
-    requirements: 
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        hash: 23
-        segments: 
-        - 1
-        - 0
-        - 0
+      - !ruby/object:Gem::Version
+        version: 0.14.6
+- !ruby/object:Gem::Dependency
+  name: dictionary
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
         version: 1.0.0
   type: :runtime
-  version_requirements: *id003
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.0.0
 description: see summary
-email: 
+email:
 - todd@spidertracks.co.nz
-executables: 
+- andrew.esler@spidertracks.co.nz
+executables:
 - s3snapshot
 extensions: []
-
 extra_rdoc_files: []
-
-files: 
+files:
 - .gitignore
 - .loadpath
 - .project
 - Gemfile
-- Gemfile.lock.bak
-- README.rdoc
 - Rakefile
 - Readme.markdown
 - bin/s3snapshot
+- changelog.markdown
 - lib/s3snapshot.rb
 - lib/s3snapshot/backup_manager.rb
 - lib/s3snapshot/cli.rb
@@ -97,36 +89,26 @@ files:
 - s3snapshot.gemspec
 homepage: https://github.com/spidertracks/s3snapshot
 licenses: []
-
 post_install_message: 
 rdoc_options: []
-
-require_paths: 
+require_paths:
 - lib
-required_ruby_version: !ruby/object:Gem::Requirement 
+required_ruby_version: !ruby/object:Gem::Requirement
   none: false
-  requirements: 
-  - - ">="
-    - !ruby/object:Gem::Version 
-      hash: 3
-      segments: 
-      - 0
-      version: "0"
-required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
-  requirements: 
-  - - ">="
-    - !ruby/object:Gem::Version 
-      hash: 3
-      segments: 
-      - 0
-      version: "0"
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
 requirements: []
-
 rubyforge_project: s3snapshot
-rubygems_version: 1.8.10
+rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: Uploads to s3
 test_files: []
-

data/Gemfile.lock.bak

@@ -1,34 +0,0 @@
-PATH
-  remote: .
-  specs:
-    s3snapshot (0.0.1)
-      fog (~> 0.8.1)
-      thor (~> 0.14.6)
-
-GEM
-  remote: http://rubygems.org/
-  specs:
-    builder (3.0.0)
-    excon (0.6.3)
-    fog (0.8.1)
-      builder
-      excon (~> 0.6.1)
-      formatador (>= 0.1.3)
-      json
-      mime-types
-      net-ssh (>= 2.1.3)
-      nokogiri (>= 1.4.4)
-      ruby-hmac
-    formatador (0.1.3)
-    json (1.5.1)
-    mime-types (1.16)
-    net-ssh (2.1.4)
-    nokogiri (1.4.4)
-    ruby-hmac (0.4.0)
-    thor (0.14.6)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  s3snapshot!

data/README.rdoc

@@ -1,7 +0,0 @@
-A simple backup utility to create time based snapshots similar to time machine on s3.  
-All options are command line based so they can easily be scripted with cron.
-Supports multiple daily backups, as well as rolling daily and weekly.  
-For daily, it keeps the newest complete backup for the day, for weekly it keeps the last day in the week. 
-
-Note that this should only be run on machines with the *nix split command available. May work on windows with cygwin, but is untested. 
-If a file is over 5GB, the only way to upload it to S3 is to use multipart uploads. To split the file into parts, s3snapshot relies on the split command. The file parts are uploaded sequentially, so there is no speed boost in using multipart upload. s3snapshot is developed on MRI 1.8.7 so multi thread upload does not work.