cprobert-s3sync 1.3.6

data/docs/README.txt

@@ -0,0 +1,184 @@
+
+
+Change Log:
+-----------
+
+2006-09-29:
+Added support for --expires and --cache-control. Eg:
+--expires="Thu, 01 Dec 2007 16:00:00 GMT"
+--cache-control="no-cache"
+
+Thanks to Charles for pointing out the need for this, and supplying a patch
+proving that it would be trivial to add =) Apologies for not including the short
+form (-e) for the expires. I have a rule that options taking arguments should
+use the long form.
+----------
+
+2006-10-04
+Several minor debugs and edge cases.
+Fixed a bug where retries didn't rewind the stream to start over.
+----------
+
+2006-10-12
+Version 1.0.5
+Finally figured out and fixed bug of trying to follow local symlink-to-directory.
+Fixed a really nasty sorting discrepancy that caused problems when files started
+with the same name as a directory.
+Retry on connection-reset on the S3 side.
+Skip files that we can't read instead of dying.
+----------
+
+2006-10-12
+Version 1.0.6
+Some GC voodoo to try and keep a handle on the memory footprint a little better.
+There is still room for improvement here.
+----------
+
+2006-10-13
+Version 1.0.7
+Fixed symlink dirs being stored to S3 as real dirs (and failing with 400)
+Added a retry catch for connection timeout error.
+(Hopefully) caught a bug that expected every S3 listing to contain results
+----------
+
+2006-10-14
+Version 1.0.8
+Was testing for file? before symlink? in localnode.stream. This meant that for
+symlink files it was trying to shove the real file contents into the symlink
+body on s3.
+----------
+
+2006-10-14
+Version 1.0.9
+Woops, I was using "max-entries" for some reason but the proper header is
+"max-keys".  Not a big deal.
+Broke out the S3try stuff into a separate file so I could re-use it for s3cmd.rb
+----------
+
+2006-10-16
+Added a couple debug lines; not even enough to call it a version revision.
+----------
+
+2006-10-25
+Version 1.0.10
+UTF-8 fixes.
+Catching a couple more retry-able errors in s3try (instead of aborting the
+program).
+----------
+
+2006-10-26
+Version 1.0.11
+Revamped some details of the generators and comparator so that directories are
+handled in a more exact and uniform fashion across local and S3. 
+----------
+
+2006-11-28
+Version 1.0.12
+Added a couple more error catches to s3try.
+----------
+
+2007-01-08
+Version 1.0.13
+Numerous small changes to slash and path handling, in order to catch several 
+cases where "root" directory nodes were not being created on S3.
+This makes restores work a lot more intuitively in many cases.
+----------
+
+2007-01-25
+Version 1.0.14
+Peter Fales' marker fix.
+Also, markers should be decoded into native charset (because that's what s3
+expects to see).
+----------
+
+2007-02-19
+Version 1.1.0
+*WARNING* Lots of path-handling changes. *PLEASE* test safely before you just
+swap this in for your working 1.0.x version.
+
+- Adding --exclude (and there was much rejoicing).
+- Found Yet Another Leading Slash Bug with respect to local nodes. It was always
+"recursing" into the first folder even if there was no trailing slash and -r
+wasn't specified. What it should have done in this case is simply create a node
+for the directory itself, then stop (not check the dir's contents).
+- Local node canonicalization was (potentially) stripping the trailing slash,
+which we need in order to make some decisios in the local generator.
+- Fixed problem where it would prepend a "/" to s3 key names even with blank
+prefix.
+- Fixed S3->local when there's no "/" in the source so it doesn't try to create
+a folder with the bucket name. 
+- Updated s3try and s3_s3sync_mod to allow SSL_CERT_FILE
+----------
+
+2007-02-22
+Version 1.1.1
+Fixed dumb regression bug caused by the S3->local bucket name fix in 1.1.0
+----------
+
+2007-02-25
+Version 1.1.2
+Added --progress
+----------
+
+2007-06-02
+Version 1.1.3
+IMPORTANT!
+Pursuant to http://s3sync.net/forum/index.php?topic=49.0 , the tar.gz now
+expands into its own sub-directory named "s3sync" instead of dumping all the
+files into the current directory.
+
+In the case of commands of the form:
+	s3sync -r somedir somebucket:
+The root directory node in s3 was being stored as "somedir/" instead of "somedir"
+which caused restores to mess up when you say:
+	s3sync -r somebucket: restoredir
+The fix to this, by coincidence, actually makes s3fox work even *less* well with 
+s3sync.  I really need to build my own xul+javascript s3 GUI some day.
+
+Also fixed some of the NoMethodError stuff for when --progress is used
+and caught Errno::ETIMEDOUT
+----------
+
+2007-07-12
+Version 1.1.4
+Added Alastair Brunton's yaml config code.
+----------
+
+2007-11-17
+Version 1.2.1
+Compatibility for S3 API revisions.
+When retries are exhausted, emit an error.
+Don't ever try to delete the 'root' local dir.    
+----------
+
+2007-11-20
+Version 1.2.2
+Handle EU bucket 307 redirects (in s3try.rb)
+--make-dirs added
+----------
+
+2007-11-20
+Version 1.2.3
+Fix SSL verification settings that broke in new S3 API.
+----------
+
+2008-01-06
+Version 1.2.4
+Run from any dir (search "here" for includes).
+Search out s3config.yml in some likely places.
+Reset connection (properly) on retry-able non-50x errors.
+Fix calling format bug preventing it from working from yml.
+Added http proxy support.
+----------
+
+2008-05-11
+Version 1.2.5
+Added option --no-md5
+----------
+
+2008-06-16
+Version 1.2.6
+Catch connect errors and retry.
+----------
+
+FNORD

data/docs/README_s3cmd.txt

@@ -0,0 +1,172 @@
+Welcome to s3cmd.rb 
+------------------- 
+This is a ruby program that wraps S3 operations into a simple command-line tool.
+It is inspired by things like rsh3ll, #sh3ll, etc., but shares no code from
+them. It's meant as a companion utility to s3sync.rb but could be used on its
+own (provided you have read the other readme file and know how to use s3sync in
+theory).
+
+I made this even though lots of other "shell"s exist, because I wanted a
+single-operation utility, instead of a shell "environment".  This lends itself
+more to scripting, etc.  Also the delete operation on rsh3ll seems to be borken
+at the moment? =(
+
+Users not yet familiar with s3sync should read about that first, since s3cmd and
+s3sync share a tremendous amount of conventions and syntax. Particularly you
+have to set up environment variables prior to calling s3cmd, and s3cmd also uses
+the "bucket:key" syntax popularized by s3sync. Many of the options are the same
+too. Really, go read the other readme first if you haven't used s3sync yet.
+Otherwise you will become confused.  It's OK, I'll wait.
+
+....
+
+In general, s3sync and s3cmd complement each other. s3sync is useful to perform
+serious synchronization operations, and s3cmd allows you to do simple things
+such as bucket management, listing, transferring single files, and the like.
+
+Here is the usage, with examples to follow.
+
+s3cmd.rb [options] <command> [arg(s)]           version 1.0.0
+  --help    -h        --verbose     -v     --dryrun    -n
+  --ssl     -s        --debug       -d
+
+Commands:
+s3cmd.rb  listbuckets  [headers]
+s3cmd.rb  createbucket|deletebucket  <bucket>  [headers]
+s3cmd.rb  list  <bucket>[:prefix]  [max/page]  [delimiter]  [headers]
+s3cmd.rb  delete  <bucket>:key  [headers]
+s3cmd.rb  deleteall  <bucket>[:prefix]  [headers]
+s3cmd.rb  get|put  <bucket>:key  <file>  [headers]
+
+
+A note about [headers]
+----------------------
+For some S3 operations, such as "put", you might want to specify certain headers
+to the request such as Cache-Control, Expires, x-amz-acl, etc.  Rather than
+supporting a load of separate command-line options for these, I just allow
+header specification.  So to upload a file with public-read access you could
+say:
+	s3cmd.rb  put  MyBucket:TheFile.txt  x-amz-acl:public-read
+
+If you don't need to add any particular headers then you can just ignore this
+whole [headers] thing and pretend it's not there.  This is somewhat of an
+advanced option.
+
+
+Examples
+--------
+List all the buckets your account owns:
+	s3cmd.rb listbuckets
+
+Create a new bucket:
+	s3cmd.rb createbucket BucketName
+
+Create a new bucket in the EU:
+	s3cmd.rb createbucket BucketName EU
+   
+Find out the location constraint of a bucket:
+   s3cmd.rb location BucketName
+
+Delete an old bucket you don't want any more:
+	s3cmd.rb deletebucket BucketName
+	
+Find out what's in a bucket, 10 lines at a time:
+	s3cmd.rb list BucketName 10
+	
+Only look in a particular prefix:
+	s3cmd.rb list BucketName:startsWithThis
+	
+Look in the virtual "directory" named foo;
+lists sub-"directories" and keys that are at this level.
+Note that if you specify a delimiter you must specify a max before it.
+(until I make the options parsing smarter)
+	s3cmd.rb list BucketName:foo/  10  /
+
+Delete a key:
+	s3cmd.rb delete BucketName:AKey
+
+Delete all keys that match (like a combo between list and delete):
+	s3cmd.rb deleteall BucketName:SomePrefix
+	
+Only pretend you're going to delete all keys that match, but list them: 
+	s3cmd.rb  --dryrun  deleteall  BucketName:SomePrefix
+	
+Delete all keys in a bucket (leaving the bucket):
+	s3cmd.rb deleteall BucketName
+	
+Get a file from S3 and store it to a local file
+	s3cmd.rb get BucketName:TheFileOnS3.txt  ALocalFile.txt
+	
+Put a local file up to S3 
+Note we don't automatically set mime type, etc.
+NOTE that the order of the options doesn't change. S3 stays first!
+	s3cmd.rb put BucketName:TheFileOnS3.txt ALocalFile.txt
+
+	
+Change Log:
+-----------
+2006-10-14:
+Created.
+-----------
+
+2006-10-16
+Version 1.0.1
+Force content length to a string value since some ruby's don't convert it right.
+-----------
+
+2006-10-25
+UTF-8 fixes.
+-----------
+
+2006-11-28
+Version 1.0.3
+Added a couple more error catches to s3try.
+----------
+
+2007-01-25
+Version 1.0.4
+Peter Fales' marker fix.
+Also, markers should be decoded into native charset (because that's what s3
+expects to see).
+----------
+
+2007-02-19
+- Updated s3try and s3_s3sync_mod to allow SSL_CERT_FILE
+----------
+
+2007-2-25
+Added --progress
+----------
+
+2007-07-12
+Version 1.0.6
+Added Alastair Brunton's yaml config code.
+----------
+
+2007-11-17
+Version 1.2.1
+Compatibility for S3 API revisions.
+When retries are exhausted, emit an error.
+----------
+
+2007-11-20
+Version 1.2.2
+Handle EU bucket 307 redirects (in s3try.rb)
+----------
+
+2007-11-20
+Version 1.2.3
+Fix SSL verification settings that broke in new S3 API.
+----------
+
+2008-01-06
+Version 1.2.4
+Run from any dir (search "here" for includes).
+Search out s3config.yml in some likely places.
+Reset connection (properly) on retry-able non-50x errors.
+Fix calling format bug preventing it from working from yml.
+Added http proxy support.
+----------
+
+
+FNORD

data/docs/s3config.yml.example

@@ -0,0 +1,3 @@
+aws_access_key_id: 11111111111111111111111
+aws_secret_access_key: 222222222222222222222
+ssl_cert_dir: /home/user/s3sync/certs

data/lib/s3sync.rb

@@ -0,0 +1,39 @@
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+
+module S3sync
+  VERSION = '1.3.0'
+
+  require 's3sync/s3try'
+  require 's3sync/s3config'
+
+  # after other mods, so we don't overwrite yaml vals with defaults
+  include S3Config
+
+
+  def S3sync.s3cmdList(bucket, path, max=nil, delim=nil, marker=nil, headers={})
+    debug(max)
+    options = Hash.new
+    options['prefix'] = path # start at the right depth
+    options['max-keys'] = max ? max.to_s : 100
+    options['delimiter'] = delim if delim
+    options['marker'] = marker if marker
+    S3try(:list_bucket, bucket, options, headers)
+  end
+
+  # turn an array into a hash of pairs
+  def S3sync.hashPairs(ar)
+    ret = Hash.new
+    ar.each do |item|
+      name = (/^(.*?):/.match(item))[1]
+      item = (/^.*?:(.*)$/.match(item))[1]
+      ret[name] = item
+    end if ar
+    ret
+  end
+end
+
+
+def debug(str)
+  $stderr.puts str if $S3syncOptions['--debug']
+end

data/lib/s3sync/HTTPStreaming.rb

@@ -0,0 +1,108 @@
+# This software code is made available "AS IS" without warranties of any        
+# kind.  You may copy, display, modify and redistribute the software            
+# code either by itself or as incorporated into your code; provided that        
+# you do not remove any proprietary notices.  Your use of this software         
+# code is at your own risk and you waive any claim against the author
+# with respect to your use of this software code. 
+# (c) 2007 s3sync.net
+#
+
+# The purpose of this file is to overlay the net/http library 
+# to add some functionality
+# (without changing the file itself or requiring a specific version)
+# It still isn't perfectly robust, i.e. if radical changes are made
+# to the underlying lib this stuff will need updating.
+
+require 'net/http'
+require 'delegate'
+
+module Net
+
+	$HTTPStreamingDebug = false
+
+	# Allow request body to be an IO stream
+	# Allow an IO stream argument to stream the response body out
+	class HTTP
+		alias _HTTPStreaming_request request
+		
+		def request(req, body = nil, streamResponseBodyTo = nil, &block)
+			if not block_given? and streamResponseBodyTo and streamResponseBodyTo.respond_to?(:write)
+				$stderr.puts "Response using streaming" if $HTTPStreamingDebug
+				# this might be a retry, we should make sure the stream is at its beginning
+				streamResponseBodyTo.rewind if streamResponseBodyTo.respond_to?(:rewind) and streamResponseBodyTo != $stdout 
+				block = proc do |res|
+					res.read_body do |chunk|
+						streamResponseBodyTo.write(chunk)
+					end
+				end
+			end
+			if body != nil && body.respond_to?(:read)
+				$stderr.puts "Request using streaming" if $HTTPStreamingDebug
+				# this might be a retry, we should make sure the stream is at its beginning
+				body.rewind if body.respond_to?(:rewind) 
+				req.body_stream = body
+				return _HTTPStreaming_request(req, nil, &block)
+			else
+				return _HTTPStreaming_request(req, body, &block)
+			end
+		end
+	end
+
+end #module
+
+module S3sync
+	class ProgressStream < SimpleDelegator
+		def initialize(s, size=0)
+			@start = @last = Time.new
+			@total = size
+			@transferred = 0
+			@closed = false
+			@printed = false
+			@innerStream = s
+			super(@innerStream)
+			__setobj__(@innerStream)
+		end
+		# need to catch reads and writes so we can count what's being transferred
+		def read(i)
+			res = @innerStream.read(i)
+			@transferred += res.respond_to?(:length) ? res.length : 0
+			now = Time.new
+			if(now - @last > 1) # don't do this oftener than once per second
+				@printed = true
+            begin
+               $stdout.printf("\rProgress: %db  %db/s  %s       ", @transferred, (@transferred/(now - @start)).floor, 
+                  @total > 0? (100 * @transferred/@total).floor.to_s + "%" : ""  
+               )
+            rescue FloatDomainError
+               #wtf?
+            end
+				$stdout.flush
+				@last = now
+			end
+			res
+		end
+		def write(s)
+			@transferred += s.length
+			res = @innerStream.write(s)
+			now = Time.new
+			if(now -@last > 1) # don't do this oftener than once per second
+				@printed = true
+				$stdout.printf("\rProgress: %db  %db/s  %s       ", @transferred, (@transferred/(now - @start)).floor, 
+					@total > 0? (100 * @transferred/@total).floor.to_s + "%" : ""  
+				)  
+				$stdout.flush
+				@last = now
+			end
+			res
+		end
+		def rewind()
+			@transferred = 0
+			@innerStream.rewind if @innerStream.respond_to?(:rewind)
+		end
+		def close()
+			$stdout.printf("\n") if @printed and not @closed
+			@closed = true
+			@innerStream.close
+		end
+	end
+end #module