s3sync 1.2.5 → 2.0.0

data/README

@@ -1,401 +0,0 @@
-Welcome to s3sync.rb         
---------------------
-Home page, wiki, forum, bug reports, etc: http://s3sync.net
-
-This is a ruby program that easily transfers directories between a local
-directory and an S3 bucket:prefix. It behaves somewhat, but not precisely, like
-the rsync program. In particular, it shares rsync's peculiar behavior that
-trailing slashes on the source side are meaningful. See examples below.
-
-One benefit over some other comparable tools is that s3sync goes out of its way
-to mirror the directory structure on S3.  Meaning you don't *need* to use s3sync
-later in order to view your files on S3.  You can just as easily use an S3
-shell, a web browser (if you used the --public-read option), etc.  Note that
-s3sync is NOT necessarily going to be able to read files you uploaded via some
-other tool.  This includes things uploaded with the old perl version!  For best
-results, start fresh!
-
-s3sync runs happily on linux, probably other *ix, and also Windows (except that
-symlinks and permissions management features don't do anything on Windows). If
-you get it running somewhere interesting let me know (see below)
-
-s3sync is free, and license terms are included in all the source files. If you
-decide to make it better, or find bugs, please let me know.
-
-The original inspiration for this tool is the perl script by the same name which
-was made by Thorsten von Eicken (and later updated by me). This ruby program
-does not share any components or logic from that utility; the only relation is
-that it performs a similar task.
-
-
-Examples: 
----------
-(using S3 bucket 'mybucket' and prefix 'pre')
-  Put the local etc directory itself into S3
-        s3sync.rb  -r  /etc  mybucket:pre
-        (This will yield S3 keys named  pre/etc/...)
-  Put the contents of the local /etc dir into S3, rename dir:
-        s3sync.rb  -r  /etc/  mybucket:pre/etcbackup
-        (This will yield S3 keys named  pre/etcbackup/...)
-  Put contents of S3 "directory" etc into local dir
-        s3sync.rb  -r  mybucket:pre/etc/  /root/etcrestore
-        (This will yield local files at  /root/etcrestore/...)
-  Put the contents of S3 "directory" etc into a local dir named etc
-        s3sync.rb  -r  mybucket:pre/etc  /root
-        (This will yield local files at  /root/etc/...)
-  Put S3 nodes under the key pre/etc/ to the local dir etcrestore
-  **and create local dirs even if S3 side lacks dir nodes**
-        s3sync.rb  -r  --make-dirs  mybucket:pre/etc/  /root/etcrestore
-        (This will yield local files at  /root/etcrestore/...)
-
-
-Prerequisites:
---------------
-You need a functioning Ruby (>=1.8.4) installation, as well as the OpenSSL ruby 
-library (which may or may not come with your ruby).
-
-How you get these items working on your system is really not any of my 
-business, but you might find the following things helpful.  If you're using 
-Windows, the ruby site has a useful "one click installer" (although it takes 
-more clicks than that, really).  On debian (and ubuntu, and other debian-like 
-things), there are apt packages available for ruby and the open ssl lib.
-
-
-Your environment:
------------------
-s3sync needs to know several interesting values to work right.  It looks for 
-them in the following environment variables -or- a s3config.yml file.
-In the yml case, the names need to be lowercase (see example file).
-Furthermore, the yml is searched for in the following locations, in order:
-   $S3CONF/s3config.yml
-   $HOME/.s3conf/s3config.yml
-   /etc/s3conf/s3config.yml
-
-Required:
-	AWS_ACCESS_KEY_ID
-	AWS_SECRET_ACCESS_KEY
-	
-	If you don't know what these are, then s3sync is probably not the
-	right tool for you to be starting out with.
-Optional:
-	AWS_S3_HOST - I don't see why the default would ever be wrong
-   HTTP_PROXY_HOST,HTTP_PROXY_PORT,HTTP_PROXY_USER,HTTP_PROXY_PASSWORD - proxy
-	SSL_CERT_DIR - Where your Cert Authority keys live; for verification
-	SSL_CERT_FILE - If you have just one PEM file for CA verification
-	S3SYNC_RETRIES - How many HTTP errors to tolerate before exiting
-	S3SYNC_WAITONERROR - How many seconds to wait after an http error
-	S3SYNC_MIME_TYPES_FILE - Where is your mime.types file
-	S3SYNC_NATIVE_CHARSET - For example Windows-1252.  Defaults to ISO-8859-1.
-   AWS_CALLING_FORMAT - Defaults to REGULAR
-       REGULAR   # http://s3.amazonaws.com/bucket/key
-       SUBDOMAIN # http://bucket.s3.amazonaws.com/key
-       VANITY    # http://<vanity_domain>/key  
-
-Important: For EU-located buckets you should set the calling format to SUBDOMAIN
-Important: For US buckets with CAPS or other weird traits set the calling format 
-to REGULAR
-
-I use "envdir" from the daemontools package to set up my env 
-variables easily: http://cr.yp.to/daemontools/envdir.html
-For example:
-	envdir /root/s3sync/env /root/s3sync/s3sync.rb -etc etc etc
-I know there are other similar tools out there as well.  
-
-You can also just call it in a shell script where you have exported the vars 
-first such as:
-#!/bin/bash
-export AWS_ACCESS_KEY_ID=valueGoesHere
-...
-s3sync.rb -etc etc etc
-
-But by far the easiest (and newest) way to set this up is to put the name:value
-pairs in a file named s3config.yml and let the yaml parser pick them up. There
-is an .example file shipped with the tar.gz to show what a yaml file looks like.
-Thanks to Alastair Brunton for this addition.
-
-You can also use some combination of .yaml and environment variables, if you
-want. Go nuts.
-
-
-Management tasks
-----------------
-For low-level S3 operations not encapsulated by the sync paradigm, try the
-companion utility s3cmd.rb. See README_s3cmd.txt.
-
-
-About single files
-------------------
-s3sync lacks the special case code that would be needed in order to handle a
-source/dest that's a single file. This isn't one of the supported use cases so
-don't expect it to work. You can use the companion utility s3cmd.rb for single
-get/puts.
-
-
-About Directories, the bane of any S3 sync-er
---------------------------------------------- 
-In S3 there's no actual concept of folders, just keys and nodes. So, every tool
-uses its own proprietary way of storing dir info (my scheme being the best
-naturally) and in general the methods are not compatible.
-
-If you populate S3 by some means *other than* s3sync and then try to use s3sync
-to "get" the S3 stuff to a local filesystem, you will want to use the
---make-dirs option. This causes the local dirs to be created even if there is no
-s3sync-compatible directory node info stored on the S3 side. In other words,
-local folders are conjured into existence whenever they are needed to make the
-"get" succeed.
-
-
-About MD5 hashes
-----------------
-s3sync's normal operation is to compare the file size and MD5 hash of each item
-to decide whether it needs syncing.  On the S3 side, these hashes are stored and
-returned to us as the "ETag" of each item when the bucket is listed, so it's
-very easy.  On the local side, the MD5 must be calculated by pushing every byte
-in the file through the MD5 algorithm.  This is CPU and IO intensive!  
-
-Thus you can specify the option --no-md5. This will compare the upload time on
-S3 to the "last modified" time on the local item, and not do md5 calculations
-locally at all. This might cause more transfers than are absolutely necessary.
-For example if the file is "touched" to a newer modified date, but its contents
-didn't change. Conversely if a file's contents are modified but the date is not
-updated, then the sync will pass over it.  Lastly, if your clock is very
-different from the one on the S3 servers, then you may see unanticipated
-behavior.
-
-
-A word on SSL_CERT_DIR:
------------------------
-On my debian install I didn't find any root authority public keys.  I installed
-some by running this shell archive: 
-http://mirbsd.mirsolutions.de/cvs.cgi/src/etc/ssl.certs.shar
-(You have to click download, and then run it wherever you want the certs to be
-placed).  I do not in any way assert that these certificates are good,
-comprehensive, moral, noble, or otherwise correct.  But I am using them.
-
-If you don't set up a cert dir, and try to use ssl, then you'll 1) get an ugly
-warning message slapped down by ruby, and 2) not have any protection AT ALL from
-malicious servers posing as s3.amazonaws.com.  Seriously... you want to get
-this right if you're going to have any sensitive data being tossed around.
---
-There is a debian package ca-certificates; this is what I'm using now.
-apt-get install ca-certificates
-and then use:
-SSL_CERT_DIR=/etc/ssl/certs
-
-You used to be able to use just one certificate, but recently AWS has started
-using more than one CA.
-
-
-Getting started:
-----------------
-Invoke by typing s3sync.rb and you should get a nice usage screen.
-Options can be specified in short or long form (except --delete, which has no 
-short form)
-
-ALWAYS TEST NEW COMMANDS using --dryrun(-n) if you want to see what will be
-affected before actually doing it. ESPECIALLY if you use --delete. Otherwise, do
-not be surprised if you misplace a '/' or two and end up deleting all your
-precious, precious files.
-
-If you use the --public-read(-p) option, items sent to S3 will be ACL'd so that
-anonymous web users can download them, given the correct URL. This could be
-useful if you intend to publish directories of information for others to see.
-For example, I use s3sync to publish itself to its home on S3 via the following
-command: s3sync.rb -v -p publish/ ServEdge_pub:s3sync Where the files live in a
-local folder called "publish" and I wish them to be copied to the URL:
-http://s3.amazonaws.com/ServEdge_pub/s3sync/... If you use --ssl(-s) then your
-connections with S3 will be encrypted. Otherwise your data will be sent in clear
-form, i.e. easy to intercept by malicious parties.
-
-If you want to prune items from the destination side which are not found on the
-source side, you can use --delete. Always test this with -n first to make sure
-the command line you specify is not going to do something terrible to your
-cherished and irreplaceable data.
-
-
-Updates and other discussion: 
------------------------------ 
-The latest version of s3sync should normally be at:
-	http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz 
-and the Amazon S3 forums probably have a few threads going on it at any given
-time. I may not always see things posted to the threads, so if you want you can
-contact me at gbs-s3@10forward.com too.
-
-
-Change Log:
------------
-
-2006-09-29:
-Added support for --expires and --cache-control. Eg:
---expires="Thu, 01 Dec 2007 16:00:00 GMT"
---cache-control="no-cache"
-
-Thanks to Charles for pointing out the need for this, and supplying a patch
-proving that it would be trivial to add =) Apologies for not including the short
-form (-e) for the expires. I have a rule that options taking arguments should
-use the long form.
-----------
-
-2006-10-04
-Several minor debugs and edge cases.
-Fixed a bug where retries didn't rewind the stream to start over.
-----------
-
-2006-10-12
-Version 1.0.5
-Finally figured out and fixed bug of trying to follow local symlink-to-directory.
-Fixed a really nasty sorting discrepancy that caused problems when files started
-with the same name as a directory.
-Retry on connection-reset on the S3 side.
-Skip files that we can't read instead of dying.
-----------
-
-2006-10-12
-Version 1.0.6
-Some GC voodoo to try and keep a handle on the memory footprint a little better.
-There is still room for improvement here.
-----------
-
-2006-10-13
-Version 1.0.7
-Fixed symlink dirs being stored to S3 as real dirs (and failing with 400)
-Added a retry catch for connection timeout error.
-(Hopefully) caught a bug that expected every S3 listing to contain results
-----------
-
-2006-10-14
-Version 1.0.8
-Was testing for file? before symlink? in localnode.stream. This meant that for
-symlink files it was trying to shove the real file contents into the symlink
-body on s3.
-----------
-
-2006-10-14
-Version 1.0.9
-Woops, I was using "max-entries" for some reason but the proper header is
-"max-keys".  Not a big deal.
-Broke out the S3try stuff into a separate file so I could re-use it for s3cmd.rb
-----------
-
-2006-10-16
-Added a couple debug lines; not even enough to call it a version revision.
-----------
-
-2006-10-25
-Version 1.0.10
-UTF-8 fixes.
-Catching a couple more retry-able errors in s3try (instead of aborting the
-program).
-----------
-
-2006-10-26
-Version 1.0.11
-Revamped some details of the generators and comparator so that directories are
-handled in a more exact and uniform fashion across local and S3. 
-----------
-
-2006-11-28
-Version 1.0.12
-Added a couple more error catches to s3try.
-----------
-
-2007-01-08
-Version 1.0.13
-Numerous small changes to slash and path handling, in order to catch several 
-cases where "root" directory nodes were not being created on S3.
-This makes restores work a lot more intuitively in many cases.
-----------
-
-2007-01-25
-Version 1.0.14
-Peter Fales' marker fix.
-Also, markers should be decoded into native charset (because that's what s3
-expects to see).
-----------
-
-2007-02-19
-Version 1.1.0
-*WARNING* Lots of path-handling changes. *PLEASE* test safely before you just
-swap this in for your working 1.0.x version.
-
-- Adding --exclude (and there was much rejoicing).
-- Found Yet Another Leading Slash Bug with respect to local nodes. It was always
-"recursing" into the first folder even if there was no trailing slash and -r
-wasn't specified. What it should have done in this case is simply create a node
-for the directory itself, then stop (not check the dir's contents).
-- Local node canonicalization was (potentially) stripping the trailing slash,
-which we need in order to make some decisios in the local generator.
-- Fixed problem where it would prepend a "/" to s3 key names even with blank
-prefix.
-- Fixed S3->local when there's no "/" in the source so it doesn't try to create
-a folder with the bucket name. 
-- Updated s3try and s3_s3sync_mod to allow SSL_CERT_FILE
-----------
-
-2007-02-22
-Version 1.1.1
-Fixed dumb regression bug caused by the S3->local bucket name fix in 1.1.0
-----------
-
-2007-02-25
-Version 1.1.2
-Added --progress
-----------
-
-2007-06-02
-Version 1.1.3
-IMPORTANT!
-Pursuant to http://s3sync.net/forum/index.php?topic=49.0 , the tar.gz now
-expands into its own sub-directory named "s3sync" instead of dumping all the
-files into the current directory.
-
-In the case of commands of the form:
-	s3sync -r somedir somebucket:
-The root directory node in s3 was being stored as "somedir/" instead of "somedir"
-which caused restores to mess up when you say:
-	s3sync -r somebucket: restoredir
-The fix to this, by coincidence, actually makes s3fox work even *less* well with 
-s3sync.  I really need to build my own xul+javascript s3 GUI some day.
-
-Also fixed some of the NoMethodError stuff for when --progress is used
-and caught Errno::ETIMEDOUT
-----------
-
-2007-07-12
-Version 1.1.4
-Added Alastair Brunton's yaml config code.
-----------
-
-2007-11-17
-Version 1.2.1
-Compatibility for S3 API revisions.
-When retries are exhausted, emit an error.
-Don't ever try to delete the 'root' local dir.    
-----------
-
-2007-11-20
-Version 1.2.2
-Handle EU bucket 307 redirects (in s3try.rb)
---make-dirs added
-----------
-
-2007-11-20
-Version 1.2.3
-Fix SSL verification settings that broke in new S3 API.
-----------
-
-2008-01-06
-Version 1.2.4
-Run from any dir (search "here" for includes).
-Search out s3config.yml in some likely places.
-Reset connection (properly) on retry-able non-50x errors.
-Fix calling format bug preventing it from working from yml.
-Added http proxy support.
-----------
-
-2008-05-11
-Version 1.2.5
-Added option --no-md5
-----------
-
-FNORD

data/README_s3cmd

@@ -1,172 +0,0 @@
-Welcome to s3cmd.rb 
-------------------- 
-This is a ruby program that wraps S3 operations into a simple command-line tool.
-It is inspired by things like rsh3ll, #sh3ll, etc., but shares no code from
-them. It's meant as a companion utility to s3sync.rb but could be used on its
-own (provided you have read the other readme file and know how to use s3sync in
-theory).
-
-I made this even though lots of other "shell"s exist, because I wanted a
-single-operation utility, instead of a shell "environment".  This lends itself
-more to scripting, etc.  Also the delete operation on rsh3ll seems to be borken
-at the moment? =(
-
-Users not yet familiar with s3sync should read about that first, since s3cmd and
-s3sync share a tremendous amount of conventions and syntax. Particularly you
-have to set up environment variables prior to calling s3cmd, and s3cmd also uses
-the "bucket:key" syntax popularized by s3sync. Many of the options are the same
-too. Really, go read the other readme first if you haven't used s3sync yet.
-Otherwise you will become confused.  It's OK, I'll wait.
-
-....
-
-In general, s3sync and s3cmd complement each other. s3sync is useful to perform
-serious synchronization operations, and s3cmd allows you to do simple things
-such as bucket management, listing, transferring single files, and the like.
-
-Here is the usage, with examples to follow.
-
-s3cmd.rb [options] <command> [arg(s)]           version 1.0.0
-  --help    -h        --verbose     -v     --dryrun    -n
-  --ssl     -s        --debug       -d
-
-Commands:
-s3cmd.rb  listbuckets  [headers]
-s3cmd.rb  createbucket|deletebucket  <bucket>  [headers]
-s3cmd.rb  list  <bucket>[:prefix]  [max/page]  [delimiter]  [headers]
-s3cmd.rb  delete  <bucket>:key  [headers]
-s3cmd.rb  deleteall  <bucket>[:prefix]  [headers]
-s3cmd.rb  get|put  <bucket>:key  <file>  [headers]
-
-
-A note about [headers]
-----------------------
-For some S3 operations, such as "put", you might want to specify certain headers
-to the request such as Cache-Control, Expires, x-amz-acl, etc.  Rather than
-supporting a load of separate command-line options for these, I just allow
-header specification.  So to upload a file with public-read access you could
-say:
-	s3cmd.rb  put  MyBucket:TheFile.txt  x-amz-acl:public-read
-
-If you don't need to add any particular headers then you can just ignore this
-whole [headers] thing and pretend it's not there.  This is somewhat of an
-advanced option.
-
-
-Examples
---------
-List all the buckets your account owns:
-	s3cmd.rb listbuckets
-
-Create a new bucket:
-	s3cmd.rb createbucket BucketName
-
-Create a new bucket in the EU:
-	s3cmd.rb createbucket BucketName EU
-   
-Find out the location constraint of a bucket:
-   s3cmd.rb location BucketName
-
-Delete an old bucket you don't want any more:
-	s3cmd.rb deletebucket BucketName
-	
-Find out what's in a bucket, 10 lines at a time:
-	s3cmd.rb list BucketName 10
-	
-Only look in a particular prefix:
-	s3cmd.rb list BucketName:startsWithThis
-	
-Look in the virtual "directory" named foo;
-lists sub-"directories" and keys that are at this level.
-Note that if you specify a delimiter you must specify a max before it.
-(until I make the options parsing smarter)
-	s3cmd.rb list BucketName:foo/  10  /
-
-Delete a key:
-	s3cmd.rb delete BucketName:AKey
-
-Delete all keys that match (like a combo between list and delete):
-	s3cmd.rb deleteall BucketName:SomePrefix
-	
-Only pretend you're going to delete all keys that match, but list them: 
-	s3cmd.rb  --dryrun  deleteall  BucketName:SomePrefix
-	
-Delete all keys in a bucket (leaving the bucket):
-	s3cmd.rb deleteall BucketName
-	
-Get a file from S3 and store it to a local file
-	s3cmd.rb get BucketName:TheFileOnS3.txt  ALocalFile.txt
-	
-Put a local file up to S3 
-Note we don't automatically set mime type, etc.
-NOTE that the order of the options doesn't change. S3 stays first!
-	s3cmd.rb put BucketName:TheFileOnS3.txt ALocalFile.txt
-
-	
-Change Log:
------------
-2006-10-14:
-Created.
------------
-
-2006-10-16
-Version 1.0.1
-Force content length to a string value since some ruby's don't convert it right.
------------
-
-2006-10-25
-UTF-8 fixes.
------------
-
-2006-11-28
-Version 1.0.3
-Added a couple more error catches to s3try.
-----------
-
-2007-01-25
-Version 1.0.4
-Peter Fales' marker fix.
-Also, markers should be decoded into native charset (because that's what s3
-expects to see).
-----------
-
-2007-02-19
-- Updated s3try and s3_s3sync_mod to allow SSL_CERT_FILE
-----------
-
-2007-2-25
-Added --progress
-----------
-
-2007-07-12
-Version 1.0.6
-Added Alastair Brunton's yaml config code.
-----------
-
-2007-11-17
-Version 1.2.1
-Compatibility for S3 API revisions.
-When retries are exhausted, emit an error.
-----------
-
-2007-11-20
-Version 1.2.2
-Handle EU bucket 307 redirects (in s3try.rb)
-----------
-
-2007-11-20
-Version 1.2.3
-Fix SSL verification settings that broke in new S3 API.
-----------
-
-2008-01-06
-Version 1.2.4
-Run from any dir (search "here" for includes).
-Search out s3config.yml in some likely places.
-Reset connection (properly) on retry-able non-50x errors.
-Fix calling format bug preventing it from working from yml.
-Added http proxy support.
-----------
-
-
-FNORD