awshucks 0.0.1

data/CHANGES

@@ -0,0 +1,4 @@
+= awshucks Changelog
+=== Version 0.0.1
+
+* Initial public release

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2007 Nathan Witmer
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README

@@ -0,0 +1,32 @@
+== awshucks
+
+* Homepage[http://awshucks.rubyforge.org/rabal/]
+* {Rubyforge Project}[http://rubyforge.org/projects/awshucks]
+* email nwitmer-rubyforge at otherward dot net
+
+== DESCRIPTION
+
+awshucks is a utility for backing up files to Amazon's Simple Storage Service (S3)
+
+== INSTALL
+
+    gem install awshucks
+    
+== USAGE
+
+    awshucks help
+
+== FEATURES
+
+* back up specific directories
+* caches file metadata for better performance
+* stores all configured backups in a single bucket
+* only uploads when files are new or changed
+
+== CREDITS
+
+Thanks to Jeremy Hinegardner and Tim Pease for the help getting started with my first gem!
+
+== LICENSE
+
+Released under the MIT license. See LICENSE.

data/Rakefile

@@ -0,0 +1,14 @@
+# make sure our project's ./lib directory is added to the ruby search path
+$: << File.join(File.dirname(__FILE__),"lib")
+
+require 'rubygems'
+require 'rake/gempackagetask'
+require 'rake/clean'
+require 'rake/rdoctask'
+require 'rake/contrib/sshpublisher'
+
+require 'awshucks'
+
+load 'tasks/setup.rb'
+
+

data/TODO

@@ -0,0 +1,143 @@
+Next:
+
+* website and documentation
+
+* look into marcel's usage of changes/changelog, and other automation
+
+Awshucks
+
+Awshucks is a utility for backing up your files to Amazon's http://aws.amazon.com/s3 Simple Storage Service (S3). 
+
+Getting started:
+
+* Install awshucks via rubygems:
+
+  sudo gem install awshucks
+  
+* Generate a new config file
+
+  awshucks generate
+  
+This will generate a default config file called awshucks.yml into the current working directory.
+
+* Edit the config file, specifically:
+
+  access_key_id: put your access key here
+
+  secret_access_key: put your secret access key here
+
+  bucket: bucket name where awshucks will store your backed-up files
+  
+* Add the locations you want to back up.
+
+  photos:
+    location: "~/pictures"
+
+  email:
+    location: "~/mail"
+  
+* Run a backup
+
+  awshucks backup
+  
+  This will back up all of the files from the locations you specified.
+    
+Limitations:
+
+* Awshucks will not ignore dotfiles or other files. It will not, however, follow symlinks.
+* If you remove a file on the local filesystem, it will currently not be deleted from your bucket on S3.
+* If you move a file on the local filesystem, awshucks will treat it as a new file and re-upload it.
+
+
+
+
+Links in the sidebar:
+
+download
+project page
+amazon s3
+
+
+
+
+
+
+
+
+Todo:
+
+* critical features before release:
+  * svn repo
+  
+* nice to have
+  * deletion of locally-removed files -- will need to flag scanned files somehow
+  * ignores
+  * better output/logging (cattr_accessor logger)
+  * archival of removed files -- same deal, but with move&rename with additional metadata
+  * list of backed up files
+  * list of archived files
+  * --dry-run -n flag
+
+* ignores
+  * list ignores with backups
+  * allow list of ignore patterns in backup config
+  * set defaults in generated config file for ignore option, and automatically merge them with backup configs
+
+* project
+  * update README, gemspec with name, version, etc. etc. including docs about how metadata is stored, how files are stored, etc. etc. (after a basic section about how things work at a high level)
+  * update README to include rspec as a dependency for running the test suite
+  * document: does not follow symlinks!
+  * sync svn repo to rubyforge: http://plans.inplanb.com/articles/2007/07/01/mirror-projects-to-a-read-only-svn-archive-with-svk
+  * write spec helper for input/output capturing?
+  * clean up the specs -- DRY up the mocks with helpers, etc.
+
+* better error handling for file not found errors for file scanner (move exceptions to another file?)
+* allow rename instead for files with matching md5s? smart algorithm for fixing data store stuff?
+* refactor the redefinition of execute to make sure everything gets the same arguments? pppossibly, also with Command 'help' do ... ?
+* add integration tests for each of the commands to make sure they get executed with the correct arguments (arity)
+* DRY up the commands that take a backup config as an option
+* DRY up the specs re: output handling -- can rspec descriptions have multiple before(:each) blocks?
+* idea: constructor(s) take a block, and instance_eval it
+
+Someday:
+
+* AES encryption -- see keybox code, using Schneier's algorithm as a reference
+* pgpme plugin / extra for encrypted backups (golgo)
+
+Done:
+
+* create rubyforge project
+* figure out where to include various gems... probably best in the main file (yep, for now)
+* rescue file-not-found error on -c flag
+* find out how mtime (in about) gets set when uploading large files --> last-modified != file's mtime (IO stream)
+* filesystem crawling
+* reorganize command and specs
+* remove alib dependency
+* add backups command to list available backups and their configuration
+* switch up IO handling in test cases and see if it's possible to replace dependence on CLI instance
+* no more passing cli around at all!
+* remove cli from args passed in to a command (piggyback help message onto config...)
+* compute/store MD5sum of file on upload
+* compare MD5 before uploading
+* find out if ensure blocks will still get executed on SystemExit (they will)
+* store metadata automatically as a separate file for better performance
+* write code to actually use metadata instead of the file objects (most of the time)
+* metadata: include bits of logic for handling touched-but-not-actually-changed files (update metadata cache)
+* metadata: write "reset_metadata_cache" command, which takes an optional backup name as well -- deletes the metadata cache file
+* write restore command
+  * default target, or specify a path
+  * compute/compare MD5sum of file on restore
+* write list command
+* generator for default config
+
+Not Gonna Do:
+
+* write sync_metadata command to use stored files' metadata as authoritative -- never mind, this happens implicitly.
+
+Questions:
+
+* can bucket names have spaces in them? --> no.
+* how to handle deleted files? set metadata status? rename them with special naming and store metadata? (archive?) --> archive FTW!
+* store metadata in special file about last-backed-up date and top-level directories? --> perhaps! it'd save bandwidth!
+* warn if the same path is specified in two configs? pppossibly! --> ehhh not now, maybe later
+* complain if a backup has a space in it? --> nah, it's allowable, it's just a YAML key. specify a string if you want.

data/bin/awshucks

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+
+begin
+    require 'awshucks'
+rescue LoadError => le
+    $: << File.expand_path(File.join(File.dirname(__FILE__),"..","lib"))
+    require 'awshucks'
+end
+
+Awshucks::CLI.execute

data/lib/awshucks.rb

@@ -0,0 +1,29 @@
+module Awshucks
+    
+    ROOT_DIR        = File.expand_path(File.join(File.dirname(__FILE__),".."))
+    LIB_DIR         = File.join(ROOT_DIR,"lib").freeze
+    RESOURCE_DIR    = File.join(ROOT_DIR,"resources").freeze
+
+    # 
+    # Utility method to require all files ending in .rb in the directory
+    # with the same name as this file minus .rb
+    #
+    def require_all_libs_relative_to(fname)
+        prepend   = File.basename(fname,".rb")
+        search_me = File.join(File.dirname(fname),prepend)
+     
+        Dir.entries(search_me).each do |rb|
+            if File.extname(rb) == ".rb" then
+                require "#{prepend}/#{File.basename(rb,".rb")}"
+            end
+        end
+    end 
+    module_function :require_all_libs_relative_to
+
+end
+
+require 'rubygems'
+require 'optparse'
+require 'aws/s3'
+
+Awshucks.require_all_libs_relative_to(__FILE__)

data/lib/awshucks/cli.rb

@@ -0,0 +1,76 @@
+module Awshucks
+
+  class CLI
+
+    def self.execute
+      new.execute(ARGV)
+    end
+    
+    # parses the options and runs the command
+    def execute(args)
+      
+      option_parser.parse!(args) # changes args!
+      
+      if parsed_options.show_help || args.empty?
+        puts help_message
+      else        
+        Command.parse_and_execute(args, config)        
+      end
+      
+    rescue CommandError => e
+      error(e)
+    end
+
+    def help_message
+      option_parser.to_s
+    end
+
+    private
+    
+    def error(msg)
+      $stderr.puts(msg)
+      exit(-1)
+    end
+    
+    def option_parser
+      
+      OptionParser.new do |opts|
+        opts.banner = "Usage: awshucks [options] <command> [command options]"
+        opts.separator "awshucks version #{Awshucks::VERSION}"
+        opts.separator "type 'awshucks help <subcommand'> for help with a specific command"
+        opts.separator "The default configuration file used is awshucks.yml in the current working directory."
+        opts.separator ""
+        opts.separator "Options:"
+      
+        opts.on('-h', '--help', "Displays this help message") do
+          parsed_options.show_help = true
+        end
+        
+        opts.on('-c', '--config-file FILE', "Set the awshucks config file to FILE") do |config_file|
+          parsed_options.config_file = config_file
+        end
+        
+        opts.separator ""
+        opts.separator "Available Commands:"
+
+        Command.commands.keys.sort.each do |cmd|
+          opts.separator "    " << Command.commands[cmd].command
+        end
+        
+      end
+    end
+  
+    def parsed_options
+      @parsed_options ||= OpenStruct.new
+    end
+    
+    def config
+      config_file = parsed_options.config_file || 'awshucks.yml'
+      @config ||= returning(Config.new(config_file)) do |config|
+        config.help_message = help_message
+      end
+    end
+  
+  end
+  
+end

data/lib/awshucks/command.rb

@@ -0,0 +1,98 @@
+module Awshucks
+  
+  class CommandError < Exception; end
+  
+  class ExecutionError < CommandError; end
+    
+  class UnknownCommandError < CommandError
+    def initialize(msg)
+      super "Could not find command '#{msg}'"
+    end    
+  end
+  
+  # wraps up command parsing functions. Command class has class-level accessor for registering commands.
+  class Command
+    
+    ##### class methods #####
+    
+    class << self
+      
+      def inherited(klass)
+        # save it for later, klass doesn't get fully defined until after this callback
+        inherited_commands << klass
+      end
+      
+      def parse_and_execute(args, config)
+        key = args.shift
+        if key && commands[key] # be explicit about nil, no invalid command inheritance hijinks allowed
+          commands[key].execute(args, config)
+        else
+          raise UnknownCommandError, key
+        end
+      end
+      
+      def commands
+        @commands ||= returning({}) do |commands|
+          inherited_commands.each do |command|
+            commands[command.command] = command
+          end
+        end
+      end
+      
+      def option_string
+        new.option_parser.to_s # ehhh not a huge fan of this...
+      end
+      
+      def execute(args, config)
+        new.execute(args, config)
+      end
+            
+      attr_accessor :command
+      attr_accessor :usage
+      attr_accessor :description
+      
+      #######
+      private
+      #######
+      
+      def inherited_commands
+        @inherited ||= []
+      end
+      
+    end
+    
+    ##### instance methods ####
+    
+    def execute(args, config)
+      raise CommandError, "must implement the execute method!"
+    end
+    
+    def option_parser
+      @option_parser ||= OptionParser.new do |opts|
+        opts.banner = self.class.usage
+        opts.separator self.class.description
+        opts.separator ""
+        if respond_to?(:custom_options)
+          opts.separator "Options:"
+          custom_options(opts)
+        else
+          opts.separator "No options."
+        end
+      end
+    end
+    
+    # define the method custom_options(opts) if options are required for a command
+        
+    #######
+    private
+    #######
+    
+    # for use in the option definitions
+    def parsed_options
+      @parsed_options ||= OpenStruct.new
+    end
+    
+  end
+  
+  
+end

data/lib/awshucks/commands.rb

@@ -0,0 +1,3 @@
+Dir.entries(File.join(File.dirname(__FILE__), 'commands')).each do |file|
+  require "awshucks/commands/#{File.basename(file,".rb")}" if File.extname(file) == '.rb'
+end

data/lib/awshucks/commands/backup.rb

@@ -0,0 +1,59 @@
+module Awshucks
+    
+  class BackupCommand < Command
+    self.command = 'backup'
+    self.usage = 'backup [config name]'
+    self.description = "Backs up all locations specified in the config file. " +
+      "If you specify a config name, awshucks will back up only the files specified in that config."
+    
+    def execute(args, config)
+      @config = config
+      list = []
+      if args.empty?
+        list += config.backups
+      else
+        list << config.backup(args.first)
+      end
+      
+      list.each {|b| backup(b) }
+      
+    rescue UnknownBackupError
+      $stderr.puts "Unknown backup: #{args.first}"
+    end
+    
+    #######
+    private
+    #######
+    
+    attr_reader :config
+    
+    def backup(backup_info)
+      
+      puts "backing up #{backup_info.location}:"
+      
+      file_store = FileStore.new(config.connection, config.bucket, backup_info.name)
+      
+      base_path = File.expand_path(backup_info.location)
+      
+      begin
+        Scanner.each_file(base_path) do |file_info|        
+          print "  #{file_info.filename}: "
+          full_path = File.join(base_path, file_info.filename)
+          if file_store.different_from?(file_info)
+            print "changed, uploading (#{file_info.size} bytes)..."
+            file_store.store(file_info)
+            puts "done."
+          else
+            puts "unchanged."
+          end
+        end
+      ensure
+        puts "saving metadata cache..."
+        file_store.save_cache
+      end
+      
+    end # backup
+    
+  end # class
+  
+end