colin-safe 0.1.6

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Astrails Ltd.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,160 @@
+astrails-safe
+=============
+
+Simple mysql and filesystem backups with S3 support (with optional encryption)
+
+Motivation
+----------
+
+We needed a backup solution that will satisfy the following requirements:
+
+* opensource
+* simple to install and configure
+* support for simple ‘tar’ backups of directories (with includes/excludes)
+* support for simple mysqldump of mysql databases
+* support for symmetric or public key encryption
+* support for local filesystem and Amazon S3 for storage
+* support for backup rotation. we don’t want backups filling all the diskspace or cost a fortune on S3
+
+And since we didn't find any, we wrote our own :)
+
+Usage
+-----
+
+    Usage:
+       astrails-safe [OPTIONS] CONFIG_FILE
+    Options:
+      -h, --help           This help screen
+      -v, --verbose        be verbose, duh!
+      -n, --dry-run        just pretend, don't do anything.
+      -L, --local          skip S3
+
+Note: CONFIG_FILE will be created from template if missing
+
+Encryption
+----------
+
+If you want to encrypt your backups you have 2 options:
+* use simple password encryption
+* use GPG public key encryption
+
+For simple password, just add password entry in gpg section.
+For public key encryption you will need to create a public/secret keypair.
+
+We recommend to create your GPG keys only on your local machine and then
+transfer your public key to the server that will do the backups.
+
+This way the server will only know how to encrypt the backups but only you
+will be able to decrypt them using the secret key you have locally. Of course
+you MUST backup your backup encryption key :)
+We recommend also pringing the hard paper copy of your GPG key 'just in case'.
+
+The procedure to create and transfer the key is as follows:
+
+1. run 'gpg --gen-gen' on your local machine and follow onscreen instructions to create the key
+   (you can accept all the defaults).
+
+2. extract your public key into a file (assuming you used test@example.com as your key email):
+   gpg -a --export test@example.com > test@example.com.pub
+
+3. transfer public key to the server
+   scp backup@example.com root@example.com:
+
+4. import public key on the remote system:
+<pre>
+   $ gpg --import test@example.com.pub 
+   gpg: key 45CA9403: public key "Test Backup <test@example.com>" imported
+   gpg: Total number processed: 1
+   gpg:               imported: 1
+</pre>
+
+5. since we don't keep the secret part of the key on the remote server, gpg has
+   no way to know its yours and can be trusted.
+   To fix that we can sign it with other trusted key, or just directly modify its
+   trust level in gpg (use level 5):
+   <pre>
+     $ gpg --edit-key test@example.com
+     ...
+     Command> trust
+     ...
+     1 = I don't know or won't say
+     2 = I do NOT trust
+     3 = I trust marginally
+     4 = I trust fully
+     5 = I trust ultimately
+     m = back to the main menu
+     
+     Your decision? 5
+     ...
+     Command> quit
+   </pre>
+
+6. export your secret key for backup
+   (we recommend to print it on paper and burn to a CD/DVD and store in a safe place):
+   <pre>
+   $ gpg -a --export-secret-key test@example.com > test@example.com.key
+   </pre>
+
+
+Example configuration
+---------------------
+<pre>
+  safe do
+    local :path => "/backup/:kind/:id"
+
+    s3 do
+      key "...................."
+      secret "........................................"
+      bucket "backup.astrails.com"
+      path "servers/alpha/:kind/:id"
+    end
+
+    gpg do
+      # symmetric encryption key
+      # password "qwe"
+
+      # public GPG key (must be known to GPG, i.e. be on the keyring)
+      key "backup@astrails.com"
+    end
+
+    keep do
+      local 2
+      s3 2
+    end
+
+    mysqldump do
+      options "-ceKq --single-transaction --create-options"
+
+      user "root"
+      password "............"
+      socket "/var/run/mysqld/mysqld.sock"
+
+      database :blog
+      database :servershape
+      database :astrails_com
+      database :secret_project_com
+
+    end
+
+    tar do
+      archive "git-repositories", :files => "/home/git/repositories"
+      archive "dot-configs",      :files => "/home/*/.[^.]*"
+      archive "etc",              :files => "/etc", :exclude => "/etc/puppet/other"
+
+      archive "blog-astrails-com" do
+        files "/var/www/blog.astrails.com/"
+        exclude ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"]
+      end
+
+      archive "astrails-com" do
+        files "/var/www/astrails.com/"
+        exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
+      end
+    end
+  end
+</pre>
+
+Copyright
+---------
+
+Copyright (c) 2009 Astrails Ltd. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,52 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "safe"
+    gem.summary = %Q{Backup filesystem and MySQL to Amazon S3 (with encryption)}
+    gem.description = "Simple tool to backup MySQL databases and filesystem locally or to Amazon S3 (with optional encryption)"
+    gem.email = "we@astrails.com"
+    gem.homepage = "http://github.com/astrails/safe"
+    gem.authors  = ["Astrails Ltd."]
+    gem.files = FileList["[A-Z]*.*", "{bin,examples,generators,lib,rails,spec,test,templates}/**/*", 'Rakefile', 'LICENSE*']
+
+    gem.add_dependency("aws-s3")
+
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'micronaut/rake_task'
+Micronaut::RakeTask.new(:examples) do |examples|
+  examples.pattern = 'examples/**/*_example.rb'
+  examples.ruby_opts << '-Ilib -Iexamples'
+end
+
+Micronaut::RakeTask.new(:rcov) do |examples|
+  examples.pattern = 'examples/**/*_example.rb'
+  examples.rcov_opts = '-Ilib -Iexamples'
+  examples.rcov = true
+end
+
+
+task :default => :examples
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION.yml')
+    config = YAML.load(File.read('VERSION.yml'))
+    version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "safe #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:major: 0
+:minor: 1
+:patch: 6

data/bin/astrails-safe

@@ -0,0 +1,59 @@
+#!/usr/bin/env ruby
+
+
+require 'tempfile'
+require 'rubygems'
+require 'fileutils'
+require "aws/s3"
+require 'yaml'
+
+#require 'ruby-debug'
+#$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+
+require 'astrails/safe'
+include Astrails::Safe
+
+def die(msg)
+  puts "ERROR: #{msg}"
+  exit 1
+end
+
+def usage
+  puts <<-END
+Usage: astrails-safe [OPTIONS] CONFIG_FILE
+Options:
+  -h, --help           This help screen
+  -v, --verbose        be verbose, duh!
+  -n, --dry-run        just pretend, don't do anything.
+  -L, --local          skip S3
+
+Note: config file will be created from template if missing
+END
+  exit 1
+end
+
+def process_options
+  usage if ARGV.delete("-h") || ARGV.delete("--help")
+  $_VERBOSE = ARGV.delete("-v") || ARGV.delete("--verbose")
+  $DRY_RUN = ARGV.delete("-n") || ARGV.delete("--dry-run")
+  $LOCAL   = ARGV.delete("-L") || ARGV.delete("--local")
+  usage unless ARGV.first
+  $CONFIG_FILE_NAME = File.expand_path(ARGV.first)
+end
+
+def main
+  process_options
+
+  unless File.exists?($CONFIG_FILE_NAME)
+    die "Missing configuration file. NOT CREATED! Rerun w/o the -n argument to create a template configuration file." if $DRY_RUN
+
+    FileUtils.cp File.join(Astrails::Safe::ROOT, "templates", "script.rb"), $CONFIG_FILE_NAME
+
+    die "Created default #{$CONFIG_FILE_NAME}. Please edit and run again."
+  end
+
+
+  load($CONFIG_FILE_NAME)
+end
+
+main

data/examples/example_helper.rb

@@ -0,0 +1,19 @@
+require 'rubygems'
+require 'micronaut'
+require 'ruby-debug'
+
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+
+require 'astrails/safe'
+
+def not_in_editor?
+  !(ENV.has_key?('TM_MODE') || ENV.has_key?('EMACS') || ENV.has_key?('VIM'))
+end
+
+Micronaut.configure do |c|
+  c.color_enabled = not_in_editor?
+  c.filter_run :focused => true
+  c.mock_with :rr
+end
+

data/examples/unit/config_example.rb

@@ -0,0 +1,126 @@
+require File.expand_path(File.dirname(__FILE__) + '/../example_helper')
+
+describe Astrails::Safe::Config do
+  it "foo" do
+    config = Astrails::Safe::Config::Node.new do
+      local do
+        path "path"
+      end
+
+      s3 do
+        key "s3 key"
+        secret "secret"
+        bucket "bucket"
+        path "path1"
+      end
+
+      gpg do
+        key "gpg-key"
+        password "astrails"
+      end
+
+      keep do
+        local 4
+        s3 20
+      end
+
+      mysqldump do
+        options "-ceKq --single-transaction --create-options"
+
+        user "astrails"
+        password ""
+        host "localhost"
+        port 3306
+        socket "/var/run/mysqld/mysqld.sock"
+
+        database :blog
+
+        database :production do
+          keep :local => 3
+
+          gpg do
+            password "custom-production-pass"
+          end
+
+          skip_tables [:logger_exceptions, :request_logs]
+        end
+
+      end
+
+
+      tar do
+        archive "git-repositories" do
+          files "/home/git/repositories"
+        end
+
+        archive "etc-files" do
+          files "/etc"
+          exclude "/etc/puppet/other"
+        end
+
+        archive "dot-configs" do
+          files "/home/*/.[^.]*"
+        end
+
+        archive "blog" do
+          files "/var/www/blog.astrails.com/"
+          exclude ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"]
+        end
+
+        archive :misc do
+          files [ "/backup/*.rb" ]
+        end
+      end
+
+    end
+
+    expected = {
+      "local" => {"path" => "path"},
+
+      "s3" => {
+        "key" => "s3 key",
+        "secret" => "secret",
+        "bucket" => "bucket",
+        "path" => "path1",
+      },
+
+      "gpg" => {"password" => "astrails", "key" => "gpg-key"},
+
+      "keep" => {"s3" => 20, "local" => 4},
+
+      "mysqldump" => {
+        "options" => "-ceKq --single-transaction --create-options",
+        "user" => "astrails",
+        "password" => "",
+        "host" => "localhost",
+        "port" => 3306,
+        "socket" => "/var/run/mysqld/mysqld.sock",
+
+        "databases" => {
+          "blog" => {},
+          "production" => {
+           "keep" => {"local" => 3},
+           "gpg" => {"password" => "custom-production-pass"},
+            "skip_tables" => ["logger_exceptions", "request_logs"],
+          },
+        },
+      },
+      "tar" => {
+        "archives" => {
+          "git-repositories" => {"files" => "/home/git/repositories"},
+          "etc-files" => {"files" => "/etc", "exclude" => "/etc/puppet/other"},
+          "dot-configs" => {"files" => "/home/*/.[^.]*"},
+          "blog" => {
+            "files" => "/var/www/blog.astrails.com/",
+            "exclude" => ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"],
+          },
+          "misc" => { "files" => ["/backup/*.rb"] },
+        },
+      },
+    }
+
+    config.to_hash.should == expected
+
+  end
+end
+

data/examples/unit/stream_example.rb

@@ -0,0 +1,33 @@
+require File.expand_path(File.dirname(__FILE__) + '/../example_helper')
+
+describe Astrails::Safe::Stream do
+
+  before(:each) do
+    @parent = Object.new
+    @stream = Astrails::Safe::Stream.new(@parent)
+    @r = rand(10)
+  end
+
+  def self.it_delegates_to_parent(prop)
+    it "delegates #{prop} to parent if not set" do
+      mock(@parent).__send__(prop) {@r}
+      @stream.send(prop).should == @r
+    end
+  end
+
+  def self.it_delegates_to_parent_with_cache(prop)
+    it_delegates_to_parent(prop)
+
+    it "uses cached value for #{prop}" do
+      dont_allow(@parent).__send__(prop)
+      @stream.instance_variable_set "@#{prop}", @r + 1
+      @stream.send(prop).should == @r + 1
+    end
+  end
+
+  it_delegates_to_parent_with_cache :id
+  it_delegates_to_parent_with_cache :config
+
+  it_delegates_to_parent :filename
+
+end

data/lib/astrails/safe.rb

@@ -0,0 +1,41 @@
+require 'extensions/mktmpdir'
+require 'astrails/safe/tmp_file'
+
+require 'astrails/safe/config/node'
+require 'astrails/safe/config/builder'
+
+require 'astrails/safe/stream'
+
+require 'astrails/safe/source'
+require 'astrails/safe/mysqldump'
+require 'astrails/safe/archive'
+
+require 'astrails/safe/pipe'
+require 'astrails/safe/gpg'
+require 'astrails/safe/gzip'
+
+require 'astrails/safe/sink'
+require 'astrails/safe/local'
+require 'astrails/safe/s3'
+
+
+module Astrails
+  module Safe
+    ROOT = File.join(File.dirname(__FILE__), "..", "..")
+
+    def timestamp
+      @timestamp ||= Time.now.strftime("%y%m%d-%H%M")
+    end
+
+    def safe(&block)
+      config = Config::Node.new(&block)
+      #config.dump
+
+      Astrails::Safe::Mysqldump.run(config[:mysqldump, :databases])
+      Astrails::Safe::Archive.run(config[:tar, :archives])
+
+      Astrails::Safe::TmpFile.cleanup
+    end
+  end
+end
+

data/lib/astrails/safe/archive.rb

@@ -0,0 +1,24 @@
+module Astrails
+  module Safe
+    class Archive < Source
+
+      def command
+        "tar -cf - #{@config[:options]} #{tar_exclude_files} #{tar_files}"
+      end
+
+      def extension; '.tar'; end
+
+      protected
+
+      def tar_exclude_files
+        [*@config[:exclude]].compact.map{|x| "--exclude=#{x}"} * " "
+      end
+
+      def tar_files
+        raise RuntimeError, "missing files for tar" unless @config[:files]
+        [*@config[:files]] * " "
+      end
+
+    end
+  end
+end

data/lib/astrails/safe/config/builder.rb

@@ -0,0 +1,60 @@
+module Astrails
+  module Safe
+    module Config
+      class Builder
+        COLLECTIONS = %w/database archive/
+        ITEMS = %w/s3 key secret bucket path gpg password keep local mysqldump options
+        user host port socket skip_tables tar files exclude filename/
+        NAMES = COLLECTIONS + ITEMS
+        def initialize(node)
+          @node = node
+        end
+
+        # supported args:
+        #   args = [value]
+        #   args = [id, data]
+        #   args = [data]
+        # id/value - simple values, data - hash
+        def method_missing(sym, *args, &block)
+          return super unless NAMES.include?(sym.to_s)
+
+          # do we have id or value?
+          unless args.first.is_a?(Hash)
+            id_or_value = args.shift # nil for args == []
+          end
+
+          id_or_value = id_or_value.map {|v| v.to_s} if id_or_value.is_a?(Array)
+
+          # do we have data hash?
+          if data = args.shift
+            die "#{sym}: hash expected: #{data.inspect}" unless data.is_a?(Hash)
+          end
+
+          #puts "#{sym}: args=#{args.inspect}, id_or_value=#{id_or_value}, data=#{data.inspect}, block=#{block.inspect}"
+
+          die "#{sym}: unexpected: #{args.inspect}" unless args.empty?
+          die "#{sym}: missing arguments" unless id_or_value || data || block
+
+          if COLLECTIONS.include?(sym.to_s) && id_or_value
+            data ||= {}
+          end
+
+          if !data && !block
+            # simple value assignment
+            @node[sym] = id_or_value
+
+          elsif id_or_value
+            # collection element with id => create collection node and a subnode in it
+            key = sym.to_s + "s"
+            collection = @node[key] || @node.set(key, {})
+            collection.set(id_or_value, data || {}, &block)
+
+          else
+            # simple subnode
+            @node.set(sym, data || {}, &block)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/astrails/safe/config/node.rb

@@ -0,0 +1,67 @@
+require 'astrails/safe/config/builder'
+module Astrails
+  module Safe
+    module Config
+      class Node
+        attr_reader :parent
+        def initialize(parent = nil, data = {}, &block)
+          @parent, @data = parent, {}
+          data.each { |k, v| self[k] = v }
+          Builder.new(self).instance_eval(&block) if block
+        end
+
+        # looks for the path from this node DOWN. will not delegate to parent
+        def get(*path)
+          key = path.shift
+          value = @data[key.to_s]
+          return value if value && path.empty?
+
+          value && value.get(*path)
+        end
+
+        # recursive find
+        # starts at the node and continues to the parent
+        def find(*path)
+          get(*path) || @parent && @parent.find(*path)
+        end
+        alias :[] :find
+
+        def set(key, value, &block)
+          @data[key.to_s] =
+            if value.is_a?(Hash)
+              Node.new(self, value, &block)
+            else
+              raise(ArgumentError, "#{key}: no block supported for simple values") if block
+              value
+            end
+        end
+        alias :[]= :set
+
+        def each(&block)
+          @data.each(&block)
+        end
+        include Enumerable
+
+        def to_hash
+          @data.keys.inject({}) do |res, key|
+            value = @data[key]
+            res[key] = value.is_a?(Node) ? value.to_hash : value
+            res
+          end
+        end
+
+        def dump(indent = "")
+          @data.each do |key, value|
+            if value.is_a?(Node)
+              puts "#{indent}#{key}:"
+              value.dump(indent + "    ")
+            else
+              puts "#{indent}#{key}: #{value.inspect}"
+            end
+          end
+        end
+
+      end
+    end
+  end
+end

data/lib/astrails/safe/gpg.rb

@@ -0,0 +1,42 @@
+module Astrails
+  module Safe
+    class Gpg < Pipe
+
+      def compressed?
+        active? || @parent.compressed?
+      end
+
+      protected
+
+      def pipe
+        if key
+          rise RuntimeError, "can't use both gpg password and pubkey" if password
+          "|gpg -e -r #{key}"
+        elsif password
+          "|gpg -c --passphrase-file #{gpg_password_file(password)}"
+        end
+      end
+
+      def extension
+        ".gpg" if active?
+      end
+
+      def active?
+        password || key
+      end
+
+      def password
+        @password ||= config[:gpg, :password]
+      end
+
+      def key
+        @key ||= config[:gpg, :key]
+      end
+
+      def gpg_password_file(pass)
+        return "TEMP_GENERATED_FILENAME" if $DRY_RUN
+        Astrails::Safe::TmpFile.create("gpg-pass") { |file| file.write(pass) }
+      end
+    end
+  end
+end

data/lib/astrails/safe/gzip.rb

@@ -0,0 +1,25 @@
+module Astrails
+  module Safe
+    class Gzip < Pipe
+
+      def compressed?
+        true
+      end
+
+      protected
+
+      def pipe
+        "|gzip" if active?
+      end
+
+      def extension
+        ".gz" if active?
+      end
+
+      def active?
+        !@parent.compressed?
+      end
+
+    end
+  end
+end

data/lib/astrails/safe/local.rb

@@ -0,0 +1,54 @@
+module Astrails
+  module Safe
+    class Local < Sink
+
+      def open(&block)
+        return @parent.open(&block) unless active?
+        run
+        File.open(path, &block) unless $DRY_RUN
+      end
+
+      protected
+
+      def active?
+        # S3 can't upload from pipe. it needs to know file size, so we must pass through :local
+        # will change once we add SSH sink
+        true
+      end
+
+      def prefix
+        @prefix ||= File.expand_path(expand(@config[:local, :path] || raise(RuntimeError, "missing :local/:path in configuration")))
+      end
+
+      def command
+        "#{@parent.command} > #{path}"
+      end
+
+      def save
+        puts "command: #{command}" if $_VERBOSE
+        unless $DRY_RUN
+          FileUtils.mkdir_p(prefix) unless File.directory?(prefix)
+          system command
+        end
+      end
+
+      def cleanup
+        return unless keep = @config[:keep, :local]
+
+        base = File.basename(filename).split(".").first
+
+        pattern = File.join(prefix, "#{base}*")
+        puts "listing files #{pattern.inspect}" if $_VERBOSE
+        files = Dir[pattern] .
+          select{|f| File.file?(f)} .
+          sort
+
+        cleanup_with_limit(files, keep) do |f|
+          puts "removing local file #{f}" if $DRY_RUN || $_VERBOSE
+          File.unlink(f) unless $DRY_RUN
+        end
+      end
+
+    end
+  end
+end

data/lib/astrails/safe/mysqldump.rb

@@ -0,0 +1,31 @@
+module Astrails
+  module Safe
+    class Mysqldump < Source
+
+      def command
+        @commanbd ||= "mysqldump --defaults-extra-file=#{mysql_password_file} #{@config[:options]} #{mysql_skip_tables} #{@id}"
+      end
+
+      def extension; '.sql'; end
+
+      protected
+
+      def mysql_password_file
+        Astrails::Safe::TmpFile.create("mysqldump") do |file|
+          file.puts "[mysqldump]"
+          %w/user password socket host port/.each do |k|
+            v = @config[k]
+            file.puts "#{k} = #{v}" if v
+          end
+        end
+      end
+
+      def mysql_skip_tables
+        if skip_tables = @config[:skip_tables]
+          [*skip_tables].map { |t| "--ignore-table=#{@id}.#{t}" } * " "
+        end
+      end
+
+    end
+  end
+end

data/lib/astrails/safe/pipe.rb

@@ -0,0 +1,15 @@
+module Astrails
+  module Safe
+    class Pipe < Stream
+
+      def command
+        "#{@parent.command}#{pipe}"
+      end
+
+      def filename
+        "#{@parent.filename}#{extension}"
+      end
+
+    end
+  end
+end

data/lib/astrails/safe/s3.rb

@@ -0,0 +1,68 @@
+module Astrails
+  module Safe
+    class S3 < Sink
+
+      protected
+
+      def active?
+        bucket && key && secret
+      end
+
+      def prefix
+        @prefix ||= expand(config[:s3, :path] || expand(config[:local, :path] || ":kind/:id"))
+      end
+
+      def save
+        # needed in cleanup even on dry run
+        AWS::S3::Base.establish_connection!(:access_key_id => key, :secret_access_key => secret, :use_ssl => true) unless $LOCAL
+
+        file = @parent.open
+        puts "Uploading #{bucket}:#{path}" if $_VERBOSE || $DRY_RUN
+        unless $DRY_RUN || $LOCAL
+          AWS::S3::Bucket.create(bucket)
+          AWS::S3::S3Object.store(path, file, bucket)
+          puts "...done" if $_VERBOSE
+        end
+        file.close if file
+
+      end
+
+      def cleanup
+
+        return if $LOCAL
+
+        return unless keep = @config[:keep, :s3]
+
+        bucket = @config[:s3, :bucket]
+
+        base = File.basename(filename).split(".").first
+
+        puts "listing files in #{bucket}:#{prefix}/#{base}"
+        files = AWS::S3::Bucket.objects(bucket, :prefix => "#{prefix}/#{base}", :max_keys => keep * 2)
+        puts files.collect {|x| x.key} if $_VERBOSE
+
+        files = files.
+          collect {|x| x.key}.
+          sort
+
+        cleanup_with_limit(files, keep) do |f|
+          puts "removing s3 file #{bucket}:#{f}" if $DRY_RUN || $_VERBOSE
+          AWS::S3::Bucket.find(bucket)[f].delete unless $DRY_RUN || $LOCAL
+        end
+      end
+
+      def bucket
+        config[:s3, :bucket]
+      end
+
+      def key
+        config[:s3, :key]
+      end
+
+      def secret
+        config[:s3, :secret]
+      end
+
+    end
+  end
+end

data/lib/astrails/safe/sink.rb

@@ -0,0 +1,33 @@
+module Astrails
+  module Safe
+    class Sink < Stream
+
+      def run
+        if active?
+          save
+          cleanup
+        else
+          @parent.run
+        end
+      end
+
+      protected
+
+      # prefix is defined in subclass
+      def path
+        @path ||= File.join(prefix, filename)
+      end
+
+      # call block on files to be removed (all except for the LAST 'limit' files
+      def cleanup_with_limit(files, limit, &block)
+        return unless files.size > limit
+
+        to_remove = files[0..(files.size - limit - 1)]
+        # TODO: validate here
+        to_remove.each(&block)
+      end
+
+    end
+  end
+end
+

data/lib/astrails/safe/source.rb

@@ -0,0 +1,31 @@
+module Astrails
+  module Safe
+    class Source < Stream
+
+      def initialize(id, config)
+        @id, @config = id, config
+      end
+
+      def filename
+        @filename ||= expand(":kind-:id.:timestamp#{extension}")
+      end
+
+      # process each config key as source (with full pipe)
+      def self.run(config)
+        unless config
+          puts "No configuration found for #{human_name}"
+          return
+        end
+
+        config.each do |key, value|
+          stream = [Gpg, Gzip, Local, S3].inject(new(key, value)) do |res, klass|
+            klass.new(res)
+          end
+          stream.run
+        end
+      end
+
+    end
+  end
+end
+

data/lib/astrails/safe/stream.rb

@@ -0,0 +1,45 @@
+module Astrails
+  module Safe
+    class Stream
+
+      def initialize(parent)
+        @parent = parent
+      end
+
+      def id
+        @id ||= @parent.id
+      end
+
+      def config
+        @config ||= @parent.config
+      end
+
+      def filename
+        @parent.filename
+      end
+
+      def compressed?
+        @parent && @parent.compressed?
+      end
+
+      protected
+
+      def self.human_name
+        name.split('::').last.downcase
+      end
+
+      def kind
+        @parent ? @parent.kind : self.class.human_name
+      end
+
+      def expand(path)
+        path .
+          gsub(/:kind\b/, kind) .
+          gsub(/:id\b/, id) .
+          gsub(/:timestamp\b/, timestamp)
+      end
+
+    end
+  end
+end
+

data/lib/astrails/safe/tmp_file.rb

@@ -0,0 +1,25 @@
+require 'tmpdir'
+module Astrails
+  module Safe
+    module TmpFile
+      @KEEP_FILES = []
+      TMPDIR = Dir.mktmpdir
+
+      def self.cleanup
+        FileUtils.remove_entry_secure TMPDIR
+      end
+
+      def self.create(name)
+        # create temp directory
+
+        file = Tempfile.new(name, TMPDIR)
+
+        yield file
+
+        file.close
+        @KEEP_FILES << file # so that it will not get gcollected and removed from filesystem until the end
+        file.path
+      end
+    end
+  end
+end

data/lib/extensions/mktmpdir.rb

@@ -0,0 +1,45 @@
+require 'tmpdir'
+
+unless Dir.respond_to?(:mktmpdir)
+  # backward compat for 1.8.6
+  class Dir
+    def Dir.mktmpdir(prefix_suffix=nil, tmpdir=nil)
+      case prefix_suffix
+      when nil
+        prefix = "d"
+        suffix = ""
+      when String
+        prefix = prefix_suffix
+        suffix = ""
+      when Array
+        prefix = prefix_suffix[0]
+        suffix = prefix_suffix[1]
+      else
+        raise ArgumentError, "unexpected prefix_suffix: #{prefix_suffix.inspect}"
+      end
+      tmpdir ||= Dir.tmpdir
+      t = Time.now.strftime("%Y%m%d")
+      n = nil
+      begin
+        path = "#{tmpdir}/#{prefix}#{t}-#{$$}-#{rand(0x100000000).to_s(36)}"
+        path << "-#{n}" if n
+        path << suffix
+        Dir.mkdir(path, 0700)
+      rescue Errno::EEXIST
+        n ||= 0
+        n += 1
+        retry
+      end
+
+      if block_given?
+        begin
+          yield path
+        ensure
+          FileUtils.remove_entry_secure path
+        end
+      else
+        path
+      end
+    end
+  end
+end

data/templates/script.rb

@@ -0,0 +1,109 @@
+safe do
+
+  # backup file path (not including filename)
+  # supported substitutions:
+  #   :kind -> backup 'engine' kind, e.g. "mysqldump" or "archive"
+  #   :id -> backup 'id', e.g. "blog", "production", etc.
+  #   :timestamp -> current run timestamp (same for all the backups in the same 'run')
+  # you can set separate :path for all backups (or once globally here)
+  local do
+    path "/backup/:kind/"
+  end
+
+  ## uncomment to enable uploads to Amazon S3
+  ## Amazon S3 auth (optional)
+  ## don't forget to add :s3 to the 'store' list
+  # s3 do
+  #   key YOUR_S3_KEY
+  #   secret YOUR_S3_SECRET
+  #   bucket S3_BUCKET
+  #   # path for uploads to S3. supports same substitution like :local/:path
+  #   path ":kind/" # this is default
+  # end
+
+  ## alternative style:
+  # s3 :key => YOUR_S3_KEY, :secret => YOUR_S3_SECRET, :bucket => S3_BUCKET
+
+  ## uncomment to enable GPG encryption.
+  ## Note: you can use public 'key' or symmetric password but not both!
+  # gpg do
+  #   # key "backup@astrails.com"
+  #   password "astrails"
+  # end
+
+  ## uncomment to enable backup rotation. keep only given number of latest
+  ## backups. remove the rest
+  # keep do
+  #   local 4 # keep 4 local backups
+  #   s3 20 # keep 20 S3 backups
+  # end
+
+  # backup mysql databases with mysqldump
+  mysqldump do
+    # you can override any setting from parent in a child:
+    options "-ceKq --single-transaction --create-options"
+
+    user "astrails"
+    password ""
+    # host "localhost"
+    # port 3306
+    socket "/var/run/mysqld/mysqld.sock"
+
+    # database is a 'collection' element. it must have a hash or block parameter
+    # it will be 'collected' in a 'databases', with database id (1st arg) used as hash key
+    # the following code will create mysqldump/databases/blog and mysqldump/databases/mysql ocnfiguration 'nodes'
+
+    # backup database with default values
+    # database :blog
+
+    # backup overriding some values
+    # database :production do
+    #   # you can override 'partially'
+    #   keep :local => 3
+    #   # keep/local is 3, and keep/s3 is 20 (from parent)
+
+    #   # local override for gpg password
+    #   gpg do
+    #     password "custom-production-pass"
+    #   end
+
+    #   skip_tables [:logger_exceptions, :request_logs] # skip those tables during backup
+    # end
+
+  end
+
+
+  tar do
+    # 'archive' is a collection item, just like 'database'
+    # archive "git-repositories" do
+    #   # files and directories to backup
+    #   files "/home/git/repositories"
+    # end
+
+    # archive "etc-files" do
+    #   files "/etc"
+    #   # exlude those files/directories
+    #   exclude "/etc/puppet/other"
+    # end
+
+    # archive "dot-configs" do
+    #   files "/home/*/.[^.]*"
+    # end
+
+    # archive "blog" do
+    #   files "/var/www/blog.astrails.com/"
+    #   # specify multiple files/directories as array
+    #   exclude ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"]
+    # end
+
+    # archive "site" do
+    #   files "/var/www/astrails.com/"
+    #   exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
+    # end
+
+    # archive :misc do
+    #   files [ "/backup/*.rb" ]
+    # end
+  end
+
+end

metadata

@@ -0,0 +1,88 @@
+--- !ruby/object:Gem::Specification 
+name: colin-safe
+version: !ruby/object:Gem::Version 
+  version: 0.1.6
+platform: ruby
+authors: 
+- Astrails Ltd.
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-04-16 00:00:00 -07:00
+default_executable: astrails-safe
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: aws-s3
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+description: Simple tool to backup MySQL databases and filesystem locally or to Amazon S3 (with optional encryption)
+email: we@astrails.com
+executables: 
+- astrails-safe
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.markdown
+files: 
+- LICENSE
+- README.markdown
+- Rakefile
+- VERSION.yml
+- bin/astrails-safe
+- examples/example_helper.rb
+- examples/unit/config_example.rb
+- examples/unit/stream_example.rb
+- lib/astrails/safe.rb
+- lib/astrails/safe/archive.rb
+- lib/astrails/safe/config/builder.rb
+- lib/astrails/safe/config/node.rb
+- lib/astrails/safe/gpg.rb
+- lib/astrails/safe/gzip.rb
+- lib/astrails/safe/local.rb
+- lib/astrails/safe/mysqldump.rb
+- lib/astrails/safe/pipe.rb
+- lib/astrails/safe/s3.rb
+- lib/astrails/safe/sink.rb
+- lib/astrails/safe/source.rb
+- lib/astrails/safe/stream.rb
+- lib/astrails/safe/tmp_file.rb
+- lib/extensions/mktmpdir.rb
+- templates/script.rb
+has_rdoc: true
+homepage: http://github.com/astrails/safe
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Backup filesystem and MySQL to Amazon S3 (with encryption)
+test_files: 
+- examples/example_helper.rb
+- examples/unit/stream_example.rb
+- examples/unit/config_example.rb