astrails-safe 0.2.7 → 0.3.0

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+tags

data/.rspec

@@ -0,0 +1,3 @@
+--colour
+--format d
+--profile

data/CHANGELOG

@@ -0,0 +1,25 @@
+0.2.8
+
+* ruby 1.9.2 compatibility (tests mostly)
+* code review, and tons of small fixes
+* check file size before attempting to upload to cloudfiles
+* testing framework changed from micronaut to rspec
+
+0.2.7
+
+* default options for gpg now include '--no-use-agent'
+* support for 'command' option for gpg
+* quote values in mysql password file
+* add 'lib' to $:
+* [EXPERIMENTAL] Rackspace Cloud Files support
+
+0.2.6
+
+* fix typo in the template config file. (change option to options in pgdump)
+* add example 'options' for tar in the template config file.
+* do not try to upload more then 5G of data to S3. print error instead
+
+0.2.5
+
+* Safety mesure: Disable overwrite of existing configuration keys except for multi-value keys
+  supported multi-value keys: skip_tables, exclude, files

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in astrails-safe.gemspec
+gemspec

data/{LICENSE → LICENSE.txt}

@@ -1,4 +1,6 @@
-Copyright (c) 2009 Astrails Ltd.
+Copyright (c) 2010-2013 Astrails Ltd.
+
+MIT License
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.markdown

@@ -1,12 +1,14 @@
-astrails-safe
-=============
+# astrails-safe
 
 Simple database and filesystem backups with S3 and Rackspace Cloud Files support (with optional encryption)
 
-Home: http://blog.astrails.com/astrails-safe
+* Home: [http://astrails.com/opensource/astrails-safe](http://astrails.com/opensource/astrails-safe)
+* Code: [http://github.com/astrails/safe](http://github.com/astrails/safe)
+* Blog: [http://blog.astrails.com/astrails-safe](http://blog.astrails.com/astrails-safe)
 
-Motivation
-----------
+[![Code Climate](https://codeclimate.com/github/astrails/safe.png)](https://codeclimate.com/github/astrails/safe)
+
+## Motivation
 
 We needed a backup solution that will satisfy the following requirements:
 
@@ -20,12 +22,11 @@ We needed a backup solution that will satisfy the following requirements:
 
 And since we didn't find any, we wrote our own :)
 
-Contributions
--------------
+## Contributions
 
 The following functionality was contributed by astrails-safe users:
 
-* PostgreSQL dump using pg_dump (by Mark Mansour <mark@stateofflux.com>)
+* PostgreSQL dump using `pg_dump` (by Mark Mansour <mark@stateofflux.com>)
 * Subversion dump using svndump (by Richard Luther <richard.luther@gmail.com>)
 * SFTP remote storage (by Adam <adam@mediadrive.ca>)
 * benchmarking output (By Neer)
@@ -36,18 +37,15 @@ The following functionality was contributed by astrails-safe users:
 
 Thanks to all :)
 
-Installation
-------------
+## Installation
 
     sudo gem install astrails-safe --source http://gemcutter.org
 
-Reporting problems
-------------------
+## Reporting problems
 
 Please report problems at the [Issues tracker](http://github.com/astrails/safe/issues)
 
-Usage
------
+## Usage
 
     Usage:
        astrails-safe [OPTIONS] CONFIG_FILE
@@ -57,10 +55,9 @@ Usage
       -n, --dry-run        just pretend, don't do anything.
       -L, --local          skip remote storage, only do local backups
 
-Note: CONFIG_FILE will be created from template if missing
+Note: CONFIG\_FILE will be created from template if missing
 
-Encryption
-----------
+## Encryption
 
 If you want to encrypt your backups you have 2 options:
 * use simple password encryption
@@ -92,24 +89,23 @@ The procedure to create and transfer the key is as follows:
    (you can accept all the defaults).
 
 2. extract your public key into a file (assuming you used test@example.com as your key email):
-   gpg -a --export test@example.com > test@example.com.pub
+   `gpg -a --export test@example.com > test@example.com.pub`
 
 3. transfer public key to the server
-   scp test@example.com.pub root@example.com:
+   `scp test@example.com.pub root@example.com:`
 
 4. import public key on the remote system:
-<pre>
-   $ gpg --import test@example.com.pub
-   gpg: key 45CA9403: public key "Test Backup <test@example.com>" imported
-   gpg: Total number processed: 1
-   gpg:               imported: 1
-</pre>
+
+    $ gpg --import test@example.com.pub
+    gpg: key 45CA9403: public key "Test Backup <test@example.com>" imported
+    gpg: Total number processed: 1
+    gpg:               imported: 1
 
 5. since we don't keep the secret part of the key on the remote server, gpg has
    no way to know its yours and can be trusted.
    To fix that we can sign it with other trusted key, or just directly modify its
    trust level in gpg (use level 5):
-   <pre>
+
      $ gpg --edit-key test@example.com
      ...
      Command> trust
@@ -124,119 +120,118 @@ The procedure to create and transfer the key is as follows:
      Your decision? 5
      ...
      Command> quit
-   </pre>
 
 6. export your secret key for backup
    (we recommend to print it on paper and burn to a CD/DVD and store in a safe place):
-   <pre>
-   $ gpg -a --export-secret-key test@example.com > test@example.com.key
-   </pre>
-
-
-Example configuration
----------------------
-<pre>
-  safe do
-    local :path => "/backup/:kind/:id"
-
-    s3 do
-      key "...................."
-      secret "........................................"
-      bucket "backup.astrails.com"
-      path "servers/alpha/:kind/:id"
-    end
 
-    cloudfiles do
-      username "..........."
-      api_key "................................."
-      container "safe_backup"
-      path ":kind/" # this is default
-      service_net false
-    end
+    $ gpg -a --export-secret-key test@example.com > test@example.com.key
 
-    sftp do
-      host "sftp.astrails.com"
-      user "astrails"
-      # port 8023
-      password "ssh password for sftp"
-    end
 
-    gpg do
-      command "/usr/local/bin/gpg"
-      options  "--no-use-agent"
-      # symmetric encryption key
-      # password "qwe"
 
-      # public GPG key (must be known to GPG, i.e. be on the keyring)
-      key "backup@astrails.com"
-    end
+## Example configuration
 
-    keep do
-      local 20
-      s3 100
-      cloudfiles 100
-      sftp 100
-    end
+    safe do
+      local :path => "/backup/:kind/:id"
 
-    mysqldump do
-      options "-ceKq --single-transaction --create-options"
+      s3 do
+        key "...................."
+        secret "........................................"
+        bucket "backup.astrails.com"
+        path "servers/alpha/:kind/:id"
+      end
 
-      user "root"
-      password "............"
-      socket "/var/run/mysqld/mysqld.sock"
+      cloudfiles do
+        user "..........."
+        api_key "................................."
+        container "safe_backup"
+        path ":kind/" # this is default
+        service_net false
+      end
 
-      database :blog
-      database :servershape
-      database :astrails_com
-      database :secret_project_com do
-        skip_tables "foo"
-        skip_tables ["bar", "baz"]
+      sftp do
+        host "sftp.astrails.com"
+        user "astrails"
+        # port 8023
+        password "ssh password for sftp"
       end
 
-    end
+      gpg do
+        command "/usr/local/bin/gpg"
+        options  "--no-use-agent"
+        # symmetric encryption key
+        # password "qwe"
 
-    svndump do
-      repo :my_repo do
-        repo_path "/home/svn/my_repo"
+        # public GPG key (must be known to GPG, i.e. be on the keyring)
+        key "backup@astrails.com"
       end
-    end
 
-    pgdump do
-      options "-i -x -O"   # -i => ignore version, -x => do not dump privileges (grant/revoke), -O => skip restoration of object ownership in plain text format
+      keep do
+        local 20
+        s3 100
+        cloudfiles 100
+        sftp 100
+      end
 
-      user "username"
-      password "............"  # shouldn't be used, instead setup ident.  Current functionality exports a password env to the shell which pg_dump uses - untested!
+      mysqldump do
+        options "-ceKq --single-transaction --create-options"
 
-      database :blog
-      database :stateofflux_com
-    end
+        user "root"
+        password "............"
+        socket "/var/run/mysqld/mysqld.sock"
+
+        database :blog
+        database :servershape
+        database :astrails_com
+        database :secret_project_com do
+          skip_tables "foo"
+          skip_tables ["bar", "baz"]
+        end
+
+      end
+
+      svndump do
+        repo :my_repo do
+          repo_path "/home/svn/my_repo"
+        end
+      end
+
+      pgdump do
+        options "-i -x -O"   # -i => ignore version, -x => do not dump privileges (grant/revoke), -O => skip restoration of object ownership in plain text format
 
-    tar do
-      options "-h" # dereference symlinks
-      archive "git-repositories", :files => "/home/git/repositories"
-      archive "dot-configs",      :files => "/home/*/.[^.]*"
-      archive "etc",              :files => "/etc", :exclude => "/etc/puppet/other"
+        user "username"
+        password "............"  # shouldn't be used, instead setup ident.  Current functionality exports a password env to the shell which pg_dump uses - untested!
 
-      archive "blog-astrails-com" do
-        files "/var/www/blog.astrails.com/"
-        exclude "/var/www/blog.astrails.com/log"
-        exclude "/var/www/blog.astrails.com/tmp"
+        database :blog
+        database :stateofflux_com
       end
 
-      archive "astrails-com" do
-        files "/var/www/astrails.com/"
-        exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
+      tar do
+        options "-h" # dereference symlinks
+        archive "git-repositories", :files => "/home/git/repositories"
+        archive "dot-configs",      :files => "/home/*/.[^.]*"
+        archive "etc",              :files => "/etc", :exclude => "/etc/puppet/other"
+
+        archive "blog-astrails-com" do
+          files "/var/www/blog.astrails.com/"
+          exclude "/var/www/blog.astrails.com/log"
+          exclude "/var/www/blog.astrails.com/tmp"
+        end
+
+        archive "astrails-com" do
+          files "/var/www/astrails.com/"
+          exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
+        end
       end
     end
-  end
-</pre>
 
-Reporting problems
-------------------
+## Contributing
 
-http://github.com/astrails/safe/issues
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
 
-Copyright
----------
+## Copyright
 
-Copyright (c) 2009 Astrails Ltd. See LICENSE for details.
+Copyright (c) 2010 Astrails Ltd. See LICENSE.txt for details.

data/Rakefile

@@ -1,58 +1,8 @@
-require 'rubygems'
-require 'rake'
+require "bundler/gem_tasks"
 
-begin
-  require 'jeweler'
-  Jeweler::Tasks.new do |gem|
-    gem.name = "astrails-safe"
-    gem.summary = %Q{Backup filesystem and databases (MySQL and PostgreSQL) locally or to a remote server/service (with encryption)}
-    gem.description = <<-DESC
-Astrails-Safe is a simple tool to backup databases (MySQL and PostgreSQL), Subversion repositories (with svndump) and just files.
-Backups can be stored locally or remotely and can be enctypted.
-Remote storage is supported on Amazon S3, Rackspace Cloud Files, or just plain SFTP.
-DESC
-    gem.email = "we@astrails.com"
-    gem.homepage = "http://blog.astrails.com/astrails-safe"
-    gem.authors  = ["Astrails Ltd."]
-    gem.files = FileList["[A-Z]*.*", "{bin,examples,generators,lib,rails,spec,test,templates}/**/*", 'Rakefile', 'LICENSE*']
+require 'rspec/core/rake_task'
 
-    gem.add_dependency("aws-s3")
-    gem.add_dependency("cloudfiles")
-    gem.add_dependency("net-sftp")
-
-    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
-  end
-rescue LoadError
-  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
-end
-
-require 'micronaut/rake_task'
-Micronaut::RakeTask.new(:examples) do |examples|
-  examples.pattern = 'examples/**/*_example.rb'
-  examples.ruby_opts << '-Ilib -Iexamples'
-end
-
-Micronaut::RakeTask.new(:rcov) do |examples|
-  examples.pattern = 'examples/**/*_example.rb'
-  examples.rcov_opts = '-Ilib -Iexamples -iastrails -x\/gems\/'
-  examples.rcov = true
-end
-
-
-task :default => :examples
-
-require 'rake/rdoctask'
-Rake::RDocTask.new do |rdoc|
-  if File.exist?('VERSION.yml')
-    config = YAML.load(File.read('VERSION.yml'))
-    version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
-  else
-    version = ""
-  end
-
-  rdoc.rdoc_dir = 'rdoc'
-  rdoc.title = "safe #{version}"
-  rdoc.rdoc_files.include('README*')
-  rdoc.rdoc_files.include('lib/**/*.rb')
-end
+desc "run specs"
+RSpec::Core::RakeTask.new
 
+task default: :spec

data/TODO

@@ -0,0 +1,11 @@
+- add 'silent'
+- handle errors from mysqldump
+- check that gpg is installed
+- support percona XtraBackup as an option instead of mysqldump [patches anyone :) ?]
+- backup validation:
+  - support for 'minsize' opition in backup that will check that produced backup is at least the expected size
+    this should catch many backup failure scenarious (like broken mysql connection, insufficient disk space etc.
+- support differencial backups
+  - it should be fairly easy for filesystem backups using tar's built in incremental functionality.
+  - for mysql need to use XtraBackup
+  - or we can keep the previous dump locally and store only diff with the latest dump