webtranslateit-safe 0.4.0

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2010-2013 Astrails Ltd.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,237 @@
+# webtranslateit-safe
+
+Simple database and filesystem backups with S3 and Rackspace Cloud Files support (with optional encryption)
+
+* Code: [http://github.com/webtranslateit/safe](http://github.com/webtranslateit/safe)
+
+## Motivation
+
+We needed a backup solution that will satisfy the following requirements:
+
+* opensource
+* simple to install and configure
+* support for simple ‘tar’ backups of directories (with includes/excludes)
+* support for simple mysqldump of mysql databases
+* support for symmetric or public key encryption
+* support for local filesystem, Amazon S3, and Rackspace Cloud Files for storage
+* support for backup rotation. we don’t want backups filling all the diskspace or cost a fortune on S3 or Cloud Files
+
+And since we didn't find any, we wrote our own :)
+
+## Contributions
+
+The following functionality was contributed by webtranslateit-safe users:
+
+* PostgreSQL dump using `pg_dump` (by Mark Mansour <mark@stateofflux.com>)
+* Subversion dump using svndump (by Richard Luther <richard.luther@gmail.com>)
+* SFTP remote storage (by Adam <adam@mediadrive.ca>)
+* benchmarking output (By Neer)
+* README fixes (by Bobby Wilson)
+* improved config file parsing (by Fedor Kocherga <fkocherga@gmail.com>)
+* mysql password file quoting (by Jonathan Sutherland <jonathan.sutherland@gmail.com>)
+* Rackspace Cloud Files support (by H. Wade Minter <minter@lunenburg.org>)
+* Plan FTP support (by seroy <seroy@bk.ru>)
+* mongodump support (by Matt Berther <matt@mattberther.com>)
+
+Thanks to all :)
+
+## Installation
+
+    gem install webtranslateit-safe
+
+## Reporting problems
+
+Please report problems at the [Issues tracker](http://github.com/webtranslateit/safe/issues)
+
+## Usage
+
+    Usage:
+       webtranslateit-safe [OPTIONS] CONFIG_FILE
+    Options:
+      -h, --help           This help screen
+      -v, --verbose        be verbose, duh!
+      -n, --dry-run        just pretend, don't do anything.
+      -L, --local          skip remote storage, only do local backups
+
+Note: CONFIG\_FILE will be created from template if missing
+
+## Encryption
+
+If you want to encrypt your backups you have 2 options:
+* use simple password encryption
+* use GPG public key encryption
+
+> IMPORTANT: some gpg installations automatically set 'use-agent' option in the default
+> configuration file that is created when you run gpg for the first time. This will cause
+> gpg to fail on the 2nd run if you don't have the agent running. The result is that
+> 'webtranslateit-safe' will work ONCE when you manually test it and then fail on any subsequent run.
+> The solution is to remove the 'use-agent' from the config file (usually /root/.gnupg/gpg.conf)
+> To mitigate this problem for the gpg 1.x series '--no-use-agent' option is added by defaults
+> to the autogenerated config file, but for gpg2 is doesn't work. as the manpage says it:
+> "This is dummy option. gpg2 always requires the agent." :(
+
+For simple password, just add password entry in gpg section.
+For public key encryption you will need to create a public/secret keypair.
+
+We recommend to create your GPG keys only on your local machine and then
+transfer your public key to the server that will do the backups.
+
+This way the server will only know how to encrypt the backups but only you
+will be able to decrypt them using the secret key you have locally. Of course
+you MUST backup your backup encryption key :)
+We recommend also pringing the hard paper copy of your GPG key 'just in case'.
+
+The procedure to create and transfer the key is as follows:
+
+1. run 'gpg --gen-key' on your local machine and follow onscreen instructions to create the key
+   (you can accept all the defaults).
+
+2. extract your public key into a file (assuming you used test@example.com as your key email):
+   `gpg -a --export test@example.com > test@example.com.pub`
+
+3. transfer public key to the server
+   `scp test@example.com.pub root@example.com:`
+
+4. import public key on the remote system:
+
+    $ gpg --import test@example.com.pub
+    gpg: key 45CA9403: public key "Test Backup <test@example.com>" imported
+    gpg: Total number processed: 1
+    gpg:               imported: 1
+
+5. since we don't keep the secret part of the key on the remote server, gpg has
+   no way to know its yours and can be trusted.
+   To fix that we can sign it with other trusted key, or just directly modify its
+   trust level in gpg (use level 5):
+
+     $ gpg --edit-key test@example.com
+     ...
+     Command> trust
+     ...
+     1 = I don't know or won't say
+     2 = I do NOT trust
+     3 = I trust marginally
+     4 = I trust fully
+     5 = I trust ultimately
+     m = back to the main menu
+
+     Your decision? 5
+     ...
+     Command> quit
+
+6. export your secret key for backup
+   (we recommend to print it on paper and burn to a CD/DVD and store in a safe place):
+
+    $ gpg -a --export-secret-key test@example.com > test@example.com.key
+
+
+
+## Example configuration
+
+    safe do
+      verbose true
+
+      local :path => "/backup/:kind/:id"
+
+      s3 do
+        key "...................."
+        secret "........................................"
+        bucket "backup.astrails.com"
+        path "servers/alpha/:kind/:id"
+      end
+
+      cloudfiles do
+        user "..........."
+        api_key "................................."
+        container "safe_backup"
+        path ":kind/" # this is default
+        service_net false
+      end
+
+      sftp do
+        host "sftp.astrails.com"
+        user "astrails"
+        # port 8023
+        password "ssh password for sftp"
+      end
+
+      gpg do
+        command "/usr/local/bin/gpg"
+        options  "--no-use-agent"
+        # symmetric encryption key
+        # password "qwe"
+
+        # public GPG key (must be known to GPG, i.e. be on the keyring)
+        key "backup@astrails.com"
+      end
+
+      keep do
+        local 20
+        s3 100
+        cloudfiles 100
+        sftp 100
+      end
+
+      mysqldump do
+        options "-ceKq --single-transaction --create-options"
+
+        user "root"
+        password "............"
+        socket "/var/run/mysqld/mysqld.sock"
+
+        database :blog
+        database :servershape
+        database :astrails_com
+        database :secret_project_com do
+          skip_tables "foo"
+          skip_tables ["bar", "baz"]
+        end
+
+      end
+
+      svndump do
+        repo :my_repo do
+          repo_path "/home/svn/my_repo"
+        end
+      end
+
+      pgdump do
+        options "-i -x -O"   # -i => ignore version, -x => do not dump privileges (grant/revoke), -O => skip restoration of object ownership in plain text format
+
+        user "username"
+        password "............"  # shouldn't be used, instead setup ident.  Current functionality exports a password env to the shell which pg_dump uses - untested!
+
+        database :blog
+        database :stateofflux_com
+      end
+
+      tar do
+        options "-h" # dereference symlinks
+        archive "git-repositories", :files => "/home/git/repositories"
+        archive "dot-configs",      :files => "/home/*/.[^.]*"
+        archive "etc",              :files => "/etc", :exclude => "/etc/puppet/other"
+
+        archive "blog-astrails-com" do
+          files "/var/www/blog.astrails.com/"
+          exclude "/var/www/blog.astrails.com/log"
+          exclude "/var/www/blog.astrails.com/tmp"
+        end
+
+        archive "astrails-com" do
+          files "/var/www/astrails.com/"
+          exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
+        end
+      end
+    end
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Copyright
+
+Copyright (c) 2010-2023 WebTranslateIt Software SL. See LICENSE.txt for details.

data/Rakefile

@@ -0,0 +1,8 @@
+require 'bundler/gem_tasks'
+
+require 'rspec/core/rake_task'
+
+desc 'run specs'
+RSpec::Core::RakeTask.new
+
+task default: :spec

data/TODO

@@ -0,0 +1,31 @@
+- refactor
+  - refactor out global variables. pass a config object around instead
+  - common logging
+  - remove 1.8.6 support
+  - module registry
+  - base => prefix ?
+  - move requires into specific modules
+	- config.foo instead of config[:foo]
+
+- features
+  - remote-only s3 support
+  - generic notifier support
+    - email notifier
+    - hipchat
+  - generic error notifier support
+    - email
+    - hipchat
+
+
+
+- add 'silent'
+- handle errors from mysqldump
+- check that gpg is installed
+- support percona XtraBackup as an option instead of mysqldump [patches anyone :) ?]
+- backup validation:
+  - support for 'minsize' opition in backup that will check that produced backup is at least the expected size
+    this should catch many backup failure scenarious (like broken mysql connection, insufficient disk space etc.
+- support differencial backups
+  - it should be fairly easy for filesystem backups using tar's built in incremental functionality.
+  - for mysql need to use XtraBackup
+  - or we can keep the previous dump locally and store only diff with the latest dump

data/bin/webtranslateit-safe

@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+
+require 'webtranslateit/safe'
+
+include WebTranslateIt::Safe
+
+def die(msg)
+  puts "ERROR: #{msg}"
+  exit 1
+end
+
+def usage
+  puts <<~END
+    Usage: webtranslateit-safe [OPTIONS] CONFIG_FILE
+    Options:
+      -h, --help           This help screen
+      -v, --verbose        be verbose, duh!
+      -n, --dry-run        just pretend, don't do anything.
+      -L, --local          skip S3 and Cloud Files
+
+    Note: config file will be created from template if missing
+  END
+  exit 1
+end
+
+OPTS = [
+  '-h', '--help',
+  '-v', '--verbose', '--not-verbose',
+  '-n', '--dry-run', '--not-dry-run',
+  '-L', '--local', '--not-local'
+].freeze
+def main
+  opts = ARGV & OPTS
+  args = ARGV - OPTS
+
+  usage unless args.first
+  usage if opts.delete('-h') || opts.delete('--help')
+
+  config_file = File.expand_path(args.first)
+
+  is_dry = (opts.delete('-n') || opts.delete('--dry-run')) && !opts.delete('--not-dry-run')
+  is_verbose = (opts.delete('-v') || opts.delete('--verbose')) && !opts.delete('--not-verbose')
+  is_local_only = (opts.delete('-L') || opts.delete('--local')) && !opts.delete('--not-local')
+
+  unless File.exist?(config_file)
+    die 'Missing configuration file. NOT CREATED! Rerun w/o the -n argument to create a template configuration file.' if is_dry
+
+    FileUtils.cp File.join(WebTranslateIt::Safe::ROOT, 'templates', 'script.rb'), config_file
+
+    die "Created default #{config_file}. Please edit and run again."
+  end
+
+  config = eval(File.read(config_file))
+
+  config[:verbose]    = is_verbose
+  config[:dry_run]    = is_dry
+  config[:local_only] = is_local_only
+
+  process config
+end
+
+main

data/lib/extensions/mktmpdir.rb

@@ -0,0 +1,45 @@
+require 'tmpdir'
+
+unless Dir.respond_to?(:mktmpdir)
+  # backward compat for 1.8.6
+  class Dir
+    def Dir.mktmpdir(prefix_suffix=nil, tmpdir=nil)
+      case prefix_suffix
+      when nil
+        prefix = 'd'
+        suffix = ''
+      when String
+        prefix = prefix_suffix
+        suffix = ''
+      when Array
+        prefix = prefix_suffix[0]
+        suffix = prefix_suffix[1]
+      else
+        raise ArgumentError, "unexpected prefix_suffix: #{prefix_suffix.inspect}"
+      end
+      tmpdir ||= Dir.tmpdir
+      t = Time.now.strftime('%Y%m%d')
+      n = nil
+      begin
+        path = "#{tmpdir}/#{prefix}#{t}-#{$$}-#{rand(0x100000000).to_s(36)}"
+        path << "-#{n}" if n
+        path << suffix
+        Dir.mkdir(path, 0700)
+      rescue Errno::EEXIST
+        n ||= 0
+        n += 1
+        retry
+      end
+
+      if block_given?
+        begin
+          yield path
+        ensure
+          FileUtils.remove_entry_secure path
+        end
+      else
+        path
+      end
+    end
+  end
+end

data/lib/webtranslateit/safe/archive.rb

@@ -0,0 +1,29 @@
+module WebTranslateIt
+
+  module Safe
+
+    class Archive < Source
+
+      def command
+        "tar -cf - #{config[:options]} #{tar_exclude_files} #{tar_files}"
+      end
+
+      def extension = '.tar'
+
+      protected
+
+      def tar_exclude_files
+        [*config[:exclude]].compact.map { |x| "--exclude=#{x}" }.join(' ')
+      end
+
+      def tar_files
+        raise 'missing files for tar' unless config[:files]
+
+        [*config[:files]].map(&:strip).join(' ')
+      end
+
+    end
+
+  end
+
+end

data/lib/webtranslateit/safe/backup.rb

@@ -0,0 +1,27 @@
+module WebTranslateIt
+
+  module Safe
+
+    class Backup
+
+      attr_accessor :id, :kind, :filename, :extension, :command, :compressed, :timestamp, :path
+
+      def initialize(opts = {})
+        opts.each do |k, v|
+          send("#{k}=", v)
+        end
+      end
+
+      def run(config, *mods)
+        mods.each do |mod|
+          mod = mod.to_s
+          mod[0] = mod[0..0].upcase
+          WebTranslateIt::Safe.const_get(mod).new(config, self).process
+        end
+      end
+
+    end
+
+  end
+
+end

data/lib/webtranslateit/safe/cloudfiles.rb

@@ -0,0 +1,77 @@
+module WebTranslateIt
+  module Safe
+    class Cloudfiles < Sink
+      MAX_CLOUDFILES_FILE_SIZE = 5368709120
+
+      def active?
+        container && user && api_key
+      end
+
+      protected
+
+      def path
+        @path ||= expand(config[:cloudfiles, :path] || config[:local, :path] || ':kind/:id')
+      end
+
+      # UGLY: we need this function for the reason that
+      # we can't double mock on ruby 1.9.2, duh!
+      # so we created this func to mock it all together
+      def get_file_size(path)
+        File.stat(path).size
+      end
+
+      def save
+        raise RuntimeError, 'pipe-streaming not supported for S3.' unless @backup.path
+
+        # needed in cleanup even on dry run
+        cf = CloudFiles::Connection.new(user, api_key, true, service_net) unless local_only?
+        puts "Uploading #{container}:#{full_path} from #{@backup.path}" if verbose? || dry_run?
+        unless dry_run? || local_only?
+          if get_file_size(@backup.path) > MAX_CLOUDFILES_FILE_SIZE
+            STDERR.puts "ERROR: File size exceeds maximum allowed for upload to Cloud Files (#{MAX_CLOUDFILES_FILE_SIZE}): #{@backup.path}"
+            return
+          end
+          benchmark = Benchmark.realtime do
+            cf_container = cf.create_container(container)
+            o = cf_container.create_object(full_path,true)
+            o.write(File.open(@backup.path))
+          end
+          puts '...done' if verbose?
+          puts('Upload took ' + sprintf('%.2f', benchmark) + ' second(s).') if verbose?
+        end
+      end
+
+      def cleanup
+        return if local_only?
+
+        return unless keep = config[:keep, :cloudfiles]
+
+        puts "listing files: #{container}:#{base}*" if verbose?
+        cf = CloudFiles::Connection.new(user, api_key, true, service_net) unless local_only?
+        cf_container = cf.container(container)
+        files = cf_container.objects(:prefix => base).sort
+
+        cleanup_with_limit(files, keep) do |f|
+          puts "removing Cloud File #{container}:#{f}" if dry_run? || verbose?
+          cf_container.delete_object(f) unless dry_run? || local_only?
+        end
+      end
+
+      def container
+        config[:cloudfiles, :container]
+      end
+
+      def user
+        config[:cloudfiles, :user]
+      end
+
+      def api_key
+        config[:cloudfiles, :api_key]
+      end
+
+      def service_net
+        config[:cloudfiles, :service_net] || false
+      end
+    end
+  end
+end

data/lib/webtranslateit/safe/config/builder.rb

@@ -0,0 +1,100 @@
+module WebTranslateIt
+
+  module Safe
+
+    module Config
+
+      class Builder
+
+        def initialize(node, data = {})
+          @node = node
+          data.each { |k, v| send k, v }
+        end
+
+
+        class << self
+
+          def simple_value(*names)
+            names.each do |m|
+              define_method(m) do |value|
+                ensure_uniq(m)
+                @node.set m, value
+              end
+            end
+          end
+
+          def multi_value(*names)
+            names.each do |m|
+              define_method(m) do |value|
+                value = value.map(&:to_s) if value.is_a?(Array)
+                @node.set_multi m, value
+              end
+            end
+          end
+
+          def hash_value(*names)
+            names.each do |m|
+              define_method(m) do |data = {}, &block|
+                ensure_uniq(m)
+                ensure_hash(m, data)
+                @node.set m, Node.new(@node, data || {}, &block)
+              end
+            end
+          end
+
+          def mixed_value(*names)
+            names.each do |m|
+              define_method(m) do |data = {}, &block|
+                ensure_uniq(m)
+                if data.is_a?(Hash) || block
+                  ensure_hash(m, data) if block
+                  @node.set m, Node.new(@node, data, &block)
+                else
+                  @node.set m, data
+                end
+              end
+            end
+          end
+
+          def collection(*names)
+            names.each do |m|
+              define_method(m) do |id, data = {}, &block|
+                raise "bad collection id: #{id.inspect}" unless id
+
+                ensure_hash(m, data)
+
+                name = "#{m}s"
+                collection = @node.get(name) || @node.set(name, Node.new(@node, {}))
+                collection.set id, Node.new(collection, data, &block)
+              end
+            end
+          end
+
+        end
+
+        simple_value :verbose, :dry_run, :local_only, :path, :command,
+                     :options, :user, :host, :port, :password, :key, :secret, :bucket,
+                     :api_key, :container, :socket, :service_net, :repo_path
+        multi_value :skip_tables, :exclude, :files
+        hash_value :mysqldump, :tar, :gpg, :keep, :pgdump, :tar, :svndump,
+                   :sftp, :ftp, :mongodump
+        mixed_value :s3, :local, :cloudfiles
+        collection :database, :archive, :repo
+
+        private
+
+        def ensure_uniq(m)
+          raise(ArgumentError, "duplicate value for '#{m}'") if @node.get(m)
+        end
+
+        def ensure_hash(k, v)
+          raise "#{k}: hash expected: #{v.inspect}" unless v.is_a?(Hash)
+        end
+
+      end
+
+    end
+
+  end
+
+end