ray-db2fog 0.7.1

data/HISTORY

@@ -0,0 +1,36 @@
+v0.6.0 (24th October 2011)
+  - Support for postgresql 8
+v0.5.4 (17th October 2011)
+  - only clean files that were created by db2fog
+v0.5.3 (12th October 2011)
+  - improve robustness of clean task
+v0.5.2 (15th September 2011)
+  - fix restoring of postgresql databases (thanks Dan Neighman)
+v0.5.1 (29th August 2011)
+  - relax fog dependency to allow higher versions
+v0.5.0 (6th August 2011)
+  - initial postgresql support
+  - removed the statistics rake task
+  - fix an encoding issue under 1.9
+v0.4.2 (24th July 2011)
+  - fixed recording of most recent backup
+  - remove runtime dependency on mysql2
+v0.4.1 (23rd July 2011)
+  - fixed cleaning of old backups
+v0.4.0 (22nd July 2011)
+  - forked db2s3 into db2fog
+0.3.1 (7 Dec 2009)
+  - Fixed hardcoded DB name in statistics task
+0.3.0 (6 Dec 2009)
+  - Added db2s3:backup:clean task to delete old backups
+  - Added dependency on activesupport for the clean task
+0.2.6 (6 Dec 2009)
+  - Remove metrics task, since it was far too inaccurate
+  - Add statistics task to show you the size of your tables
+  - Only add username to mysql command line if provided in database.yml
+0.2.5
+  - Use host provided in database.yml
+0.2.4 (2 Sep 2009)
+  - Fix credentials bug
+0.2.3
+  - Keep old backups around

data/README.rdoc

@@ -0,0 +1,113 @@
+= DB2Fog
+
+A rails plugin to backup Mysql or PostgreSQL to a cloud storage provider. You're looking at
+a monthly spend of four cents. So pony up you cheap bastard, and store your
+backups offsite.
+
+A grandfather style system is used to decide what backups to keep copies of:
+
+* all backups from the past 24 hours
+* one backup per day for the past week
+* one backup per week forever
+
+Depending on your tolerance for data loss you should be running a backup at
+least once a day, probably more.
+
+== Installation
+
+Add the following to your project Gemfile
+
+  gem "db2fog"
+
+== Configuration
+
+Add the following to config/initializers/db2fog.rb
+
+In general, you can use any configuration options supported by
+Fog::Storage.new, plus the :directory option. If fog adds support for extra
+providers they should work with just a config change to Db2Fog.
+
+=== Amazon S3
+
+  DB2Fog.config = {
+    :aws_access_key_id     => 'yourkey',
+    :aws_secret_access_key => 'yoursecretkey',
+    :directory             => 'bucket-name',
+    :provider              => 'AWS'
+  }
+
+=== Rackspace Cloudfiles
+
+  DB2Fog.config = {
+    :rackspace_username => 'username',
+    :rackspace_api_key  => 'api key',
+    :directory          => 'bucket-name',
+    :provider           => 'Rackspace'
+  }
+
+=== Local Storage
+
+  DB2Fog.config = {
+    :directory  => 'bucket-name',
+    :local_root => Rails.root.to_s + '/db/backups',
+    :provider   => 'Local'
+  }
+
+== Passing options to the database driver
+
+Database adaptors may support further configuration via the :database_options hash.
+
+  DB2Fog.config = {
+    :directory  => 'bucket-name',
+    :local_root => Rails.root.to_s + '/db/backups',
+    :provider   => 'Local',
+    :database_options => {
+      :pg_version => 8
+    }
+  }
+
+Supported database options:
+
+pg_version    8 or 9
+  The major version of PostgreSQL installed on the server. Defaults to 9.
+
+== Usage
+
+  # Add to your crontab or whatever
+  rake db2fog:full
+
+  # Handy tasks
+  rake db2fog:restore  # You should be testing this regularly
+  rake db2fog:clean    # Clean up old backups - cron this
+
+=== Alternative
+
+If you want to trigger backups from ruby (say, from a delayed job) you can
+do this:
+
+    DB2Fog.new.backup
+    DB2Fog.new.clean
+
+== Compatibility
+
+This is pure so ruby should run on most ruby VMs. I develop on MRI 1.9.2.
+
+This will only work with rails 3. Supporting earlier versions is more
+complication than I feel like handling at the moment.
+
+== Development
+
+Specs are a little weak and mysql based. This code is bit hackish but is being
+used by quite a few people.
+
+== Kudos
+
+This is a fork of Xavier Shay's db2s3 gem. It worked perfectly, but only
+supported Amazon S3 within US-east. By switching the dependency to using fog
+this now supports all S3 regions and multiple storage providers
+
+Xavier's original gem is available at https://github.com/xaviershay/db2s3
+
+Xavier quotes the following example as inspiration:
+
+http://github.com/pauldowman/blog_code_examples/tree/master/mysql_s3_backup

data/lib/db2fog.rb

@@ -0,0 +1,227 @@
+# coding: utf-8
+
+require 'active_support'
+require 'active_support/core_ext/class/attribute_accessors'
+require 'active_support/core_ext/hash/except'
+require 'fog'
+require 'tempfile'
+require 'db2fog/railtie'
+
+class DB2Fog
+  cattr_accessor :config
+
+  def backup
+    file_name = "dump-#{db_credentials[:database]}-#{Time.now.utc.strftime("%Y%m%d%H%M")}.sql.gz"
+    store.store(file_name, open(database.dump))
+    store.store(most_recent_dump_file_name, file_name)
+  end
+
+  def restore
+    dump_file_name = store.fetch(most_recent_dump_file_name).read
+    file = store.fetch(dump_file_name)
+    database.restore(file.path)
+  end
+
+  def clean
+    to_keep = []
+    # only consider files that belong to db2fog. Other files are ignored
+    filelist = store.list.select {|file|
+      file.include?(db_credentials[:database]) && file.match(/\d{12}.sql.gz\Z/)
+    }
+    files = filelist.map { |file|
+      {
+        :path => file,
+        :date => Time.parse(file.split('-').last.split('.').first)
+      }
+    }
+    # Keep all backups from the past day
+    files.select {|x| x[:date] >= 1.day.ago }.each do |backup_for_day|
+      to_keep << backup_for_day
+    end
+
+    # Keep one backup per day from the last week
+    files.select {|x| x[:date] >= 1.week.ago }.group_by {|x| x[:date].strftime("%Y%m%d") }.values.each do |backups_for_last_week|
+      to_keep << backups_for_last_week.sort_by{|x| x[:date].strftime("%Y%m%d") }.first
+    end
+
+    # Keep one backup per week since forever
+    files.group_by {|x| x[:date].strftime("%Y%W") }.values.each do |backups_for_week|
+      to_keep << backups_for_week.sort_by{|x| x[:date].strftime("%Y%m%d") }.first
+    end
+
+    to_destroy = filelist - to_keep.uniq.collect {|x| x[:path] }
+    to_destroy.each do |file|
+      store.delete(file.split('/').last)
+    end
+  end
+
+  private
+
+  def store
+    @store ||= FogStore.new
+  end
+
+  def most_recent_dump_file_name
+    "most-recent-dump-#{db_credentials[:database]}.txt"
+  end
+
+  def db_credentials
+    ActiveRecord::Base.connection.instance_eval { @config } # Dodgy!
+  end
+
+  def database
+    @database ||= case db_credentials[:adapter]
+                  when /mysql/    then MysqlAdaptor.new(db_credentials)
+                  when /postgres/ then PsqlAdaptor.new(db_credentials)
+                  else
+                    raise "database adaptor '#{db_credentials[:adapter]}' not supported"
+                  end
+  end
+
+  class MysqlAdaptor
+
+    def initialize(credentials)
+      @credentials = credentials
+    end
+
+    def dump
+      dump_file = Tempfile.new("dump")
+
+      cmd = "mysqldump --quick --single-transaction --create-options #{mysql_options}"
+      cmd += " | gzip -9 > #{dump_file.path}"
+      run(cmd)
+
+      dump_file.path
+    end
+
+    def restore(path)
+      run "gunzip -c #{path} | mysql #{mysql_options}"
+    end
+
+    private
+
+    def mysql_options
+      cmd = ''
+      cmd += " -u #{@credentials[:username]} " unless @credentials[:username].nil?
+      cmd += " -p'#{@credentials[:password]}'" unless @credentials[:password].nil?
+      cmd += " -h '#{@credentials[:host]}'"    unless @credentials[:host].nil?
+      cmd += " --default-character-set=#{@credentials[:encoding]}" unless @credentials[:encoding].nil?
+      cmd += " #{@credentials[:database]}"
+    end
+
+    def run(command)
+      result = system(command)
+      raise("error, process exited with status #{$?.exitstatus}") unless result
+    end
+
+  end
+
+  class PsqlAdaptor
+
+    def initialize(credentials)
+      @credentials = credentials
+    end
+
+    def dump
+      dump_file = Tempfile.new("dump")
+
+      cmd = "pg_dump --clean --format=p #{pg_dump_options}"
+      cmd += " | gzip -9 > #{dump_file.path}"
+      run(cmd)
+
+      dump_file.path
+    end
+
+    def restore(path)
+      run "gunzip -c #{path} | psql #{psql_options}"
+    end
+
+    private
+
+    def pg_dump_options
+      cmd = ''
+      cmd += " -U #{@credentials[:username]} " unless @credentials[:username].nil?
+      cmd += " -h '#{@credentials[:host]}'"    unless @credentials[:host].nil?
+      cmd += " -w"                             if     pg_version >= 9
+      cmd += " #{@credentials[:database]}"
+    end
+
+    def psql_options
+      cmd = ''
+      cmd += " -U #{@credentials[:username]} " unless @credentials[:username].nil?
+      cmd += " -h '#{@credentials[:host]}'"    unless @credentials[:host].nil?
+      cmd += " -w"                             if     pg_version >= 9
+      cmd += " -d #{@credentials[:database]}"
+    end
+
+    def pg_version
+      opts = database_options || {}
+      opts[:pg_version] || 9
+    end
+
+    def run(command)
+      result = system(command)
+      raise("error, process exited with status #{$?.exitstatus}") unless result
+    end
+
+    def database_options
+      if DB2Fog.config.respond_to?(:[])
+        DB2Fog.config[:database_options]
+      else
+        raise "DB2Fog not configured"
+      end
+    end
+
+  end
+
+  class FogStore
+
+    def store(remote_filename, io)
+      directory.files.create(:key => remote_filename, :body => io, :public => false)
+    end
+
+    def fetch(remote_filename)
+      remote_file = directory.files.get(remote_filename)
+
+      file = Tempfile.new("dump")
+      open(file.path, 'wb') { |f| f.write(remote_file.body) }
+      file
+    end
+
+    def list
+      directory.files.map { |f| f.key }
+    end
+
+    def delete(remote_filename)
+      remote_file = directory.files.head(remote_filename)
+      remote_file.destroy if remote_file
+    end
+
+    private
+
+    def fog_options
+      if DB2Fog.config.respond_to?(:[])
+        DB2Fog.config.except(:directory, :database_options)
+      else
+        raise "DB2Fog not configured"
+      end
+    end
+
+    def directory_name
+      if DB2Fog.config.respond_to?(:[])
+        DB2Fog.config[:directory]
+      else
+        raise "DB2Fog not configured"
+      end
+    end
+
+    def directory
+      @directory ||= storage.directories.get(directory_name)
+    end
+
+    def storage
+      @storage = Fog::Storage.new(fog_options)
+    end
+  end
+
+end

data/lib/db2fog/railtie.rb

@@ -0,0 +1,11 @@
+require 'rails'
+require 'db2fog'
+
+class DB2Fog
+  class Railtie < Rails::Railtie
+
+    rake_tasks do
+      load File.expand_path('tasks.rb', File.dirname(__FILE__))
+    end
+  end
+end

data/lib/db2fog/tasks.rb

@@ -0,0 +1,35 @@
+# coding: utf-8
+
+namespace :db2fog do
+  desc "Save a full back to S3"
+  task :backup => :environment do
+    DB2Fog.new.backup
+  end
+
+  desc "Restore your DB from S3"
+  task :restore => :environment do
+    DB2Fog.new.restore
+  end
+
+  desc "Keep all backups for the last day, one per day for the last week, and one per week before that. Delete the rest."
+  task :clean => :environment do
+    DB2Fog.new.clean
+  end
+
+  namespace :backup do
+    task :full => :environment do
+      $stderr.puts "the db2fog:backup:full rake task is deprecated, use db2fog:backup instead"
+      DB2Fog.new.backup
+    end
+
+    task :restore => :environment do
+      $stderr.puts "the db2fog:backup:restore rake task is deprecated, use db2fog:restore instead"
+      DB2Fog.new.restore
+    end
+
+    task :clean => :environment do
+      $stderr.puts "the db2fog:backup:clean rake task is deprecated, use db2fog:clean instead"
+      DB2Fog.new.clean
+    end
+  end
+end

metadata

@@ -0,0 +1,179 @@
+--- !ruby/object:Gem::Specification 
+name: ray-db2fog
+version: !ruby/object:Gem::Version 
+  hash: 1
+  prerelease: 
+  segments: 
+  - 0
+  - 7
+  - 1
+  version: 0.7.1
+platform: ruby
+authors: 
+- James Healy, Sasan Padidar
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2012-01-04 00:00:00 Z
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: rails
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 7
+        segments: 
+        - 3
+        - 0
+        version: "3.0"
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: activerecord
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 7
+        segments: 
+        - 3
+        - 0
+        version: "3.0"
+  type: :runtime
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: fog
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 15
+        segments: 
+        - 1
+        - 0
+        version: "1.0"
+  type: :runtime
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: rake
+  prerelease: false
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id004
+- !ruby/object:Gem::Dependency 
+  name: mysql2
+  prerelease: false
+  requirement: &id005 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id005
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  prerelease: false
+  requirement: &id006 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 15
+        segments: 
+        - 2
+        - 6
+        version: "2.6"
+  type: :development
+  version_requirements: *id006
+- !ruby/object:Gem::Dependency 
+  name: timecop
+  prerelease: false
+  requirement: &id007 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 25
+        segments: 
+        - 0
+        - 3
+        - 5
+        version: 0.3.5
+  type: :development
+  version_requirements: *id007
+description: ray-db2fog was forked from https://github.com/yob/db2fog.git.
+email: 
+- spadidar@raybeam.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- lib/db2fog.rb
+- lib/db2fog/tasks.rb
+- lib/db2fog/railtie.rb
+- README.rdoc
+- HISTORY
+homepage: http://github.com/spadidar/db2fog
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --title
+- DB2Fog
+- --line-numbers
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 57
+      segments: 
+      - 1
+      - 8
+      - 7
+      version: 1.8.7
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 31
+      segments: 
+      - 1
+      - 3
+      - 2
+      version: 1.3.2
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.8.10
+signing_key: 
+specification_version: 3
+summary: db2fog provides rake tasks for backing up and restoring your DB to cloud storage providers
+test_files: []
+