git_friendly_dumper 0.0.1

data/History.txt

@@ -0,0 +1,10 @@
+== 0.0.1
+
+  * gem release
+  * Updated for latest rspec, cucumber, and ActiveRecord (3.x)
+
+== Prior to gem release
+
+* Improvements made while using for http://www.distinctivedoors.co.uk, (Ian White & Nick Rutherford)
+
+* Initial release

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008-2012 Ian White - ian.w.white@gmail.com
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,48 @@
+= git friendly dumper
+
+Use fixtures to create a db independent and git friendly db dump.
+
+It's git friendly because each record is in its own file (git doesn't do 
+very well with large files that are committed all at once).
+
+If using the schema option (ie. db:dump & db:load), then each table's schema
+definition is included as is the schema_migrations table.
+
+== Example usage:
+
+  # drop-in replacement for mysqldump
+  
+  rake db:dump DUMP_PATH=some/path
+  # just like mysqldump -u <user> -p<pass> <app>_development > some/path.sql
+
+  rake db:load DUMP_PATH=some/path
+  # just like script/dbconsole -p < some/path.sql
+    
+  # without schema information, just the data
+  
+  rake db:data:dump DUMP_PATH=some/path # dump the data without touching the structure
+  rake db:data:load DUMP_PATH=some/path # load the data without touching the structure
+    
+  # just load particular table data
+  rake db:data:load DUMP_PATH=some/path TABLES=users,things
+  
+See `lib/tasks/git_friendly_dumper_tasks.rake` and the features for more info
+
+== Notes
+
+It's probably a bad idea to check your dump into the same repo as your app.
+
+I created this so that we could have incremental scheduled backups of the db.  With a database of 120M 
+containing many binary columns, this resulted in a massive saving in bandwidth.
+
+Basically, we have the db backup inside the app, but ignored by the main app, the db/dump
+has it's own git repo.
+
+== Run the specs and features
+
+Grab the last known good set of deps and run the specs and features
+
+  cp Gemfile.lock.development Gemfile.lock
+  bundle
+  rake
+    

data/Rakefile

@@ -0,0 +1,43 @@
+#!/usr/bin/env rake
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+begin
+  require 'rdoc/task'
+rescue LoadError
+  require 'rdoc/rdoc'
+  require 'rake/rdoctask'
+  RDoc::Task = Rake::RDocTask
+end
+
+RDoc::Task.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'GitFriendlyDumper'
+  rdoc.options << '--line-numbers'
+  rdoc.rdoc_files.include('README.rdoc')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core/rake_task'
+require "cucumber/rake/task"
+
+task :default => [:spec, :features]
+
+desc "Set up tmp stuff for running specs etc"
+file "tmp/db" do
+  mkdir_p 'tmp/db'
+end
+
+desc "Run the specs"
+RSpec::Core::RakeTask.new(:spec => ['tmp/db']) do |t|
+  t.pattern = "./spec/**/*_spec.rb"
+end
+
+desc "Run the features"
+Cucumber::Rake::Task.new(:features) do |task|
+  task.cucumber_opts = ["features"]
+end

data/features/dump/change_dump_path.feature

@@ -0,0 +1,26 @@
+Feature: Change dump path
+  In order to dump somewhere else than db/dump
+  I want the rake task to use the DUMP_PATH env variable to override the data path used
+
+  Background:
+    Given I am in an empty app
+    Given a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And the database has a "users" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+      | Jane          | Heidie           |
+    And the database has a "schema_migrations" table:
+      | version (string) | 
+      | 01001010123      |
+
+
+
+  Scenario: change dump path
+    When I successfully run `rake db:dump FORCE=1 DUMP_PATH=db/override`
+    Then the output should contain "Dumping data and structure from database to db/override"
+    And the following directories should exist:
+      | db/override/users |
+    And the following files should exist:
+      | db/override/users/schema.rb |

data/features/dump/data_dump.feature

@@ -0,0 +1,27 @@
+Feature: Data dump
+  In order to … 
+  I want to dump the data fixtures but not the schema or migrations table
+
+  Background:
+    Given I am in an empty app
+    Given a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And the database has a "users" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+      | Jane          | Heidie           |
+    And the database has a "schema_migrations" table:
+      | version (string) | 
+      | 01001010123      |
+
+
+  Scenario: rake db:data:dump dumps all tables' contents but does not dump the schema or migrations table
+    When I successfully run `rake db:data:dump FORCE=1`
+    Then the output should contain "Dumping data from database to db/dump"
+    And the following directories should exist:
+      | db/dump/users             |
+    But the following directories should not exist:
+      | db/dump/schema_migrations |
+    And the following files should not exist:
+      | db/dump/users/schema.rb |

data/features/dump/default_dump.feature

@@ -0,0 +1,39 @@
+Feature: Default dump
+  In order to produce incremental backups of my database using git
+  I want to backup my database in a form git can play nicely with
+
+  Background:
+    Given I am in an empty app
+    Given a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And the database has a "users" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+      | Jane          | Heidie           |
+    And the database has a "schema_migrations" table:
+      | version (string) | 
+      | 01001010123      |
+
+
+
+  Scenario: rake db:dump dumps all tables' contents and the schema
+    When I successfully run `rake db:dump FORCE=1`
+    Then the output should contain "Dumping data and structure from database to db/dump"
+    And the following directories should exist:
+      | db/dump/users             |
+      | db/dump/schema_migrations |
+    And the following files should exist:
+      | db/dump/users/schema.rb |
+      
+    And the data in the dumped "users" yaml files should match the database contents
+
+    Given an empty database
+    When I execute the schema "db/dump/users/schema.rb"
+    Then a "users" table should exist with structure:
+      | name       | type         |
+      | id         | INTEGER      |
+      | created_at | datetime     |
+      | updated_at | datetime     |
+      | name       | varchar(255) |
+      | surname    | varchar(255) |

data/features/dump/dump_and_raise_error.feature

@@ -0,0 +1,58 @@
+Feature: Toggle halting dump when runtime errors occur
+
+  Background:
+    Given I am in an empty app
+    Given a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And the database has a "users" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+      | Jane          | Heidie           |
+    And the database has a "schema_migrations" table:
+      | version (string) | 
+      | 01001010123      |
+
+
+
+  Scenario: by default runtime errors are raised
+    When I run `rake db:dump FORCE=1 TABLES=doesntexist`
+    Then the exit status should be 1
+    And the output should contain "dumping doesntexist failed: SQLite3::SQLException: no such table: doesntexist: SELECT COUNT(*) FROM doesntexist"
+
+
+
+  Scenario Outline: RAISE_ERROR= toggles silence or raise runtime errors
+    When I run `rake db:dump FORCE=1 TABLES=doesntexist RAISE_ERROR=<RAISE_ERROR>`
+    Then the exit status should be <EXIT_STATUS>
+    But the output should contain "dumping doesntexist failed: SQLite3::SQLException: no such table: doesntexist: SELECT COUNT(*) FROM doesntexist"
+
+    Scenarios: don't raise
+    | RAISE_ERROR | EXIT_STATUS |
+    | false       | 0           |
+    | 0           | 0           |
+
+    Scenarios: raise
+    | RAISE_ERROR | EXIT_STATUS |
+    | true        | 1           |
+    | 1           | 1           |
+
+
+
+  Scenario: dirty directory warning on failed dump
+    When I run `rake db:dump FORCE=1 TABLES=doesntexist`
+    Then the exit status should be 1
+    And the following directories should exist:
+      | db/dump/doesntexist     |
+    And the output should contain "Partial dump files have been left behind and you should clean up before continuing (e.g. git status, git checkout, git clean)."
+
+
+
+  Scenario: invalid FIXTURES argument raises an error
+    When I run `rake db:dump FORCE=1 FIXTURES=a_fixture.yml`
+    Then the exit status should be 1
+    And the output should contain "GitFriendlyDumper if :fixtures option given, neither :include_schema nor :clobber_fixtures can be given"
+
+    When I run `rake db:data:dump FORCE=1 FIXTURES=a_fixture.yml`
+    Then the exit status should be 1
+    And the output should contain "Cannot dump when :fixtures option is given"

data/features/dump/dump_deleted_records_and_clobber.feature

@@ -0,0 +1,98 @@
+Feature: What happens to fixtures belonging to deleted records when I perform a new dump
+  When dumping a table, any existing fixtures for that table are first removed
+  Any tables not being dumped, however, are ignored, and the old fixtures will remain in place
+
+  In order to have fixtures for what's in my database and nothing else
+  That is, no leftovers (deleted records, dropped tables, etc) from previous dumps
+  I want to CLOBBER them all before dumping
+
+  Background:
+    Given I am in an empty app
+    Given a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And the database has a "users" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+      | Jane          | Heidie           |
+    And the database has a "schema_migrations" table:
+      | version (string) | 
+      | 01001010123      |
+
+
+
+  Scenario: fixtures belonging to deleted records are deleted and not recreated
+    When I run `rake db:dump FORCE=1`
+    Then the following files should exist:
+      | db/dump/users/0000/0001.yml |
+      | db/dump/users/0000/0002.yml |
+      | db/dump/users/0000/0003.yml |
+
+    When I destroy record 1 from the "users" table
+    And I run `rake db:dump FORCE=1`
+    Then the following files should exist:
+      | db/dump/users/0000/0002.yml |
+      | db/dump/users/0000/0003.yml |
+    But the following files should not exist:
+      | db/dump/users/0000/0001.yml |
+
+
+
+  Scenario: when dumping specific TABLES= any other tables' fixtures are left as they were (even if stale or deleted)
+    Given the database has a "debts" table (with timestamps):
+     | name (string) | surname (string) |
+     | Fred          | Bloggs           |
+     | Ethel         | Smith            |
+
+    When I run `rake db:dump FORCE=1`
+    Then the following files should exist:
+      | db/dump/users/0000/0001.yml |
+      | db/dump/users/0000/0002.yml |
+      | db/dump/users/0000/0003.yml |
+      | db/dump/debts/0000/0001.yml |
+      | db/dump/debts/0000/0002.yml |
+
+    When I destroy record 1 from the "debts" table
+    And I run `rake db:dump FORCE=1 TABLES=users`
+    Then the following files should exist:
+      | db/dump/users/0000/0001.yml |
+      | db/dump/users/0000/0002.yml |
+      | db/dump/users/0000/0003.yml |
+      | db/dump/debts/0000/0001.yml |
+      | db/dump/debts/0000/0002.yml |
+
+
+
+  Scenario Outline: CLOBBER= removes all existing fixtures before dumping new ones, combining with TABLES= means only those tables' fixtures will exist
+    Given the database has a "debts" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+    When I run `rake db:dump FORCE=1`
+    Then the following files should exist:
+      | db/dump/users/0000/0001.yml |
+      | db/dump/users/0000/0002.yml |
+      | db/dump/users/0000/0003.yml |
+      | db/dump/debts/0000/0001.yml |
+      | db/dump/debts/0000/0002.yml |
+
+    When I destroy record 1 from the "users" table
+    And I run `rake db:dump FORCE=1 TABLES=users CLOBBER=<CLOBBER>`
+    Then the following files should exist:
+      | db/dump/users/0000/0002.yml |
+      | db/dump/users/0000/0003.yml |
+    But the following files should not exist:
+      | db/dump/users/0000/0001.yml |
+    And the following files <EXISTENCE>:
+      | db/dump/debts/0000/0001.yml |
+      | db/dump/debts/0000/0002.yml |
+
+    Scenarios: deleting existing dump fixtures with CLOBBER=true|1
+      | CLOBBER | EXISTENCE         |
+      | true    | should not exist  |
+      | 1       | should not exist  |
+
+    Scenarios: not deleting existing dump fixtures with CLOBBER=false|0
+      | CLOBBER | EXISTENCE    |
+      | false   | should exist |
+      | 0       | should exist |

data/features/dump/dump_specific_tables.feature

@@ -0,0 +1,39 @@
+Feature: Dump specific tables
+  In order to only backup certain tables
+  I want TABLES to list the tables the rake task will dump
+
+  Background:
+    Given I am in an empty app
+    Given a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And the database has a "users" table (with timestamps):
+      | name (string) | surname (string) |
+      | Fred          | Bloggs           |
+      | Ethel         | Smith            |
+      | Jane          | Heidie           |
+    And the database has a "schema_migrations" table:
+      | version (string) | 
+      | 01001010123      |
+
+
+
+  Scenario: dump specific tables
+    Given the database has a "seen" table (with timestamps):
+      | bird (string) | seen (integer) |
+      | Parrot        | 1              |
+      | Robin         | 3              |
+      | Goldfinch     | 6              |
+    Given the database has a "notes" table:
+      | content (text)          |
+      | Fred spotted a parrot?? |
+
+    When I successfully run `rake db:dump FORCE=1 TABLES=seen,users`
+    Then the output should contain "Dumping data and structure from database to db/dump"
+    And the following directories should exist:
+      | db/dump/users |
+      | db/dump/seen |
+    And the following files should exist:
+      | db/dump/users/schema.rb |
+      | db/dump/seen/schema.rb  |
+    But the following directories should not exist:
+      | db/dump/notes |

data/features/load/clobber_load.feature

@@ -0,0 +1,49 @@
+Feature: CLOBBER is ignored when loading
+  CLOBBER=false|0        clobber fixtures on dump  (default true)
+  Is ignored by db:load
+
+  Scenario: setting CLOBBER on load does nothing (fixtures are not deleted afterwards)
+    Given I am in an empty app
+    And a Rakefile exists which has an environment task and loads git_friendly_dumper tasks
+    And an empty database
+    And a file named "db/dump/users/schema.rb" with:
+    """
+    create_table "users", :force => true do |t|
+      t.datetime "created_at"
+      t.datetime "updated_at"
+      t.string   "name"
+      t.string   "surname"
+    end
+    
+    """
+    And a file named "db/dump/users/0000/0001.yml" with:
+    """
+    --- 
+    name: Jane
+    created_at: 2003-07-26 12:38:10
+    updated_at: 2007-07-26 12:48:10
+    id: 1
+    surname: Heidie
+    
+    """
+    And a file named "db/dump/users/0000/0002.yml" with:
+    """
+    --- 
+    name: Ethel
+    created_at: 2003-07-26 10:38:10
+    updated_at: 2007-07-26 10:48:10
+    id: 2
+    surname: Smith
+    """
+
+    When I successfully run `rake db:load FORCE=1 CLOBBER=true`
+    And I refresh the database tables cache
+    And the "users" table should match exactly:
+    | id    | name  | surname | created_at          | updated_at          |
+    | 1     | Jane  | Heidie  | 2003-07-26 12:38:10 | 2007-07-26 12:48:10 |
+    | 2     | Ethel | Smith   | 2003-07-26 10:38:10 | 2007-07-26 10:48:10 |
+
+    But the following files should exist:
+      | db/dump/users/schema.rb     |
+      | db/dump/users/0000/0001.yml |
+      | db/dump/users/0000/0002.yml |