replicate 1.0 → 1.1

data/HACKING

@@ -0,0 +1,61 @@
+Grab a local clone of rtomayko/replicate:
+
+    git clone git://github.com/rtomayko/replicate.git
+    cd replicate
+
+The default rake task installs the latest supported activerecord
+version to a project local GEM_HOME=./vendor and runs the unit
+tests:
+
+    $ rake
+    installing activerecord-3.1.0 to ./vendor/1.8
+    installing sqlite3 to ./vendor/1.8
+    Using activerecord 3.1.0
+    Loaded suite ...
+    Started
+    ....................
+    Finished in 0.150186 seconds.
+
+    20 tests, 106 assertions, 0 failures, 0 errors
+
+Use `rake test:all' to run tests under all activerecord versions:
+
+    $ rake test:all
+    installing activerecord ~> 2.2.3 to ./vendor
+    installing activerecord ~> 2.3.14 to ./vendor
+    installing activerecord ~> 3.0.10 to ./vendor
+    installing activerecord ~> 3.1.0 to ./vendor
+    ==> testing activerecord ~> 2.2.3
+    Started
+    ....................
+    Finished in 0.119517 seconds.
+
+    20 tests, 106 assertions, 0 failures, 0 errors
+    ==> testing activerecord ~> 2.3.14
+    Started
+    ....................
+    Finished in 0.119517 seconds.
+
+    20 tests, 106 assertions, 0 failures, 0 errors
+    <snip>
+
+rake test:all should always be passing under latest stable MRI
+1.8.7 and MRI 1.9.x.
+
+Running individual test files directly requires setting the
+GEM_HOME environment variable and ensuring ./lib is on the load
+path:
+
+    export GEM_HOME=vendor/1.9 # or 1.8.7
+    ruby -Ilib test/active_record_test.rb
+
+You can also control which activerecord version is used in the
+test with the AR_VERSION environment variable:
+
+    rake setup:all
+    export GEM_HOME=vendor/1.8.7
+    AR_VERSION=3.1.0 ruby -rubygems -Ilib test/active_record_test.rb
+
+If you have something worth sharing, please send a pull request:
+
+    https://github.com/rtomayko/replicate

data/README.md

@@ -1,15 +1,20 @@
-# Replicate
-
 Dump and load relational objects between Ruby environments.
+===========================================================
+
+The project started at GitHub to simplify the process of getting real production
+data into development and staging environments. We use it to replicate entire
+repository data (including associated issue, pull request, commit comment, etc.
+records) from production to our development environments with a single command.
+It's excessively useful for troubleshooting issues, support requests, and
+exception reports as well as for establishing real data for evaluating design
+concepts.
 
-The project was started at GitHub to ease the process of getting real production
-data into staging and development environments. We have a custom command that
-uses the replicate machinery to dump entire repository data (including
-associated objects like issues, pull requests, commit comments, etc.) from
-production and load it into the current environment. This is excessively useful
-for troubleshooting issues, support requests, and exception reports.
+Synopsis
+--------
 
-## Synopsis
+Installing:
+
+    $ gem install replicate
 
 Dumping objects:
 
@@ -32,16 +37,34 @@ Dumping and loading over SSH:
     $ remote_command="replicate -r /app/config/environment -d 'User.find(1234)'"
     $ ssh example.org "$remote_command" |replicate -r config/environment -l
 
-## ActiveRecord
+ActiveRecord
+------------
+
+Basic support for dumping and loading ActiveRecord objects is included. The
+tests pass under ActiveRecord versions 2.2.3, 2.3.14, 3.0.10, and 3.1.0 under
+MRI 1.8.7 as well as under MRI 1.9.2.
+
+To use customization macros in your models, require the replicate library after
+ActiveRecord (in e.g., `config/initializers/libraries.rb`):
+
+    require 'active_record'
+    require 'replicate'
+
+ActiveRecord support works sensibly without customization so this isn't strictly
+necessary to use the `replicate` command. The following sections document the
+available customization macros.
 
-*NOTE: Replicate has been tested only under ActiveRecord 2.2. Support for
-ActiveRecord 3.x is planned.*
+### Association Dumping
 
-Basic support for dumping and loading ActiveRecord objects is included. When an
-object is dumped, all `belongs_to` and `has_one` associations are automatically
-followed and included in the dump. You can mark `has_many` and
-`has_and_belongs_to_many` associations for automatic inclusion using the
-`replicate_associations` macro:
+The baked in support adds some more or less sensible default behavior for all
+subclasses of `ActiveRecord::Base` such that dumping an object will bring in
+objects related via `belongs_to` and `has_one` associations.
+
+Unlike 1:1 associations, `has_many` and `has_and_belongs_to_many` associations
+are not automatically included. Doing so would quickly lead to the entire
+database being sucked in. It can be useful to mark specific associations for
+automatic inclusion using the `replicate_associations` macro. For instance,
+to always include `EmailAddress` records belonging to a `User`:
 
     class User < ActiveRecord::Base
       belongs_to :profile
@@ -50,10 +73,13 @@ followed and included in the dump. You can mark `has_many` and
       replicate_associations :email_addresses
     end
 
-By default, the loader attempts to create a new record for all objects. This can
-lead to unique constraint errors when a record already exists with matching
-attributes. To update existing records instead of always creating new ones,
-define a natural key for the model using the `replicate_natural_key` macro:
+### Natural Keys
+
+By default, the loader attempts to create a new record with a new primary key id
+for all objects. This can lead to unique constraint errors when a record already
+exists with matching attributes. To update existing records instead of
+creating new ones, define a natural key for the model using the `replicate_natural_key`
+macro:
 
     class User < ActiveRecord::Base
       belongs_to :profile
@@ -63,13 +89,45 @@ define a natural key for the model using the `replicate_natural_key` macro:
       replicate_associations :email_addresses
     end
 
-Multiple attribute names may be specified to define a compound key.
+    class EmailAddress < ActiveRecord::Base
+      belongs_to :user
+      replicate_natural_key :user_id, :email
+    end
+
+Multiple attribute names may be specified to define a compound key. Foreign key
+column attributes (`user_id`) are often included in natural keys.
+
+### Validations and Callbacks
+
+__IMPORTANT:__ All ActiveRecord validations and callbacks are disabled on the
+loading side. While replicate piggybacks on AR for relationship information and
+uses `ActiveRecord::Base#save` to write objects to the database, it's designed
+to act as a simple dump / load tool.
+
+It's sometimes useful to run certain types of callbacks on replicate. For
+instance, you might want to create files on disk or load information into a
+separate data store any time an object enters the database. The best way to go
+about this currently is to override the model's `load_replicant` class method:
+
+    class User < ActiveRecord::Base
+      def self.load_replicant(type, id, attrs)
+        id, object = super
+        object.register_in_redis
+        object.some_other_callback
+        [id, object]
+      end
+    end
+
+This interface will be improved in future versions.
 
-## Custom Objects
+Custom Objects
+--------------
 
 Other object types may be included in the dump stream so long as they implement
 the `dump_replicant` and `load_replicant` methods.
 
+### dump_replicant
+
 The dump side calls `#dump_replicant(dumper)` on each object. The method must
 call `dumper.write()` with the class name, id, and hash of primitively typed
 attributes for the object:
@@ -84,6 +142,8 @@ attributes for the object:
       end
     end
 
+### load_replicant
+
 The load side calls `::load_replicant(type, id, attributes)` on the class to
 load each object into the current environment. The method must return an
 `[id, object]` tuple:
@@ -97,3 +157,28 @@ load each object into the current environment. The method must return an
         [user.id, user]
       end
     end
+
+How it works
+------------
+
+The dump format is designed for streaming relational data. Each object is
+encoded as a `[type, id, attributes]` tuple and marshalled directly onto the
+stream. The `type` (class name string) and `id` must form a distinct key when
+combined, `attributes` must consist of only string keys and simply typed values.
+
+Relationships between objects in the stream are managed as follows:
+
+ - An object's attributes may encode references to objects that precede it
+   in the stream using a simple tuple format: [:id, 'User', 1234].
+
+ - The dump side ensures that objects are written to the dump stream in
+   "reference order" such that when an object A includes a reference attribute
+   to an object B, B is guaranteed to arrive before A.
+
+ - The load side maintains a mapping of ids from the dumping system to the newly
+   replicated objects on the loading system. When the loader encounters a
+   reference value [:id, 'User', 1234] in an object's attributes, it converts it
+   to the load side id value.
+
+Dumping and loading happens in a streaming fashion. There is no limit on the
+number of objects included in the stream.

data/Rakefile

@@ -1,6 +1,71 @@
-task :default => :test
+require 'rbconfig'
+require 'rake/clean'
+task :default => [:setup, :test]
+
+vendor_dir   = './vendor'
+ruby_version = RbConfig::CONFIG['ruby_version']
+ENV['GEM_HOME'] = "#{vendor_dir}/#{ruby_version}"
+
+desc "Install gem dependencies for development"
+task :setup => 'setup:latest' do
+  verbose(false) { gem_install 'sqlite3' }
+end
 
 desc "Run tests"
 task :test do
+  ENV['RUBYOPT'] = [ENV['RUBYOPT'], 'rubygems'].compact.join(' ')
+  ENV['RUBYLIB'] = ['lib', ENV['RUBYLIB']].compact.join(':')
   sh "testrb test/*_test.rb", :verbose => false
 end
+CLEAN.include 'test/db'
+
+desc "Build gem"
+task :build do
+  sh "gem build replicate.gemspec"
+end
+
+# supported activerecord gem versions
+AR_VERSIONS = []
+AR_VERSIONS << '2.2.3' if RUBY_VERSION < '1.9'
+AR_VERSIONS.concat %w[2.3.14 3.0.10 3.1.0]
+
+desc "Run unit tests under all supported AR versions"
+task 'test:all' => 'setup:all' do
+  failures = []
+  AR_VERSIONS.each do |vers|
+    warn "==> testing activerecord ~> #{vers}"
+    ENV['AR_VERSION'] = vers
+    ok = system("rake -s test")
+    failures << vers if !ok
+    warn ''
+  end
+  fail "activerecord version failures: #{failures.join(', ')}" if failures.any?
+end
+
+# file tasks for installing each AR version
+desc 'Install gem dependencies for all supported AR versions'
+task 'setup:all' => 'setup'
+AR_VERSIONS.each do |vers|
+  version_file = "#{ENV['GEM_HOME']}/versions/activerecord-#{vers}"
+  file version_file do |f|
+    verbose(false) { gem_install 'activerecord', vers }
+  end
+  task "setup:#{vers}" => version_file
+  task "setup:all"     => "setup:#{vers}"
+end
+task "setup:latest" => "setup:#{AR_VERSIONS.last}"
+CLEAN.include 'vendor'
+
+# Install a gem to the local GEM_HOME but only if it isn't already installed
+def gem_install(name, version = nil)
+  version_name = [name, version].compact.join('-')
+  version_file = "#{ENV['GEM_HOME']}/versions/#{version_name}"
+  return if File.exist?(version_file)
+  warn "installing #{version_name} to #{ENV['GEM_HOME']}"
+  command = "gem install --no-rdoc --no-ri #{name}"
+  command += " -v '~> #{version}'" if version
+  command += " >/dev/null"
+  sh command
+  mkdir_p File.dirname(version_file)
+  File.open(version_file, 'wb') { }
+end

data/bin/replicate

@@ -10,6 +10,7 @@
 #/   -r, --require        Require the library. Often used with 'config/environment'.
 #/   -d, --dump           Dump the repository and all related objects to stdout.
 #/   -l, --load           Load dump file data from stdin.
+#/   -i, --keep-id        Use replicated ids when loading dump file.
 #/
 #/   -v, --verbose        Write more status output.
 #/   -q, --quiet          Write less status output.
@@ -20,6 +21,7 @@ require 'optparse'
 mode     = nil
 verbose  = false
 quiet    = false
+keep_id  = false
 out      = $stdout
 
 # parse arguments
@@ -31,6 +33,7 @@ ARGV.options do |opts|
   opts.on("-r", "--require=f") { |file| require file }
   opts.on("-v", "--verbose")   { verbose = true }
   opts.on("-q", "--quiet")     { quiet = true }
+  opts.on("-i", "--keep-id")   { keep_id = true }
   opts.on_tail("-h", "--help", &usage)
   opts.parse!
 end
@@ -38,8 +41,9 @@ end
 # load rails environment and replicator lib.
 require 'replicate'
 
-# hack to enable AR query cache
 if defined?(ActiveRecord::Base)
+  ActiveRecord::Base.replicate_id = keep_id
+  # hack to enable AR query cache
   ActiveRecord::ConnectionAdapters::QueryCache.
     send :attr_writer, :query_cache, :query_cache_enabled
   ActiveRecord::Base.connection.send(:query_cache=, {})

data/lib/replicate/active_record.rb

@@ -126,6 +126,18 @@ module Replicate
         @replicate_natural_key = attribute_names
       end
 
+      # Set or retrieve whether replicated object should keep its original id.
+      # When not set, replicated objects will be created with new id.
+      def replicate_id(boolean=nil)
+        self.replicate_id = boolean unless boolean.nil?
+        @replicate_id.nil? ? superclass.replicate_id : @replicate_id
+      end
+
+      # Set flag for replicating original id.
+      def replicate_id=(boolean)
+        @replicate_id = boolean
+      end
+
       # Load an individual record into the database. If the models defines a
       # replicate_natural_key then an existing record will be updated if found
       # instead of a new record being created.
@@ -156,17 +168,47 @@ module Replicate
 
       # Update an AR object's attributes and persist to the database without
       # running validations or callbacks.
+      #
+      # Returns the [id, object] tuple for the newly replicated objected.
       def create_or_update_replicant(instance, attributes)
-        def instance.callback(*args);end # Rails 2.x hack to disable callbacks.
-
+        # write replicated attributes to the instance
         attributes.each do |key, value|
-          next if key == primary_key
-          instance.write_attribute key, value
+          next if key == primary_key and not replicate_id
+          instance.send :write_attribute, key, value
+        end
+
+        # save the instance bypassing all callbacks and validations
+        replicate_disable_callbacks instance
+        if ::ActiveRecord::VERSION::MAJOR >= 3
+          instance.save :validate => false
+        else
+          instance.save false
         end
 
-        instance.save false
         [instance.id, instance]
       end
+
+      # Disable all callbacks on an ActiveRecord::Base instance. Only the
+      # instance is effected. There is no way to re-enable callbacks once
+      # they've been disabled on an object.
+      def replicate_disable_callbacks(instance)
+        if ::ActiveRecord::VERSION::MAJOR >= 3
+          # AR 3.1.x
+          def instance.run_callbacks(*args); yield; end
+
+          # AR 3.0.x
+          def instance._run_save_callbacks(*args); yield; end
+          def instance._run_create_callbacks(*args); yield; end
+          def instance._run_update_callbacks(*args); yield; end
+        else
+          # AR 2.x
+          def instance.callback(*args)
+          end
+          def instance.record_timestamps
+            false
+          end
+        end
+      end
     end
 
     # Special object used to dump the list of associated ids for a
@@ -213,5 +255,6 @@ module Replicate
     ::ActiveRecord::Base.send :extend,  ClassMethods
     ::ActiveRecord::Base.replicate_associations = []
     ::ActiveRecord::Base.replicate_natural_key  = []
+    ::ActiveRecord::Base.replicate_id           = false
   end
 end

data/test/active_record_test.rb

@@ -1,12 +1,22 @@
 require 'test/unit'
 require 'stringio'
+
+# require a specific AR version.
+version = ENV['AR_VERSION']
+gem 'activerecord', "~> #{version}" if version
 require 'active_record'
+require 'active_record/version'
+warn "Using activerecord #{ActiveRecord::VERSION::STRING}"
+
+# replicate must be loaded after AR
 require 'replicate'
 
+# create the sqlite db on disk
 dbfile = File.expand_path('../db', __FILE__)
 File.unlink dbfile if File.exist?(dbfile)
 ActiveRecord::Base.establish_connection(:adapter => "sqlite3", :database => dbfile)
 
+# load schema
 ActiveRecord::Migration.verbose = false
 ActiveRecord::Schema.define do
   create_table "users", :force => true do |t|
@@ -28,25 +38,25 @@ ActiveRecord::Schema.define do
   end
 end
 
+# models
 class User < ActiveRecord::Base
   has_one  :profile, :dependent => :destroy
   has_many :emails,  :dependent => :destroy, :order => 'id'
-
   replicate_natural_key :login
 end
 
 class Profile < ActiveRecord::Base
   belongs_to :user
-
   replicate_natural_key :user_id
 end
 
 class Email < ActiveRecord::Base
   belongs_to :user
-
   replicate_natural_key :user_id, :email
 end
 
+# The test case loads some fixture data once and uses transaction rollback to
+# reset fixture state for each test's setup.
 class ActiveRecordTest < Test::Unit::TestCase
   def setup
     self.class.fixtures
@@ -250,4 +260,93 @@ class ActiveRecordTest < Test::Unit::TestCase
       assert !user.emails.empty?, "#{login} has no emails" if login != 'tmm1'
     end
   end
+
+  def test_loading_with_replicating_id
+    objects = []
+    @dumper.listen do |type, id, attrs, obj|
+      objects << [type, id, attrs, obj] if type == 'User'
+    end
+
+    dumped_users = {}
+    %w[rtomayko kneath tmm1].each do |login|
+      user = User.find_by_login(login)
+      @dumper.dump user
+      dumped_users[login] = user
+    end
+    assert_equal 3, objects.size
+
+    User.destroy_all
+    User.replicate_id = false
+
+    # load everything back up
+    objects.each { |type, id, attrs, obj| User.load_replicant type, id, attrs }
+
+    user = User.find_by_login('rtomayko')
+    assert_not_equal dumped_users['rtomayko'].id, user.id
+
+    User.destroy_all
+    User.replicate_id = true
+
+    # load everything back up
+    objects.each { |type, id, attrs, obj| User.load_replicant type, id, attrs }
+
+    user = User.find_by_login('rtomayko')
+    assert_equal dumped_users['rtomayko'].id, user.id
+  end
+
+  def test_loader_saves_without_validations
+    # note when a record is saved with validations
+    ran_validations = false
+    User.class_eval { validate { ran_validations = true } }
+
+    # check our assumptions
+    user = User.create(:login => 'defunkt')
+    assert ran_validations, "should run validations here"
+    ran_validations = false
+
+    # load one and verify validations are not run
+    user = nil
+    @loader.listen { |type, id, attrs, obj| user = obj }
+    @loader.feed 'User', 1, 'login' => 'rtomayko'
+    assert_not_nil user
+    assert !ran_validations, 'validations should not run on save'
+  end
+
+  def test_loader_saves_without_callbacks
+    # note when a record is saved with callbacks
+    callbacks = false
+    User.class_eval { after_save { callbacks = true } }
+
+    # check our assumptions
+    user = User.create(:login => 'defunkt')
+    assert callbacks, "should run callbacks here"
+    callbacks = false
+
+    # load one and verify validations are not run
+    user = nil
+    @loader.listen { |type, id, attrs, obj| user = obj }
+    @loader.feed 'User', 1, 'login' => 'rtomayko'
+    assert_not_nil user
+    assert !callbacks, 'callbacks should not run on save'
+  end
+
+  def test_loader_saves_without_updating_created_at_timestamp
+    timestamp = Time.at((Time.now - (24 * 60 * 60)).to_i)
+    user = nil
+    @loader.listen { |type, id, attrs, obj| user = obj }
+    @loader.feed 'User', 23, 'login' => 'brianmario', 'created_at' => timestamp
+    assert_equal timestamp, user.created_at
+    user = User.find(user.id)
+    assert_equal timestamp, user.created_at
+  end
+
+  def test_loader_saves_without_updating_updated_at_timestamp
+    timestamp = Time.at((Time.now - (24 * 60 * 60)).to_i)
+    user = nil
+    @loader.listen { |type, id, attrs, obj| user = obj }
+    @loader.feed 'User', 29, 'login' => 'rtomayko', 'updated_at' => timestamp
+    assert_equal timestamp, user.updated_at
+    user = User.find(user.id)
+    assert_equal timestamp, user.updated_at
+  end
 end

data/test/dumper_test.rb

@@ -46,6 +46,7 @@ class DumperTest < Test::Unit::TestCase
 
   def test_writing_to_io
     io = StringIO.new
+    io.set_encoding 'BINARY' if io.respond_to?(:set_encoding)
     @dumper.marshal_to io
     @dumper.dump object = thing
     data = Marshal.dump(['Replicate::Object', object.id, object.attributes])

metadata

@@ -1,12 +1,12 @@
 --- !ruby/object:Gem::Specification 
 name: replicate
 version: !ruby/object:Gem::Version 
-  hash: 15
+  hash: 13
   prerelease: 
   segments: 
   - 1
-  - 0
-  version: "1.0"
+  - 1
+  version: "1.1"
 platform: ruby
 authors: 
 - Ryan Tomayko
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-09-08 00:00:00 Z
+date: 2011-09-09 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: activerecord
@@ -24,15 +24,29 @@ dependencies:
     requirements: 
     - - ~>
       - !ruby/object:Gem::Version 
-        hash: 7
+        hash: 5
         segments: 
-        - 2
-        - 2
-        version: "2.2"
+        - 3
+        - 1
+        version: "3.1"
   type: :development
   version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: sqlite3
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id002
 description: Dump and load relational objects between Ruby environments.
-email: r@tomayko.com
+email: ryan@github.com
 executables: 
 - replicate
 extensions: []
@@ -41,6 +55,7 @@ extra_rdoc_files: []
 
 files: 
 - COPYING
+- HACKING
 - README.md
 - Rakefile
 - bin/replicate
@@ -55,7 +70,7 @@ files:
 - test/dumper_test.rb
 - test/loader_test.rb
 - test/replicate_test.rb
-homepage: http://github.com/rtomayko/replicate/
+homepage: http://github.com/rtomayko/replicate
 licenses: []
 
 post_install_message: