mongoid_lazy_migration 0.2.1 → 0.5.1

data/README.md

@@ -3,45 +3,50 @@ Mongoid Lazy Migration
 
 [![Build Status](https://secure.travis-ci.org/nviennot/mongoid_lazy_migration.png?branch=master)](http://travis-ci.org/nviennot/mongoid_lazy_migration)
 
-LazyMigration allows you to migrate your document on the fly.
-The migration is run when an instance of your model is initialized.
-While your frontend servers are migrating the documents on demand, some workers
-traverse the collection to migrate all the documents.
-
-It allows your application logic to be tolerant from having only a portion of
-the collection migrated. We use it at [Crowdtap](http://crowdtap.it/about-us).
-
-LazyMigration is designed to support only one production environment. There are
-no versioning as opposed to traditional ActiveRecord migrations.
-With this in mind, the migration code is directly implemented in the model
-itself, as opposed to having a decorator.
-Once the migration is complete, the migration code must be deleted, and code
-must be redeployed before cleaning up the database.
-
-LazyMigration has two modes of operations: *atomic* and *locked*.
-Atomic is the default one, which is less prone to fuck ups, but adds
-constrains on what you can do during your migration.
-
-Atomic Mode
------------
+LazyMigration allows you to migrate a Mongoid collection on the fly. As
+instances of your model are initialized, the migration is run. In the
+background, workers can traverse the collection and migrate other documents.
+Thus, your application acts as though your migration as already taken place.
+
+LazyMigration can be used for any app which uses Mongoid. It is most commonly
+used for Rails apps. We use it at [Crowdtap](http://crowdtap.it/about-us).
+
+
+Workflow
+--------
+
+Once installed with
+
+```ruby
+gem 'mongoid_lazy_migration'
+```
+
+You can use the following recipe to perform a migration:
+
+1. Include Mongoid::LazyMigration in your document and write your migration
+   specification. Modify your application code to reflect the changes from the
+   migration.
+2. Deploy.
+3. Run `rake db:mongoid:migrate`.
+4. Remove the migration block from your model.
+5. Deploy.
+6. Run `rake db:mongoid:cleanup[Model]`.
+
+The migration specification can be written in one of two modes: *atomic* or
+*locked*. Atomic is the default, and is tolerant to dying workers, but it
+introduces some constraints on your migration.
+
+Atomic Mode (default)
+---------------------
 
-Your migration code must respect a few constrains:
+### Constraints
 
-1. Your migrate code should never write anything to the database.
-   You should not call save yourself, but let LazyMigration do it.
-   Basically you are only allowed to set some document fields.
-2. Your migrate code should be deterministic (don't play with Random)
-   to keep some sort of consistency even in the presence of races with
-   migrations.
+Atomic migration code must respect a few constrains:
 
-When multiple clients are trying to perform the migration at the same
-time, all of them will run the migration code, but only one will commit
-the changes to the database.
-After the migration, the document will not be reloaded, regardless if the
-write to the database happened. This implies that the field values are
-the one from the migration, and may not correspond to what is actually in
-the database. Which is why using a non deterministic migration is not
-recommended.
+1. It can never write anything to the database. This means you should not call
+   save yourself; LazyMigration will do this for you. Essentially, you should
+   only be setting document fields.
+2. It must be deterministic to ensure consistency in the face of migration races.
 
 ### Migration States
 
@@ -53,7 +58,7 @@ A document migration goes through the following states:
 ### Atomic Migration Example
 
 Suppose we have a model GroupOfPeople with two fields: num_males and
-num_females. We wish to add a field num_people. Here is how it goes:
+num_females. We wish to add the field num_people. We can do this as follows:
 
 ```ruby
 class GroupOfPeople
@@ -68,6 +73,8 @@ class GroupOfPeople
     self.num_people = num_males + num_females
   end
 
+  # This can be used by other code while the migration is running
+  # because it is atomic
   def inc_num_people
     self.inc(:num_people, 1)
   end
@@ -80,19 +87,18 @@ because only one client will be able to commit the migration to the database.
 Locked Mode
 -----------
 
-The locked mode  guarantees that only one client will run the migration code,
-because a lock is taken. As a consequence, your migration code can pretty much
-do anything.
+Locked mode guarantees that only one client will run the migration code, because
+it uses locking. Thus, the restrictions on an atomic migration are removed.
 
-In case of contention, the other clients will have to wait until the migration
-is complete. This has some heavy consequences: if the owner of the lock dies
-(exception, ctrl+c, lost the ssh connection, asteroid crashes on your
-datacenter, ...) the document will stay locked in the processing state. There
-is no rollback. You are responsible to clean up the mess. Be aware that you
-cannot instantiate a document stuck in locked state state without removing
-the migration block.
+However, this has some consequences. Most importantly, if the owner of the lock
+dies (exception, ctrl+c, a lost ssh connection, an explosion in your
+datacenter, etc.), the document will stay locked in the processing state.
+There is no automatic rollback. You are responsible for cleaning up. Be aware
+that you cannot instantiate a document stuck in locked state state without
+removing the migration block.
 
-Because of the locking, a locked migration runs slower than an atomic one.
+Because the lock involves additional database requests, including writes,
+a locked migration runs slower than an atomic one.
 
 ### Migration States
 
@@ -105,8 +111,9 @@ A document migration goes through the following states:
 ### Locked Migration Example
 
 Suppose we have a model GroupOfPeople with an array people_names, which is an
-array of strings. Our migration consists of introducing a new model Person, and
-removing the array people_names from GroupOfPeople. Here it goes:
+
+an array of strings. Our migration consists of introducing a new model called
+Person and removing the array people_names from GroupOfPeople.
 
 ```ruby
 class Person
@@ -131,56 +138,56 @@ class GroupOfPeople
 end
 ```
 
+We cannot perform an atomic migration in this case because new documents are
+created while running the migration block.  In locked mode, we are guaranteed
+that we only create the associations once.
+
 Since only one client can execute the migration block, we are guaranteed that
 we only create the associations once.
 
-Just to be safe, we don't unset the people_names in the migration. It would
-be done manually once the migration is complete.
+Notice that we don't unset the people_names field in the migration. We keep it
+until the entire collection has migrated allowing us to rollback in case of
+failure.
 
 Background Migration
 --------------------
 
-While the frontend servers are migrating models on demand, it is recommended to
-migrate the rest of the collection in the background.
+While the some of your server migrating models on demand, it is recommended to
+migrate the rest of the collection in the background. Otherwise, if one
+document in your collection is not accessed for a year, your migration will
+take one year to complete.
 
-It can be done programmatically:
+A rake task is provided:
 
 ```ruby
-# migrate all the models that have a migration block.
-Mongoid::LazyMigration.migrate
+# Migrate all the documents that have a pending migration
+rake db:mongoid:migrate
 
-# migrate only Model.
-Mongoid::LazyMigration.migrate(Model)
+# Migrate all the documents of GroupOfPeople
+rake db:mongoid:migrate[GroupOfPeople]
 
-# migrate only active documents of Model.
-Mongoid::LazyMigration.migrate(Model.where(:active => true))
+# Migrate all the documents of GroupOfPeople that have specific people in the group
+rake db:mongoid:migrate[GroupOfPeople.where(:people_names.in => ["marie", "joelle"])]
 ```
 
-A rake task is provided wrapping the migrate method:
+Note that you might need to use single quotes around the whole rake argument,
+otherwise your shell might be tempted to evaluate the expression.
 
-```ruby
-rake db:mongoid:migrate
-rake db:mongoid:migrate[Model]
-rake db:mongoid:migrate[Model.where(:active => true)]
-```
+The task displays a progress bar.
 
-Both methods display a progress bar.
+For performance, it is recommended to migrate the most accessed documents first
+so they don't need to be migrated when a user requests them. You can also use
+multiple workers on different shards.
 
-It is recommended to migrate the most accessed documents first so they don't
-need to be migrated when a user requests them. It is also possible to use
-multiple workers. Example:
+If your database doesn't fit entirely in memory, be very careful when migrating
+rarely accessed documents since your working set may be evicted from cache.
+MongoDB could start trashing in unexpected ways.
 
-Worker 1:
-```ruby
-rake db:mongoid:migrate[User.where(:shard => 1, :active => true)]
-rake db:mongoid:migrate[User.where(:shard => 1, :active => false)]
-```
+The migration is be performed one document at a time, so we avoid holding up
+the global lock on MongoDB for long period of time.
 
-Worker 2:
-```ruby
-rake db:mongoid:migrate[User.where(:shard => 2, :active => false)]
-rake db:mongoid:migrate[User.where(:shard => 2, :active => true)]
-```
+Only atomic migration can be safely aborted with Ctrl+C. Support for aborting a
+locked migration will be added in the future.
 
 Cleaning up
 -----------
@@ -189,71 +196,49 @@ Once all the document have their migration state equal to done, you must do two
 
 1. Remove the migration block from your model.
 2. Cleanup the database (removing the migration_state field) to ensure that you
-   can run a future migration.
-
-The two ways to cleanup a collection are:
+   can run a future migration with the following rake task:
 
 ```ruby
-# programmatically
-Mongoid::LazyMigration.cleanup(Model)
-
-# rake task
 rake db:mongoid:cleanup_migration[Model]
 ```
 
 The cleanup process will be aborted if any of the document migrations are still
-in the processing state, or if you haven't removed the migration block.
+the processing state or if you haven't removed the migration block.
 
 Important Considerations
 ------------------------
 
 A couple of points that you need to be aware of:
 
+* save() will be called on the document once done with the migration. Don't
+  do it yourself. It's racy in atomic mode and unnecessary in locked mode.
+* save and update callbacks will not be called when persisting the migration.
+* No validation will be performed during the migration.
+* The migration will not update the `updated_at` field.
+* LazyMigration is designed to support a single production environment and does
+  not support versioning, unlike traditional ActiveRecord migrations. Thus, the
+  migration code can be included directly in the model and can be removed after
+  migration is done.
 * Because the migrations are run during the model instantiation,
   using some query like `Member.count` will not perform any migration.
-  With the same rational, If you do something sneaky in your application by
-  interacting directly with the mongo driver and bypassing mongoid, you are on
-  your own.
+  Similarly, if you bypass Mongoid and use the Mongo driver directly in your
+  application, LazyMigration might not be run.
 * If you use only() in some of your queries, make sure that you include the
   migration_state field.
 * Do not use identity maps and threads at the same time. It is currently
-  unsupported, but support may be added in the future.
+  unsupported, though support may be added in the future.
 * Make sure you can migrate a document quickly, because migrations will be
   performed while processing user requests.
 * SlaveOk is fine, even in locked mode.
-* save() will be called on the document once done with the migration, don't
-  do it yourself.
-* The save and update callbacks will not be called when persisting the
-  migration.
-* No validation will be performed during the migration.
-* The migration will not update the updated_at field.
-
-Workflow
---------
-
-First, add the following line in your Gemfile:
-
-```ruby
-gem 'mongoid_lazy_migration'
-```
-
-Follow the following steps to perform a migration:
-
-1. Run `rake db:mongoid:cleanup[Model]` just to be safe.
-2. Include `Mongoid::LazyMigration` in your document and write your migration.
-   Modify your application code to reflect the changes from the migration.
-3. Deploy.
-4. Run `rake db:mongoid:migrate`.
-5. Remove the migration block from your model.
-6. Deploy.
-7. Run `rake db:mongoid:cleanup[Model]`.
+* You might want to try a migration on a staging environment which replicates a
+  production workload to evaluate the impact of the lazy migration.
 
 Compatibility
 -------------
 
 LazyMigration is tested against against MRI 1.8.7, 1.9.2, 1.9.3, JRuby-1.8, JRuby-1.9.
 
-Only Mongoid 2.4.x is currently supported.
+Both Mongoid 2.4.x and Mongoid 3.0.x are supported.
 
 License
 -------

data/lib/mongoid/lazy_migration.rb

@@ -11,6 +11,9 @@ module Mongoid
     extend ActiveSupport::Concern
     extend Tasks
 
+    mattr_reader :mongoid3
+    @@mongoid3 = Gem.loaded_specs['mongoid'].version >= Gem::Version.new('3.0.0')
+
     mattr_reader :models_to_migrate
     @@models_to_migrate = Set.new
 

data/lib/mongoid/lazy_migration/document.rb

@@ -9,9 +9,9 @@ module Mongoid::LazyMigration::Document
   end
 
   def atomic_selector
-    return super unless @running_migrate_block
+    return super unless @migrating
 
-    unless self.class.lock_migration
+    if @running_migrate_block && !self.class.lock_migration
         raise ["You cannot save during an atomic migration,",
                "You are only allowed to set the document fields",
                "The document will be commited once the migration is complete.",
@@ -19,20 +19,22 @@ module Mongoid::LazyMigration::Document
         ].join("\n")
     end
 
+    # see perform migration
     super.merge(:migration_state => { "$ne" => :done })
   end
 
   def run_callbacks(*args, &block)
-    if @migrating
-      block.call if block
-    else
-      super(*args, &block)
-    end
+    return super(*args, &block) unless @migrating
+
+    block.call if block
   end
 
   private
 
   def perform_migration
+    # For locked migrations, perform_migration is bypassed when raced by
+    # another client. We busy wait until the other client is done with the
+    # migration in wait_for_completion.
     return if self.class.lock_migration && !try_lock_migration
 
     begin
@@ -63,10 +65,16 @@ module Mongoid::LazyMigration::Document
     # transitions from the pending to the progress state. This operation is
     # done atomically by MongoDB (test and set).
     self.migration_state = :processing
-    collection.update(
-      atomic_selector.merge(:migration_state => { "$in" => [nil, :pending] }),
-      { "$set" => { :migration_state => :processing }},
-      { :safe => true })['updatedExisting']
+
+    selector = atomic_selector.merge(:migration_state => { "$in" => [nil, :pending] })
+    changes  = { "$set" => { :migration_state => :processing }}
+    safety   = { :safe => true }
+
+    if Mongoid::LazyMigration.mongoid3
+      self.class.with(safety).where(selector).query.update(changes)
+    else
+      collection.update( selector, changes, safety)
+    end['updatedExisting']
   end
 
   def wait_for_completion

data/lib/mongoid/lazy_migration/tasks.rb

@@ -25,9 +25,15 @@ module Mongoid::LazyMigration::Tasks
              "Don't forget to remove the migration block"].join("\n")
     end
 
-    model.collection.update(
-      { :migration_state => { "$exists" => true }},
-      {"$unset" => { :migration_state => 1}},
-      { :safe => true, :multi => true })
+    selector = { :migration_state => { "$exists" => true }}
+    changes  = {"$unset" => { :migration_state => 1}}
+    safety   = { :safe => true, :multi => true }
+    multi    = { :multi => true }
+
+    if Mongoid::LazyMigration.mongoid3
+      model.with(safety).where(selector).query.update(changes, multi)
+    else
+      model.collection.update(selector, changes, safety.merge(multi))
+    end
   end
 end

data/lib/mongoid/lazy_migration/version.rb

@@ -1,5 +1,5 @@
 module Mongoid
   module LazyMigration
-    VERSION = "0.2.1"
+    VERSION = "0.5.1"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mongoid_lazy_migration
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.5.1
   prerelease: 
 platform: ruby
 authors:
@@ -9,14 +9,14 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-07-10 00:00:00.000000000 Z
+date: 2012-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mongoid
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ~>
+    - - ! '>='
       - !ruby/object:Gem::Version
         version: '2.4'
   type: :runtime
@@ -24,7 +24,7 @@ dependencies:
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ~>
+    - - ! '>='
       - !ruby/object:Gem::Version
         version: '2.4'
 - !ruby/object:Gem::Dependency
@@ -69,9 +69,9 @@ files:
 - lib/mongoid_lazy_migration.rb
 - lib/mongoid/lazy_migration/version.rb
 - lib/mongoid/lazy_migration/tasks.rb
+- lib/mongoid/lazy_migration/document.rb
 - lib/mongoid/lazy_migration/railtie/migrate.rake
 - lib/mongoid/lazy_migration/railtie.rb
-- lib/mongoid/lazy_migration/document.rb
 - lib/mongoid/lazy_migration.rb
 - README.md
 homepage: http://github.com/nviennot/mongoid_lazy_migration