zd 0.0.1

data/README.md

@@ -0,0 +1,220 @@
+# ZD
+
+ZD is a zero-downtime data migration framework that sits on top of the data store (or stores) of your choice. It implements zero-downtime by putting your data through a series of states:
+
+1.  **Unrun**:
+    The migration is implemented, but has not been run yet.
+2.  **Prepared**:
+    Any necessary creation of a "space" for the migrated data has been done. Typically not needed for schema-less stores. Your application code now writes to both the old and new data structures.
+3.  **Migrated**:
+    All pre-existing data has been copied and/or mutated in to the new locations while continuing to exist as-is in the old locations. The code base is still operating off of the old locations (and new data continues to be written to both locations).
+4.  **Switched**:
+    The code base now uses the new locations and ignores the old locations. Data is still written to both old and new locations. This is the point at which you would want to test the system to make sure the migration worked as expected. If something isn't right, the migration can still be rolled all the way back to the `unrun` state.
+5.  **Completed**:
+    The migration has been verified to be working, and new data is no longer written to the old locations. Migration-specific code can be stripped out of the code base now. Once a migration is `completed`, it cannot be rolled back. Any references to the migration in the codebase will generate a warning.
+6.  **Destroyed**:
+    The old locations for data have been removed from the data store. Any references to the migration in the codebase will raise an error.
+
+## Installation
+
+Just add zd to your Gemfile:
+
+    gem 'zd'
+
+And `bundle install`.
+
+## Usage
+
+To show how ZD works, lets walk through a simple example. Lets say you have a Person class, which used to store separate first names and last names. You've since expanded internationally and realized what a bad idea this is in general, and so now you need to fix your mistake down to the data level *without* taking your application down (though rolling restarts are OK). Your Person class looks like this initially:
+
+    class Person
+      include AwesomeDB
+
+      def first_name
+        read(:first_name)
+      end
+
+      def last_name
+        read(:last_name)
+      end
+
+      def first_name=(value)
+        write(:first_name, value)
+      end
+
+      def last_name=(value)
+        write(:last_name, value)
+      end
+
+      def name
+        [first_name, last_name].compact.join(" ")
+      end
+
+      def name=(value)
+        first_name, *rest = value.split(/\s+/)
+        write(:first_name, first_name)
+        write(:last_name, rest.join(" "))
+      end
+    end
+
+(The `read` and `write` methods are made-up access methods for the made-up AwesomeDB data store.)
+
+Currently you still have code using both the `name` and `first_name`/`last_name`, but you're slowly cleaning it up. The key thing is that all the methods on the class continue to obey their contract throughout the data migration.
+
+To get started you'll want to generate a new migration with `zd new <name>`. Migrations go in the db/migrate folder in your project, and use a timestamped filename (similar to ActiveRecord migrations). A fresh migration looks something like this:
+
+    class Migrations::MergeFirstAndLastName < ZD::Migration
+      register! depends_on: :nothing
+
+      def prepare
+      end
+
+      def migrate
+      end
+
+      def destroy
+      end
+    end
+
+
+And here is what it might look like after the migration is filled out:
+
+    class Migrations::MergeFirstAndLastName < ZD::Migration
+      register! depends_on: :nothing
+
+      def prepare
+        Person.add_field :name
+      end
+
+      def migrate
+        Person.each do |person|
+          person.name = [person.first_name, person.last_name].compact.join(" ")
+        end
+      end
+
+      def destroy
+        Person.remove_field :first_name
+        Person.remove_field :last_name
+      end
+    end
+
+
+The `Person.add_field` and `Person.remove_field` methods are made up; you would just use whatever your data store provides (if necessary; many schemaless datastores won't even need the `prepare` step).
+
+This is all well and good, but how does the model handle the fact that the data format is shifting around underneath it? ZD provides state-based methods that can be used to mark when which code should be run:
+
+    class Person
+      include AwesomeDB
+
+      def first_name
+        ZD[:merge_first_and_last_name].HANDLE do |m|
+          m.UNTIL_SWITCHED{read(:first_name)}
+          m.ONCE_SWITCHED{@first_name ||= name.split(/\s+/).first}
+        end
+      end
+
+      def last_name
+        ZD[:merge_first_and_last_name].HANDLE do |m|
+          m.UNTIL_SWITCHED{read(:last_name)}
+          m.ONCE_SWITCHED{@last_name ||= name.split(/\s+/)[1..-1].join(" ")}
+        end
+      end
+
+      def first_name=(value)
+        ZD[:merge_first_and_last_name].HANDLE do |m|
+          m.ONCE_PREPARED{write(:name, [value, last_name].compact.join(" "))}
+          m.UNTIL_COMPLETED{write(:first_name, value)}
+        end
+      end
+
+      def last_name=(value)
+        ZD[:merge_first_and_last_name].HANDLE do |m|
+          m.ONCE_PREPARED{write(:name, [first_name, value].compact.join(" "))}
+          m.UNTIL_COMPLETED{write(:last_name, value)}
+        end
+      end
+
+      def name
+        ZD[:merge_first_and_last_name].HANDLE do |m|
+          m.UNTIL_SWITCHED{return [first_name, last_name].compact.join(" ")}
+          m.ONCE_SWITCHED{read(:name)}
+        end
+      end
+
+      def name=(value)
+        ZD[:merge_first_and_last_name].HANDLE do |m|
+          m.ONCE_PREPARED{write(:name, value)}
+          m.UNTIL_COMPLETED do
+            first_name, *rest = value.split(/\s+/)
+            write(:first_name, first_name)
+            write(:last_name, rest.join(" "))
+          end
+        end
+      end
+    end
+
+
+The first thing you're probably thinking after seeing that is, "Who hit my code with the ugly stick!?!" But that's actually a feature of ZD: migration-specific code sticks out like a sore thumb so that there will be lots of motivation to strip it out once the migration is complete. Migration code should be robust but temporary.
+
+Once your migration and migration-specific code is in place, you can start walking your data through the migration states using `zd`:
+
+    $ zd prepare
+
+All migrations in the `unrun` state will be transitioned to the `prepared` state via the `prepare` action.
+
+    $ zd migrate
+
+All migrations in the `prepared` state will be transitioned to the `migrated` state via the `migrate` action.
+
+    $ zd switch
+
+All migrations in the `migrated` state will flip over to `switched`. This triggers all code to start using the new code paths.
+
+This is the point at which you should verify that your migrations have been successful and all the new code is working as expected in production. Getting back to the old state is as easy as `zd switchoff [name]`.
+
+    $ zd complete
+
+All migrations in the `switched` state will flip over to `completed`. This triggers all code to stop writing to old locations, and puts you past the point of no return for an easy rollback. Once you get here, it's time to go through your codebase and rip out the migration-specific code blocks, just leaving the code that deals with the new data structure.
+
+    $ zd destroy
+
+All migrations in the `completed` state will be transitioned to the `destroyed` state via the `destroy` action. Typically this is the point at which old data gets cleaned up. Note that once your migration gets to this state, continued references to it in your code will raise an error.
+
+And that's all there is to it! You can either leave old migration files from db/migrate, or delete them once you're done with them - the overhead for each one is very small. Oh, and here's what the Person class looks like once you're done:
+
+    class Person
+      include AwesomeDB
+
+      def first_name
+        @first_name ||= name.split(/\s+/).first
+      end
+
+      def last_name
+        @last_name ||= name.split(/\s+/)[1..-1].join(" ")
+      end
+
+      def first_name=(value)
+        write(:name, [value, last_name].compact.join(" "))
+      end
+
+      def last_name=(value)
+        write(:name, [first_name, value].compact.join(" "))
+      end
+
+      def name
+        read(:name)
+      end
+
+      def name=(value)
+        write(:name, value)
+      end
+    end
+
+No more ugly!
+
+
+## State Tracking
+
+## Dependencies
+
+## Philosophy

data/lib/zd.rb

@@ -0,0 +1,159 @@
+require "active_support/core_ext/string"
+
+module ZD
+  def self.[](name)
+    migrations[name]
+  end
+
+  def HANDLE(name)
+    yield(self[name])
+  end
+
+  def self.load!(path)
+    Dir["#{path}/*.rb"].each do |e|
+      if block_given?
+        yield(e)
+      else
+        require e
+      end
+    end
+  end
+
+  def self.migrations
+    @migrations ||= {}
+  end
+
+  def self.register_migration(name, migration, dependencies)
+    if dependencies.empty?
+      raise ArgumentError, "You must declare migration dependencies (or pass :nothing)"
+    end
+    migrations[name] = migration
+    dependencies.each do |d|
+      add_dependency(name, d)
+    end
+  end
+
+  def self.add_dependency(migration, depends_on)
+    raise ArgumentError, "Dependency #{depends_on} not found for #{migration}" unless(depends_on == :nothing || self[depends_on])
+    dependencies[depends_on] << migration
+  end
+
+  def self.dependencies
+    @dependencies ||= Hash.new{|h,k| h[k] = []}
+  end
+
+  class Migration
+    def self.register!(options={})
+      migration_name = (options[:name] || name.split(/::/).last.underscore.to_sym)
+      initial_state = (options[:initial_state] || :destroyed)
+      initial_state = initial_state.call if initial_state.respond_to?(:call)
+      ZD.register_migration(migration_name, new(initial_state), Array(options[:depends_on]))
+    end
+
+    STATES = [:unrun, :prepared, :migrated, :switched, :completed, :destroyed]
+    TRANSITIONS = {
+      unrun: :prepare,
+      prepared: :migrate,
+      migrated: :switch,
+      switched: :complete,
+      completed: :destroy,
+    }
+
+    attr_reader :state
+    def initialize(initial_state)
+      self.state = initial_state
+    end
+
+    def state=(new_state)
+      check_state(new_state)
+      @state = new_state
+    end
+
+    def check_state(state)
+      raise ArgumentError, "Unknown state #{state}" unless STATES.include?(state)
+    end
+
+    def migrate_to(new_state, silent=false)
+      @silent = silent
+      check_state(new_state)
+      currently = STATES.index(state)
+      target = STATES.index(new_state)
+      to_run = STATES[(currently)..(target-1)]
+      to_run.each do |s|
+        action_name = TRANSITIONS[s]
+        puts "Transitioning from #{s} via #{action_name}..."
+        send(action_name) if action_name && respond_to?(action_name)
+        self.state = s
+      end
+    end
+
+    def already?(in_state)
+      STATES.index(@state) >= STATES.index(in_state)
+    end
+
+    def HANDLE
+      yield self
+    end
+
+    def UNTIL_PREPARED
+      unless already?(:prepared)
+        yield
+      end
+    end
+
+    def ONCE_PREPARED
+      if already?(:prepared)
+        yield
+      end
+    end
+
+    def UNTIL_SWITCHED
+      unless already?(:switched)
+        yield
+      end
+    end
+
+    def ONCE_SWITCHED
+      if already?(:switched)
+        yield
+      end
+    end
+
+    def UNTIL_COMPLETED
+      unless already?(:completed)
+        yield
+      end
+    end
+
+    def ONCE_COMPLETED
+      if already?(:completed)
+        yield
+      end
+    end
+
+    def ONCE_DESTROYED
+      if already?(:destroyed)
+        yield
+      end
+    end
+
+    def TEST_ONCE_SWITCHED
+      old_state = state
+      migrate_to(:switched, true)
+      yield
+    ensure
+      self.state = old_state
+    end
+
+    def puts(v="\n")
+      super(v) unless @silent
+    end
+
+    def print(v)
+      super(v) unless @silent
+    end
+
+  end
+end
+
+module Migrations; end

data/lib/zd/version.rb

@@ -0,0 +1,3 @@
+module ZD
+  VERSION = "0.0.1"
+end

metadata

@@ -0,0 +1,62 @@
+--- !ruby/object:Gem::Specification
+name: zd
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Nathaniel Talbott
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2011-12-23 00:00:00.000000000Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: &70132156417960 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *70132156417960
+description: ZD implements zero-downtime migrations as a managed series of state changes,
+  with hooks for you to manage your code and data throughout the migration.
+email:
+- nathaniel@talbott.ws
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/zd.rb
+- lib/zd/version.rb
+homepage: http://github.com/ntalbott/zd
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.9.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.5.2
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.10
+signing_key: 
+specification_version: 3
+summary: A zero-downtime data migration framework that sits on top of the data store
+  (or stores) of your choice.
+test_files: []
+has_rdoc: