active_column 0.0.2 → 0.1

data/docs/Migrate.md

@@ -0,0 +1,100 @@
+## Data Migrations
+
+The very first thing I would like to say about ActiveColumn Cassandra data migration is that *I stole most of the code
+for this from the Rails gem (in ActiveSupport)*.  I made the necessary changes to update a Cassandra database
+instead of a relational DB.  These changes were sort of significant, but I just wanted to give credit where credit
+is due.
+
+With that out of the way, we can discuss how you would use ActiveColumn to perform data migrations.
+
+### Creating keyspaces
+
+First we will create our project's keyspaces.
+
+1. Make sure your cassandra 0.7 (or above) server is running.
+
+2. Make sure you have your _config/cassandra.yml_ file created.  The [README](../README.md) has an example of
+this file.
+
+The ActiveColumn gem gives you several rake tasks within the **ks:** namespace.  "ks" stands for keyspace, which is
+the equivalent of a database in MySQL (or other relational dbs).  To see the available tasks, run this rake command:
+
+<pre>
+rake -T ks
+</pre>
+
+3. Create your databases with the **ks:create:all** rake task:
+
+<pre>
+rake ks:create:all
+</pre>
+
+Voila!  You have now successfully created your keyspaces.  Now let's generate some migration files.
+
+### Creating and running migrations
+
+4. ActiveColumn includes a generator to help you create blank migration files.  To create a new migration, run this
+command:
+
+<pre>
+rails g active_column:migration NameOfYourMigration
+</pre>
+
+The name of the migration might be something like "CreateUsersColumnFamily".  After you run this command, you should see
+a new file that is located here:
+
+<pre>
+ks/migrate/20101229183849_create_users_column_family.rb
+</pre>
+
+Note that the date stamp on the file will be different depending on when you create the migration.  The migration file
+will look like this:
+
+<pre>
+class CreateUsersColumnFamily &lt; ActiveColumn::Migration
+
+  def self.up
+
+  end
+
+  def self.down
+
+  end
+
+end
+</pre>
+
+5. Edit your new migration file to do what you want it to.  For this migration, it would probably wind up looking like
+this:
+
+<pre>
+class CreateUsersColumnFamily &lt; ActiveColumn::Migration
+
+  def self.up
+    create_column_family :users
+  end
+
+  def self.down
+    drop_column_family :users
+  end
+
+end
+</pre>
+
+6. Run the migrate rake task (for development):
+
+<pre>
+rake ks:migrate
+</pre>
+
+This will create the column family for your development environment.  But you also need it in your test environment.
+For now, you have to do this like the following.  However, soon this will be updated to work more like ActiveRecord
+migrations.
+
+7. Run the migrate rake task (for test):
+
+<pre>
+RAILS_ENV=test rake ks:migrate
+</pre>
+
+And BAM!  You have your development and test keyspaces set up correctly.

data/docs/Query.md

@@ -0,0 +1,43 @@
+### Finding data
+
+Ok, congratulations - now you have a bunch of fantastic data in Cassandra.  How do you get it out?  ActiveColumn can
+help you here too.
+
+Here is how you look up data that have a simple key:
+
+<pre>
+tweets = Tweet.find( 'mwynholds', :reversed => true, :count => 3 )
+</pre>
+
+This code will find the last 10 tweets for the 'mwynholds' user in reverse order.  It comes back as a hash of arrays,
+and would looks like this if represented in JSON:
+
+<pre>
+{
+  'mwynholds': [ { 'user_id': 'mwynholds', 'message': 'I\'m going to bed now' },
+                 { 'user_id': 'mwynholds', 'message': 'It\'s lunch time' },
+                 { 'user_id': 'mwynholds', 'message': 'Just woke up' } ]
+}
+</pre>
+
+Here are some other examples and their return values:
+
+<pre>
+Tweet.find( [ 'mwynholds', 'all' ], :count => 2 )
+
+{
+  'mwynholds': [ { 'user_id': 'mwynholds', 'message': 'Good morning' },
+                 { 'user_id': 'mwynholds', 'message': 'Good afternoon' } ],
+  'all': [ { 'user_id': 'mwynholds', 'message': 'Good morning' },
+             'user_id': 'bmurray', 'message': 'Who ya gonna call!' } ]
+}
+</pre>
+
+<pre>
+Tweet.find( { 'user_id' => 'all', 'recipient_id' => [ 'fsinatra', 'dmartin' ] }, :reversed => true, :count => 1 )
+
+{
+  'all:fsinatra' => [ { 'user_id': 'mwynholds', 'recipient_ids' => [ 'fsinatra', 'dmartin' ], 'message' => 'Here we come Vegas!' } ],
+  'all:dmartin' => [ { 'user_id': 'fsinatra', 'recipient_ids' => [ 'dmartin' ], 'message' => 'Vegas was fun' } ]
+}
+</pre>

data/lib/active_column.rb

@@ -1,8 +1,27 @@
+require 'cassandra/0.7'
+require 'active_support'
+require 'active_support/core_ext/string'
+
 module ActiveColumn
 
-  autoload :Connection, 'active_column/connection'
-  autoload :Base,       'active_column/base'
-  autoload :Version,    'active_column/version'
+  autoload :Base,           'active_column/base'
+  autoload :Connection,     'active_column/connection'
+  autoload :KeyConfig,      'active_column/key_config'
+  autoload :Version,        'active_column/version'
+
+  require                   'active_column/errors'
+  require                   'active_column/migration'
+
+  module Tasks
+    autoload :Keyspace,     'active_column/tasks/keyspace'
+    autoload :ColumnFamily, 'active_column/tasks/column_family'
+
+    require                 'active_column/tasks/ks'
+  end
+
+  module Generators
+    require                 'active_column/generators/migration_generator'
+  end
 
   extend Connection
   

data/lib/active_column/base.rb

@@ -1,59 +1,33 @@
 module ActiveColumn
 
-  class Base
-
-    attr_reader :attributes
+  def self.included(base)
+    base.extend ClassMethods
+  end
 
-    def initialize(attrs = {})
-      @attributes = attrs
-    end
+  module ClassMethods
 
-    def self.column_family(column_family = nil)
-      return @column_family if column_family.nil?
+    def column_family(column_family = nil)
+      return @column_family || self.name.tableize.to_sym if column_family.nil?
       @column_family = column_family
     end
 
-    def self.key(key, options = {})
+    def key(key, options = {})
       @keys ||= []
       @keys << KeyConfig.new(key, options)
     end
 
-    def save()
-      value = { SimpleUUID::UUID.new => self.to_json }
-      key_parts = self.class.keys.each_with_object( {} ) do |key_config, key_parts|
-        key_parts[key_config.key] = get_keys(key_config)
-      end
-      keys = self.class.generate_keys(key_parts)
-
-      keys.each do |key|
-        ActiveColumn.connection.insert(self.class.column_family, key, value)
-      end
-
-      self
+    def keys
+      @keys
     end
 
-    def self.find(key_parts, options = {})
+    def find(key_parts, options = {})
       keys = generate_keys key_parts
       ActiveColumn.connection.multi_get(column_family, keys, options).each_with_object( {} ) do |(user, row), results|
-        results[user] = row.to_a.collect { |(_uuid, col)| new(JSON.parse(col)) }
+        results[user] = row.to_a.collect { |(_uuid, col)| new(ActiveSupport::JSON.decode(col)) }
       end
     end
 
-    def to_json(*a)
-      @attributes.to_json(*a)
-    end
-
-    private
-
-    def self.keys
-      @keys
-    end
-
-    def get_keys(key_config)
-      key_config.func.nil? ? attributes[key_config.key] : self.send(key_config.func)
-    end
-
-    def self.generate_keys(key_parts)
+    def generate_keys(key_parts)
       if keys.size == 1
         key_config = keys.first
         value = key_parts.is_a?(Hash) ? key_parts[key_config.key] : key_parts
@@ -73,17 +47,29 @@ module ActiveColumn
 
   end
 
-  class KeyConfig
-    attr_accessor :key, :func
+  def initialize(attrs = {})
+    attrs.each do |attr, value|
+      send("#{attr}=", value) if respond_to?("#{attr}=")
+    end
+  end
 
-    def initialize(key, options)
-      @key = key
-      @func = options[:values]
+  def save()
+    value = { SimpleUUID::UUID.new => ActiveSupport::JSON.encode(self) }
+    key_parts = self.class.keys.each_with_object( {} ) do |key_config, key_parts|
+      key_parts[key_config.key] = self.send(key_config.func)
     end
+    keys = self.class.generate_keys(key_parts)
 
-    def to_s
-      "KeyConfig[#{key}, #{func or '-'}]"
+    keys.each do |key|
+      ActiveColumn.connection.insert(self.class.column_family, key, value)
     end
+
+    self
+  end
+
+  class Base
+    include ActiveColumn
   end
 
-end
+end
+

data/lib/active_column/errors.rb

@@ -0,0 +1,6 @@
+module ActiveColumn
+
+  class ActiveColumnError < StandardError
+  end
+
+end

data/lib/active_column/generators/migration_generator.rb

@@ -0,0 +1,31 @@
+require 'rails/generators'
+require 'rails/generators/named_base'
+
+module ActiveColumn
+  module Generators
+    class MigrationGenerator < Rails::Generators::NamedBase
+
+      source_root File.expand_path("../templates", __FILE__)
+
+      def self.banner
+        "rails g active_column:migration NAME"
+      end
+
+      def self.desc(description = nil)
+<<EOF
+Description:
+  Create an empty Cassandra migration file in 'ks/migrate'.  Very similar to Rails database migrations.
+
+Example:
+  `rails g active_column:migration CreateFooColumnFamily`
+EOF
+      end
+
+      def create
+        timestamp = Time.now.utc.strftime("%Y%m%d%H%M%S")
+        template 'migration.rb.erb', "ks/migrate/#{timestamp}_#{file_name.tableize}.rb"
+      end
+
+    end
+  end
+end

data/lib/active_column/generators/templates/migration.rb.erb

@@ -0,0 +1,11 @@
+class <%= name %> < ActiveColumn::Migration
+
+  def self.up
+
+  end
+
+  def self.down
+
+  end
+
+end

data/lib/active_column/key_config.rb

@@ -0,0 +1,16 @@
+module ActiveColumn
+
+  class KeyConfig
+    attr_accessor :key, :func
+
+    def initialize(key, options)
+      @key = key
+      @func = options[:values] || key
+    end
+
+    def to_s
+      "KeyConfig[#{key}, #{func or '-'}]"
+    end
+  end
+  
+end

data/lib/active_column/migration.rb

@@ -0,0 +1,269 @@
+module ActiveColumn
+
+  class IrreversibleMigration < ActiveColumnError
+  end
+
+  class DuplicateMigrationVersionError < ActiveColumnError#:nodoc:
+    def initialize(version)
+      super("Multiple migrations have the version number #{version}")
+    end
+  end
+
+  class DuplicateMigrationNameError < ActiveColumnError#:nodoc:
+    def initialize(name)
+      super("Multiple migrations have the name #{name}")
+    end
+  end
+
+  class UnknownMigrationVersionError < ActiveColumnError #:nodoc:
+    def initialize(version)
+      super("No migration with version number #{version}")
+    end
+  end
+
+  class IllegalMigrationNameError < ActiveColumnError#:nodoc:
+    def initialize(name)
+      super("Illegal name for migration file: #{name}\n\t(only lower case letters, numbers, and '_' allowed)")
+    end
+  end
+
+  class Migration
+
+    def self.connection
+      $cassandra
+    end
+
+    def self.migrate(direction)
+      return unless respond_to?(direction)
+      send direction
+    end
+
+    def self.create_column_family(name, options = {})
+      ActiveColumn::Tasks::ColumnFamily.new.create(name, options)
+    end
+
+    def self.drop_column_family(name)
+      ActiveColumn::Tasks::ColumnFamily.new.drop(name)
+    end
+
+  end
+
+  # MigrationProxy is used to defer loading of the actual migration classes
+  # until they are needed
+  class MigrationProxy
+
+    attr_accessor :name, :version, :filename
+
+    delegate :migrate, :announce, :write, :to=>:migration
+
+    private
+
+    def migration
+      @migration ||= load_migration
+    end
+
+    def load_migration
+      require(File.expand_path(filename))
+      name.constantize
+    end
+
+  end
+
+  class Migrator
+
+    def self.migrate(migrations_path, target_version = nil)
+      case
+        when target_version.nil?
+          up(migrations_path, target_version)
+        when current_version == 0 && target_version == 0
+        when current_version > target_version
+          down(migrations_path, target_version)
+        else
+          up(migrations_path, target_version)
+      end
+    end
+
+    def self.rollback(migrations_path, steps = 1)
+      move(:down, migrations_path, steps)
+    end
+
+    def self.forward(migrations_path, steps = 1)
+      move(:up, migrations_path, steps)
+    end
+
+    def self.up(migrations_path, target_version = nil)
+      self.new(:up, migrations_path, target_version).migrate
+    end
+
+    def self.down(migrations_path, target_version = nil)
+      self.new(:down, migrations_path, target_version).migrate
+    end
+
+    def self.run(direction, migrations_path, target_version)
+      self.new(direction, migrations_path, target_version).run
+    end
+
+    def self.migrations_path
+      'ks/migrate'
+    end
+
+    def self.schema_migrations_column_family
+      :schema_migrations
+    end
+
+    def self.get_all_versions
+      cas = ActiveColumn.connection
+      cas.get(schema_migrations_column_family, 'all').map {|(name, _value)| name.to_i}.sort
+    end
+
+    def self.current_version
+      sm_cf = schema_migrations_column_family
+      cf = ActiveColumn::Tasks::ColumnFamily.new
+      if cf.exists?(sm_cf)
+        get_all_versions.max || 0
+      else
+        0
+      end
+    end
+
+    private
+
+    def self.move(direction, migrations_path, steps)
+      migrator = self.new(direction, migrations_path)
+      start_index = migrator.migrations.index(migrator.current_migration)
+
+      if start_index
+        finish = migrator.migrations[start_index + steps]
+        version = finish ? finish.version : 0
+        send(direction, migrations_path, version)
+      end
+    end
+
+    public
+
+    def initialize(direction, migrations_path, target_version = nil)
+      cf = ActiveColumn::Tasks::ColumnFamily.new
+      sm_cf = self.class.schema_migrations_column_family
+
+      unless cf.exists?(sm_cf)
+        cf.create(sm_cf, :comparator_type => 'LongType')
+      end
+
+      @direction, @migrations_path, @target_version = direction, migrations_path, target_version
+    end
+
+    def current_version
+      migrated.last || 0
+    end
+
+    def current_migration
+      migrations.detect { |m| m.version == current_version }
+    end
+
+    def run
+      target = migrations.detect { |m| m.version == @target_version }
+      raise UnknownMigrationVersionError.new(@target_version) if target.nil?
+      unless (up? && migrated.include?(target.version.to_i)) || (down? && !migrated.include?(target.version.to_i))
+        target.migrate(@direction)
+        record_version_state_after_migrating(target)
+      end
+    end
+
+    def migrate
+      current = migrations.detect { |m| m.version == current_version }
+      target = migrations.detect { |m| m.version == @target_version }
+
+      if target.nil? && !@target_version.nil? && @target_version > 0
+        raise UnknownMigrationVersionError.new(@target_version)
+      end
+
+      start = up? ? 0 : (migrations.index(current) || 0)
+      finish = migrations.index(target) || migrations.size - 1
+      runnable = migrations[start..finish]
+
+      # skip the last migration if we're headed down, but not ALL the way down
+      runnable.pop if down? && !target.nil?
+
+      runnable.each do |migration|
+        #puts "Migrating to #{migration.name} (#{migration.version})"
+
+        # On our way up, we skip migrating the ones we've already migrated
+        next if up? && migrated.include?(migration.version.to_i)
+
+        # On our way down, we skip reverting the ones we've never migrated
+        if down? && !migrated.include?(migration.version.to_i)
+          migration.announce 'never migrated, skipping'; migration.write
+          next
+        end
+
+        migration.migrate(@direction)
+        record_version_state_after_migrating(migration)
+      end
+    end
+
+    def migrations
+      @migrations ||= begin
+        files = Dir["#{@migrations_path}/[0-9]*_*.rb"]
+
+        migrations = files.inject([]) do |klasses, file|
+          version, name = file.scan(/([0-9]+)_([_a-z0-9]*).rb/).first
+
+          raise IllegalMigrationNameError.new(file) unless version
+          version = version.to_i
+
+          if klasses.detect { |m| m.version == version }
+            raise DuplicateMigrationVersionError.new(version)
+          end
+
+          if klasses.detect { |m| m.name == name.camelize }
+            raise DuplicateMigrationNameError.new(name.camelize)
+          end
+
+          migration = MigrationProxy.new
+          migration.name     = name.camelize
+          migration.version  = version
+          migration.filename = file
+          klasses << migration
+        end
+
+        migrations = migrations.sort_by { |m| m.version }
+        down? ? migrations.reverse : migrations
+      end
+    end
+
+    def pending_migrations
+      already_migrated = migrated
+      migrations.reject { |m| already_migrated.include?(m.version.to_i) }
+    end
+
+    def migrated
+      @migrated_versions ||= self.class.get_all_versions
+    end
+
+    private
+
+    def record_version_state_after_migrating(migration)
+      cas = ActiveColumn.connection
+      sm_cf = self.class.schema_migrations_column_family
+
+      @migrated_versions ||= []
+      if down?
+        @migrated_versions.delete(migration.version)
+        cas.remove sm_cf, 'all', migration.version
+      else
+        @migrated_versions.push(migration.version).sort!
+        cas.insert sm_cf, 'all', { migration.version => migration.name }
+      end
+    end
+
+    def up?
+      @direction == :up
+    end
+
+    def down?
+      @direction == :down
+    end
+
+  end
+
+end