cassie 1.0.0.beta.33 → 1.0.0.1

data/lib/cassie/schema/README.md

@@ -0,0 +1,306 @@
+# Cassie Schema Migrations
+
+Cassie provides versioned migrations similar to many existing adapters and frameworks.
+
+Practically speaking, this gives your cluster Schema its own "version" and simplifies upgrading or rolling back the schema.
+
+As such, Cassie uses semantic versioning, where a defined version describes the keyspace defined in the cluster configuration.
+
+Major, minor, patch, and build versions are used, however semantic extensions are not recommended (prerelase or metadata, eg: 1.0.0.beta).
+
+### Schema Migrations
+
+Schema version and migrations are managed through various `cassie` tasks. See below for basic usage.
+
+List command descriptions and advanced options:
+```
+cassie --help
+```
+
+#### Getting Started
+
+If no schema has been defined yet (e.g. no keyspace, tables, or types), simply initialize Cassie versioning:
+
+```
+cassie schema:init
+```
+
+```
+-- Initializing Cassie Versioning
+-- done
+-- Initializing 'my_app_development' Keyspace
+-- done
+```
+
+If an existing schema (e.g. keyspace, tables, types) is alredy defined, see below on how to import it.
+
+#### Coming from `cassandra_migrations`
+
+Import your existing `cassandra_migrations` migration files with a single task:
+
+```
+cassie migrations:import
+```
+```bash
+-- Importing `cassandra_migrations` migration files
+   - Importing db/cassandra_migrate/20161206214301_initial_database.rb
+     > created /db/cassandra/migrations/0000_0000_0000_0001_initial_database.rb
+     > recorded version 0.0.0.1
+   - done
+   - Importing db/cassandra_migrate/20161212210447_add_username_to_users.rb
+     > created /db/cassandra/migrations/0000_0000_0001_0000_add_username_to_users.rb
+     > recorded version 0.0.1.0
+   - done
+   - Importing db/cassandra_migrate/20161213163201_add_reserved_to_users.rb
+     > created /db/cassandra/migrations/0000_0000_0002_0000_add_reserved_to_users.rb
+     > recorded version 0.0.2.0
+   - done
+-- done
+```
+
+> The original `cassandra_migrations` migration files and schema in the physical layer are not changed. Remove the old files when comfortable.
+
+
+#### Coming from no explicit migration/versioning management
+
+Import your existing schema held in Cassandra with a single task:
+
+```
+cassie schema:import
+```
+
+```
+-- Initializing Cassie Versioning
+-- done
+-- Initializing 'my_app_development' Keyspace
+   > 'my_app_development' already exists
+-- done
+-- Importing Schema from Cassandra
+   - Creating initial version
+     > created db/cassandra/migrations/0000_0000_0001_0000_import_my_app_development.rb
+     > recorded version 0.0.1.0
+   - done
+-- done
+-- Dumping Cassandra schema (version 0.0.1.0)
+   - Writing to db/cassandra/schema.cql
+   - done
+-- done
+```
+
+#### Creating a migration
+
+```
+cassie migration:create that killer feature
+```
+
+```
+-- Creating migration file for version 0.0.1.0
+   > created db/cassandra/migrations/0000_0000_0001_0000_that_killer_feature.rb
+-- done
+```
+
+```ruby
+# db/cassandra/migrations/0000_0000_0001_0000_that_killer_feature.rb
+class Migration_0_0_1_0 < Cassie::Schema::Migration
+  def up
+    # Code to execute when applying this migration
+    # Supports the excellent `cassandra_migrations` DSL
+    # or call `execute` to call `Cassandra::Session.execute`
+  end
+
+  def down
+    # Code to execute when rolling back this migration
+    # Supports the excellent `cassandra_migrations` DSL
+    # or call `execute` to call `Cassandra::Session.execute`
+  end
+end
+
+```
+
+By default, the `patch` version will be bumped. Use the `--major` (`-M`), `--minor` (`-m`), or `--build` (`-b`) switches to bump differently. Or, explicitly set the version with `--version` (`-v`): `cassie migration:create that fixup -v 0.1.3.15`.
+
+> *Note:* The class name only needs to match the version in the filename. The description suffix can be changed without having to change the classname.
+
+#### Executing migrations
+
+##### Roll up to latest version
+
+```
+cassie migrate
+```
+
+##### Roll up to a specific version
+
+```
+cassie migrate 0.2.0
+```
+
+##### Rolling back
+
+Use the same interface to migrate up or down to a specific version.
+
+```
+cassie migrate 0.1.9
+```
+
+#### Reporting the current version
+```
+cassie schema:version
+```
+```
++-----------+----------------+-------------+---------------------------+
+|  Version  | Description    | Migrated by | Migrated at               |
++-----------+----------------+-------------+---------------------------+
+| * 0.2.0.0 |  create users  | serverbot   | 2016-09-08 10:23:54 -0500 |
++-----------+----------------+-------------+---------------------------+
+```
+
+#### Reporting the version history
+```
+cassie schema:history
+```
+```
++-----------+----------------+-------------+---------------------------+
+|  Version  | Description    | Migrated by | Migrated at               |
++-----------+----------------+-------------+---------------------------+
+| * 0.2.0.0 |  create users  | serverbot   | 2016-09-08 10:23:54 -0500 |
+|   0.1.0.0 | initial schema | eprothro    | 2016-09-08 09:23:54 -0500 |
++-----------+----------------+-------------+---------------------------+
+```
+
+#### Reporting the version status
+```
+cassie schema:status
+```
+```
++-----------+----------------+-------------+---------------------------+
+|  Version  | Description    |   Status    | Migration File            |
++-----------+----------------+-------------+---------------------------+
+| * 0.2.0.0 |  create users  | serverbot   | 2016-09-08 10:23:54 -0500 |
+|   0.1.0.0 | initial schema | eprothro    | 2016-09-08 09:23:54 -0500 |
++-----------+----------------+-------------+---------------------------+
+```
+
+### Schema Management
+
+The full schema is stored in `schema.cql`, this is recommended to be checked into source control.
+It is updated (with a full dump) after each migration, to maintain a truth-store for the schema when used with multiple developers.
+
+### Dump the schema
+
+```
+cassie schema:dump
+```
+```
+-- Dumping Cassandra schema (version 0.2.0.0)
+   - Writing to db/cassandra/schema.cql
+   - done
+-- done
+```
+
+### Drop the schema
+```
+cassie schema:drop
+```
+```
+-- Dropping 2 keyspaces
+   - Dropping 'my_app_development'
+   - done
+   - Dropping 'cassie_schema'
+   - done
+-- done
+```
+
+### Load the schema
+```
+cassie schema:load
+```
+```
+-- Loading Schema from db/cassandra/schema.cql
+   > Schema is now at version 0.2.0.0
+-- done
+```
+
+### Reset the schema
+
+```
+cassie schema:reset
+```
+```
+-- Dropping 2 keyspaces
+   - Dropping 'my_app_development'
+   - done
+   - Dropping 'cassie_schema'
+   - done
+-- done
+-- Loading Schema from db/cassandra/schema.cql
+   > Schema is now at version 0.2.0.0
+-- done
+```
+
+### Reset the schema and migrate
+
+This task reload the schema from the schema file, and then proceeds with incremental migrations up to the latest migration.
+
+```
+cassie migrate:reset
+```
+```
+-- Dropping 2 keyspaces
+   - Dropping 'cassie_development'
+   - done
+   - Dropping 'cassie_schema'
+   - done
+-- done
+-- Loading Schema from db/cassandra/schema.cql
+   > Schema is now at version 0.2.0.0
+-- done
+-- Migrating to version 0.2.1.0
+   - Migragting version 0.2.1.0 UP
+   - done (4.89 ms)
+-- done
+-- Dumping Cassandra schema (version 0.2.1.0)
+   - Writing to db/cassandra/schema.cql
+   - done
+-- done
+```
+
+### Architecture (`cassie` developers)
+
+#### Versions
+
+A `Version` is a container class for a `migration`. A version may be applied in the current database's schema, or not.
+
+There are multiple independent, but potentially related/overlapping collections of `versions`.
+
+One is `Cassie::Schema.applied_versions`, which is the set of versions that have been applied, in the past, to a Cassandra database.
+
+Another is `Cassie::Schema.local_versions`, which the set of versions represented by migration files found in the `migrations_directory`.
+
+Versions are migrated up or down, and then recorded or forgotten, respectively, in the database's configurable `<schema_keyspace>.<versions_table>` (`cassie_schema.versions` by default).
+
+#### Migrations
+
+Strictly speaking, the *version* is what is being migrated up or down. The `Migration` is the class defining code to execute `up` or `down`. The version object delegates implementation of the migration to the `Migration` object.
+
+This class embeds the version number in it, but the `Migration` object does not know about the concept of its version. A migration file is used to load a `Version`, which contains the `Migration` object.
+
+Cassie only expects that the version number emedded in the class name match the one embedded in the file name.
+
+This means you can change the description embedded in the migration file without having to rename the class.
+
+#### A Note on rollback
+
+Migrating down rolls back the state of the schema in the Cassandra database. The in-database schema history keeps track of what migrations have been applied and rolls them back in that order (as opposed to whatever order the files indicate). This ensures the following scenario is supported:
+
+* Given the following mirgations exist:
+  * 0.1.0.0
+  * 0.1.1.0
+* And both migrations have been executed.
+* When a migraiton is created for version `0.1.0.99`
+* And the schema is migrated with `cassie migrate 0.1.0.0`
+* Then the `down` method for `0.1.0.99` is NOT executed
+
+* And then when the schema is migrated with `cassie migrate`, the `up` methods from the following migrations are executed:
+  * 0.1.0.99
+  * 0.1.1.0

data/lib/cassie/schema/apply_command.rb

@@ -0,0 +1,24 @@
+module Cassie::Schema
+  class ApplyCommand
+    attr_reader :version
+
+    def initialize(version)
+      @version = version
+    end
+
+    def direction
+      :up
+    end
+
+    def execute
+      version.migration.up
+      apply
+    end
+
+    protected
+
+    def apply
+      Cassie::Schema.record_version(version)
+    end
+  end
+end

data/lib/cassie/schema/cassandra_migrations/importer.rb

@@ -0,0 +1,91 @@
+require_relative 'migration_file'
+
+module Cassie::Schema
+  module CassandraMigrations
+    class Importer
+      # The source directory containing the +cassandra_migrations+ files.
+      # Defaults to +db/cassandra_migrate+
+      attr_accessor :source
+      # The version the schema will rest at after importing.
+      # Defaults to +0.1.0+ with a description of +Remove cassandra_migrations schema+
+      attr_accessor :final_version
+      # The migration files to be imported
+      attr_accessor :migration_files
+      # A callback fired before importing each migration
+      attr_accessor :before_each
+      # A callback fired after importing each migration
+      attr_accessor :after_each
+
+      def initialize(source_path=nil)
+        @source = source_path || default_source_path
+        @final_version = Cassie::Schema::Version.new("0.0.1.0", "Remove cassandra_migrations schema")
+        @migration_files = find_migration_files
+        @before_each      = Proc.new{}
+        @after_each       = Proc.new{}
+      end
+
+      def import
+        new_version = initial_version
+        new_version.executor = "cassandra_migrations"
+        new_version.executed_at = Time.now
+
+        migration_files.each do |old_migration_file|
+          before_each.call(old_migration_file)
+          new_version.id = Cassandra::TimeUuid::Generator.new.now
+          new_version.description = old_migration_file.description.humanize
+          writer = VersionWriter.new(new_version)
+          # new_version will automatically find
+          # the new cassie::schema::migration
+          # that is built from the old file
+          writer.migration_contents = old_migration_file.build_migration_class(new_version)
+
+          writer.write
+
+          Cassie::Schema.record_version(new_version, false)
+          after_each.call(new_version)
+
+          new_version = new_version.next
+        end
+      end
+
+      protected
+
+      def default_source_path
+        "db/cassandra_migrate"
+      end
+
+      def find_migration_files
+        paths = Dir.glob("#{absolute_source}/#{migration_template}")
+        paths = Dir.glob("#{relative_source}/#{migration_template}") if paths.empty?
+        raise_files_not_found if paths.empty?
+
+        paths.map{ |path| MigrationFile.new(path) }
+      end
+
+      def absolute_source
+        source
+      end
+
+      def relative_source
+        File.join(root, source)
+      end
+
+      def root
+        Dir.pwd
+      end
+
+      def migration_template
+        # 20161206214301_initial_database.rb
+        "[0-9]*_*.rb"
+      end
+
+      def raise_files_not_found
+        raise "No Cassandra Migration files were found. Looked in #{absolute_source} and #{relative_source}. Check the path and try again."
+      end
+
+      def initial_version
+        Cassie::Schema::Version.new("0.0.0.1")
+      end
+    end
+  end
+end

data/lib/cassie/schema/cassandra_migrations/migration_file.rb

@@ -0,0 +1,51 @@
+module Cassie::Schema
+  module CassandraMigrations
+    class MigrationFile
+
+      attr_reader :filename, :source, :description
+
+      def initialize(filename)
+        @filename = filename
+        @description = parse_description
+      end
+
+      # Builds a Cassie::Schema::Migration
+      # from the CassandraMigrations migration file
+      #
+      # @return [String] source string of new migration class
+      def build_migration_class(version)
+        @source = load_source
+        redefine_class(version.migration_class_name)
+        define_new_migration_class
+        version.migration_class_name.constantize
+        @source
+      ensure
+        @source = nil #free for GC
+      end
+
+      protected
+
+      def parse_description
+        matches = File.basename(filename).match(/[0-9]*_(.*).rb$/).captures
+        matches.first
+      end
+
+      def load_source
+        File.read(filename)
+      end
+
+      def redefine_class(name)
+        class_def = "class #{name} < Cassie::Schema::Migration"
+        # class UserMigration < CassandraMigrations::Migration
+        @source.sub!(/class\s.*<\s*CassandraMigrations::Migration/, class_def)
+      end
+
+      def define_new_migration_class
+        # define class in context of Object
+        # as a migration file does, rather
+        # than in the current context
+        Object.class_eval(@source)
+      end
+    end
+  end
+end

data/lib/cassie/schema/configuration.rb

@@ -0,0 +1,35 @@
+require 'pathname'
+
+module Cassie::Schema
+  # Extend a module/class with Configuration to enable migration management
+  module Configuration
+
+    # The keyspace in which to store Cassie schema data
+    attr_accessor :schema_keyspace
+    # The table in which to store Cassie schema applied versions data
+    attr_accessor :versions_table
+
+    # @!visibility private
+    def self.extended(extender)
+      extender.paths["schema_file"] = "db/cassandra/schema.cql"
+      extender.paths["migrations_directory"] = "db/cassandra/migrations"
+      extender.schema_keyspace = "cassie_schema"
+      extender.versions_table = "versions"
+    end
+
+    # Paths used for configuration loading.
+    #
+    # @return [Hash]
+    #   * +:schema_file+ - The .cql file defining the current schema structure
+    #   * +:migrations_directory+ - The directory containing the versioned schema migration .rb files
+    def paths
+      @paths ||= {}.with_indifferent_access
+    end
+
+    protected
+
+    def root
+      Pathname.new(Dir.pwd)
+    end
+  end
+end

data/lib/cassie/schema/migration/cassandra_support.rb

@@ -0,0 +1,34 @@
+module Cassie::Schema
+  class Migration
+    require_relative 'dsl'
+
+    module CassandraSupport
+      extend ActiveSupport::Concern
+
+      included do
+        include Cassie::Connection
+        include DSL
+      end
+
+      def execute(*params)
+        session.execute(*params)
+      end
+
+      # compatibility with casandra_migrations
+      # until import handles this
+      def using_keyspace(val)
+        raise ArgumentError, "block required for using temporary keyspace" unless block_given?
+        orignal = @keyspace if defined?(@keyspace)
+        @keyspace = val
+
+        yield
+
+        if defined?(orignal)
+          @keyspace = original
+        else
+          remove_instance_variable(:@keyspace)
+        end
+      end
+    end
+  end
+end

data/lib/cassie/schema/migration/dsl/announcing.rb

@@ -0,0 +1,47 @@
+module Cassie::Schema::Migration::DSL
+  module Announcing
+    extend ActiveSupport::Concern
+
+    module ClassMethods
+      def announcing_stream
+        return @announcing_stream if defined?(@announcing_stream)
+        $stdout
+      end
+
+      def announcing_stream=(val)
+        @announcing_stream = val
+      end
+    end
+
+    def announcing_stream
+      self.class.announcing_stream
+    end
+
+    def announce(msg)
+      announcing_stream << msg
+    end
+
+    protected
+
+    # Generates output labeled with name of migration and a line that goes up
+    # to 75 characters long in the terminal
+    def announce_migration(message)
+      text = "#{name}: #{message}"
+      length = [0, 75 - text.length].max
+
+      announce("== %s %s" % [text, "=" * length])
+    end
+
+    def announce_operation(message)
+      announce("  " + message)
+    end
+
+    def announce_suboperation(message)
+      announce("  -> " + message)
+    end
+
+    def name
+      version.description || version.number
+    end
+  end
+end

data/lib/cassie/schema/migration/dsl/column_operations.rb

@@ -0,0 +1,42 @@
+require_relative 'table_definition'
+
+module Cassie::Schema::Migration::DSL
+    # Module grouping methods used in migrations to make table operations like:
+    # - adding/removing columns
+    # - changing column types
+    # - renaming columns
+  module ColumnOperations
+    # Adds a column to a table.
+    #
+    # options: same options you would pass to create a table with that column
+    # (i.e. :limit might be applicable)
+
+    def add_column(table_name, column_name, type, options = {})
+      table_definition = TableDefinition.new
+
+      if !table_definition.respond_to?(type)
+        raise Errors::MigrationDefinitionError("Type '#{type}' is not valid for cassandra migration.")
+      end
+
+      table_definition.send(type, column_name, options)
+
+      announce_operation "add_column(#{column_name}, #{type})"
+
+      cql =  "ALTER TABLE #{table_name} ADD "
+      cql << table_definition.to_add_column_cql
+      announce_suboperation cql
+
+      execute cql
+    end
+
+    # Removes a column from the table
+    def remove_column(table_name, column_name)
+      announce_operation "drop_table(#{table_name})"
+
+      cql =  "ALTER TABLE #{table_name} DROP #{column_name}"
+      announce_suboperation cql
+
+      execute cql
+    end
+  end
+end