data_migrate 3.0.1 → 4.0.0

data/.travis.yml

@@ -0,0 +1,16 @@
+language: ruby
+rvm:
+  - 2.2.6
+  - 2.3.4
+script: bundle exec rspec
+gemfile:
+  - gemfiles/rails_4.2.gemfile
+  - gemfiles/rails_5.0.gemfile
+  - gemfiles/rails_5.1.gemfile
+  - gemfiles/rails_5.2.gemfile
+matrix:
+  exclude:
+    - rvm: 2.0.0-p648
+      gemfile: gemfiles/rails_5.0.gemfile
+    - rvm: 2.1.10
+      gemfile: gemfiles/rails_5.0.gemfile

data/Appraisals

@@ -0,0 +1,17 @@
+appraise 'rails-4.2' do
+  gem 'rails', '4.2.8'
+  # Nokogiri 1.7+ requires Ruby 2.1+
+  gem 'nokogiri', '1.6.8.1'
+end
+
+appraise 'rails-5.0' do
+  gem 'rails', '5.0.3'
+end
+
+appraise 'rails-5.1' do
+  gem 'rails', '5.1'
+end
+
+appraise 'rails-5.2' do
+  gem 'rails', '~> 5.2.0'
+end

data/Changelog.md

@@ -1,6 +1,41 @@
 Changelog
 =========
 
+## 4.0.0
+
+Support for Rails 5.2
+Deprecated support for Rails 4.1
+Internal changes to make data-migrate behavior more similar to Rails migrations
+
+## 3.5.0
+
+Deprecated support for rails 4.0
+Improvements to timestamped migrations, thanks to [Pierre-Michard](https://github.com/Pierre-Michard)
+
+## 3.4.0
+`rake data:migrate:status` to return result in chronological order
+## 3.3.1
+Regression fix, thanks to [subakva](https://github.com/subakva)
+
+## 3.3.0
+The concept of schema:dump to data migrations, thanks to
+[tobyndockerill](https://github.com/tobyndockerill)
+
+## 3.2.1
+
+data_migrate table into rails schema dump, thanks to
+[jturkel](https://github.com/jturkel) 
+
+
+## 3.2.0
+- Add support for Rails 5.1
+- No longer testing EOL rubies
+
+## 3.1.0
+
+Rails 5.0 support thanks to
+[jturkel](https://github.com/jturkel) and [abreckner](https://github.com/abreckner)
+
 ## 3.0.1
 
 ([gacha](https://github.com/gacha)) Capistrano fixes

data/Gemfile.rails5

@@ -0,0 +1,5 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in data_migrate.gemspec
+gemspec
+gem 'rails', '~> 5.0.0'

data/Gemfile.rails5.1

@@ -0,0 +1,5 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in data_migrate.gemspec
+gemspec
+gem 'rails', '~> 5.1'

data/Gemfile.rails5.2

@@ -0,0 +1,5 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in data_migrate.gemspec
+gemspec
+gem 'rails', '~> 5.2'

data/README.md

@@ -3,6 +3,8 @@ Data Migrate
 
 - [![Version](http://img.shields.io/gem/v/data_migrate.svg?style=flat-square)](https://rubygems.org/gems/data_migrate)
 - [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat-square)](http://opensource.org/licenses/MIT)
+- [![Travis](https://img.shields.io/travis/ilyakatz/data-migrate.svg)](https://travis-ci.org/ilyakatz/data-migrate)
+
 
 Run data migrations alongside schema migrations.
 
@@ -11,8 +13,9 @@ migrations, except they should be reserved for data migrations. For
 instance, if you realize you need to titleize all your titles, this
 is the place to do it.
 
-Why should I use this?
-----------------------
+![Travis](./screenshot.png)
+
+# Why should I use this?
 
 Its seems when a project hits a certain size, I get to manipulate data
 outside the application itself.  Changing defaults, new validations,
@@ -25,36 +28,37 @@ migration.  It'd be much better to keep concerns separate. The benefit
 of having them separate has to do with your data model.
 
 For instance, lets take an absurd example, to illustrate: You have
-your infamous [Rails blog](http://media.rubyonrails.org/video/rails-0-5.mov)
-that has posts with many comments.  After some use, you decide you are
-going to be a trend setter, and want only one comment per post, and
-just the text. "Frist!!1!1" rules the day. Given that you:
-- write a migration to add a comment column to Post
-- write a migration to move the contents of the first comments to the Post
-- drop the column_id column from Post
-- drop the Comment model
-- fix all your test/controller/view mojo.
+a blog application that has posts with many comments.
+After some use, you decide you are going to be a trend setter,
+and want only one comment per post, and just the text.
+
+Given that you:
+- write a migration to add a comment column to `Post`
+- write a migration to move the contents of the first comments to the `Post`
+- drop the `column_id` column from `Post`
+- drop the `Comment` model
+- fix all your tests
 
 You've just got bit.  When you `rake setup:development`, the mess gets
 mad at you after it creates your database, and starts cranking through
 migrations.  It gets to the part where you iterate over the comments
-and it blows up.  You don't have a comment model anymore for it to
+and it blows up.  You don't have a `Comment` model anymore for it to
 even try and get 'all' from.  You think you are smarter, and wrap the
-AR call in a conditional based on the environment. That's fine until
-you get that QA gal, and she wants her own thing. Then the UI people
-get tired of waiting for the full stack to load on page refreshes, so
-you have to edit past migrations...
+ActiveRecord call in a conditional based on the environment.
+
+That's fine until you get that QA gal, and she wants her own thing.
+Then the UI people get tired of waiting for the full stack to load on page
+refreshes, so you have to edit past migrations...
 
 With Data Migrate, you have the control.  You can generate your
 migrations as schema or data as you would as your work flow. For
 setting tasks that don't require any intermediate AR activity, like
-dev and test, you stick with db:migrate.  For your prod, and qa, you
+dev and test, you stick with `db:migrate`.  For production and QA, you
 change their scripts to `db:migrate:with_data`.  Of course you want to
 test your migration, so you have the choice of `db:migrate:with_data` or
 `data:migrate` to just capture that data change.
 
-What's it do?
--------------
+## What does it do?
 
 Data migrations are stored in `db/data`. They act like schema
 migrations, except they should be reserved for data migrations. For
@@ -63,16 +67,13 @@ is the place to do it. Running any of the provided rake tasks also
 creates a data schema table to mirror the usual schema migrations
 table to track all the goodness.
 
-Rails Support
---------------------
+## Rails Support
 
-Rails 3.1: Version 1.2 supports Rails 3.1.0 and higher **but** is no longer maintained.
+Support Rails 4.2 through 5.2
 
-Rails 4: Version 2.0 supports Rails 4.0 and higher
+### Important notes for older versions
 
-Rails 5: Not tested
-
-### Important note
+#### v2
 
 If you upgraded to Rails 4 while using `data_migrate` prior to version 2,
 the gem wrote data migration versions into
@@ -81,8 +82,12 @@ the gem wrote data migration versions into
 
 This may cause some unintended consequences. See [#22](https://github.com/ilyakatz/data-migrate/issues/22)
 
-Installation
-------------
+#### v1
+
+If you've installed previous to v1.1.0, you'll want to delete the
+`create_data_migrations_table` migration.
+
+## Installation
 Add the gem to your project
 
     # Gemfile
@@ -91,13 +96,11 @@ Add the gem to your project
 Then `bundle install` and you are ready to go.
 
 So you know, when you use one of the provide rake tasks, a table
-called 'data_migrations' will be created in your database. This
-is to mirror the way the standard 'db' rake tasks work. If you've
-installed previous to v1.1.0, you'll want to delete the
-'create\_data\_migrations_table' migration.
+called `data_migrations` will be created in your database. This
+is to mirror the way the standard `db` rake tasks work.
 
-Usage
------
+
+## Usage
 
 ### Generating Migrations
 
@@ -108,31 +111,35 @@ You can generate a data migration as you would a schema migration:
 ### Rake Tasks
 
     $> rake -T data
+    rake data:dump                    # Create a db/data_schema.rb file that stores the current data version
     rake data:forward                 # Pushes the schema to the next version (specify steps w/ STEP=n)
     rake data:migrate                 # Migrate data migrations (options: VERSION=x, VERBOSE=false)
     rake data:migrate:down            # Runs the "down" for a given migration VERSION
-    rake data:migrate:redo            # Rollbacks the database one migration and re migrate up (options: STEP=x, VERSIO...
+    rake data:migrate:redo            # Rollbacks the database one migration and re migrate up (options: STEP=x, VERSION=x)
     rake data:migrate:status          # Display status of data migrations
     rake data:migrate:up              # Runs the "up" for a given migration VERSION
     rake data:rollback                # Rolls the schema back to the previous version (specify steps w/ STEP=n)
     rake data:version                 # Retrieves the current schema version number for data migrations
     rake db:forward:with_data         # Pushes the schema to the next version (specify steps w/ STEP=n)
     rake db:migrate:down:with_data    # Runs the "down" for a given migration VERSION
-    rake db:migrate:redo:with_data    # Rollbacks the database one migration and re migrate up (options: STEP=x, VERSIO...
+    rake db:migrate:redo:with_data    # Rollbacks the database one migration and re migrate up (options: STEP=x, VERSION=x)
     rake db:migrate:status:with_data  # Display status of data and schema migrations
     rake db:migrate:up:with_data      # Runs the "up" for a given migration VERSION
     rake db:migrate:with_data         # Migrate the database data and schema (options: VERSION=x, VERBOSE=false)
     rake db:rollback:with_data        # Rolls the schema back to the previous version (specify steps w/ STEP=n)
+    rake db:schema:load:with_data     # Load both schema.rb and data_schema.rb file into the database
     rake db:version:with_data         # Retrieves the current schema version numbers for data and schema migrations
 
-Tasks work as they would with the 'vanilla' db version.  The 'with_data' addition to the 'db' tasks will run the task in the context of both the data and schema migrations.  That is, `rake db:rollback:with_data` will check to see if it was a schema or data migration invoked last, and do that.  Tasks invoked in that space also have an additional line of output, indicating if the action is performed on data or schema.
+Tasks work as they would with the 'vanilla' db version. The 'with_data' addition to the 'db' tasks will run the task in the context of both the data and schema migrations. That is, rake db:rollback:with_data will check to see if it was a schema or data migration invoked last, and do that. Tasks invoked in that space also have an additional line of output, indicating if the action is performed on data or schema.
 
 With 'up' and 'down', you can specify the option 'BOTH', which defaults to false. Using true, will migrate both the data and schema (in the desired direction) if they both match the version provided.  Again, going up, schema is given precedence. Down its data.
 
 `rake db:migrate:status:with_data` provides and additional column to indicate which type of migration.
+### Configuration
+
+`data_migrate` respects `ActiveRecord::Base.dump_schema_after_migration`. If it is set to `false`, data schema file will not be generated
 
-Capistrano Support
-------------------
+## Capistrano Support
 
 The gem comes with a capistrano task that can be used instead of `capistrano/rails/migrations`.
 
@@ -144,9 +151,19 @@ require 'capistrano/data_migrate'
 
 From now on capistrano will run `rake db:migrate:with_data` in every deploy.
 
+### Contributing
+
+## Testing
+
+Run tests for a specific version of Rails
+
+```
+bundle exec appraisal install
+bundle exec appraisal rails-4.2 rspec
+bundle exec appraisal rails-5.0 rspec
+```
 
-Thanks
-------
+## Thanks
 [Andrew J Vargo](http://github.com/ajvargo) Andrew was the original creator and maintainer of this project!
 
 [Jeremy Durham](http://jeremydurham.com/) for fleshing out the idea with me, and providing guidance.

data/data_migrate.gemspec

@@ -15,7 +15,18 @@ Gem::Specification.new do |s|
 
   s.rubyforge_project = "data_migrate"
 
-  s.add_dependency('rails', '>= 4.0')
+  s.add_dependency('rails', '>= 4.2')
+  s.add_development_dependency "appraisal"
+  s.add_development_dependency "rake"
+  s.add_development_dependency "rspec"
+  s.add_development_dependency "rspec-core"
+  s.add_development_dependency "pry"
+  s.add_development_dependency "rb-readline"
+  s.add_development_dependency "sqlite3"
+  s.add_development_dependency "timecop"
+  s.add_development_dependency "rubocop"
+  s.add_development_dependency "overcommit"
+
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")

data/gemfiles/rails_4.1.gemfile

@@ -0,0 +1,7 @@
+# This file was generated by Appraisal
+
+source "http://rubygems.org"
+
+gem "rails", "4.1.16"
+
+gemspec path: "../"

data/gemfiles/rails_4.2.gemfile

@@ -0,0 +1,8 @@
+# This file was generated by Appraisal
+
+source "http://rubygems.org"
+
+gem "rails", "4.2.8"
+gem "nokogiri", "1.6.8.1"
+
+gemspec path: "../"

data/gemfiles/rails_5.0.gemfile

@@ -0,0 +1,7 @@
+# This file was generated by Appraisal
+
+source "http://rubygems.org"
+
+gem "rails", "5.0.3"
+
+gemspec path: "../"

data/gemfiles/rails_5.1.gemfile

@@ -0,0 +1,7 @@
+# This file was generated by Appraisal
+
+source "http://rubygems.org"
+
+gem "rails", "5.1"
+
+gemspec path: "../"

data/gemfiles/rails_5.2.gemfile

@@ -0,0 +1,7 @@
+# This file was generated by Appraisal
+
+source "http://rubygems.org"
+
+gem "rails", "~> 5.2.0"
+
+gemspec path: "../"

data/lib/data_migrate/data_migrator.rb

@@ -1,4 +1,6 @@
-require 'active_record'
+# frozen_string_literal: true
+
+require "active_record"
 
 module DataMigrate
 
@@ -14,9 +16,14 @@ module DataMigrate
       end
     end
 
+
     class << self
+      def current_version(connection = ActiveRecord::Base.connection)
+        get_all_versions(connection).max || 0
+      end
+
       def get_all_versions(connection = ActiveRecord::Base.connection)
-        if connection.table_exists?(schema_migrations_table_name)
+        if table_exists?(connection, schema_migrations_table_name)
           # Certain versions of the gem wrote data migration versions into
           # schema_migrations table. After the fix, it was corrected to write into
           # data_migrations. However, not to break anything we are going to
@@ -33,20 +40,43 @@ module DataMigrate
       end
 
       def schema_migrations_table_name
-        ActiveRecord::Base.table_name_prefix + 'data_migrations' + ActiveRecord::Base.table_name_suffix
+        ActiveRecord::Base.table_name_prefix + "data_migrations" +
+          ActiveRecord::Base.table_name_suffix
       end
 
       def migrations_path
-        'db/data'
+        "db/data"
+      end
+
+      ##
+      # Provides the full migrations_path filepath
+      # @return (String)
+      def full_migrations_path
+        File.join(Rails.root, *migrations_path.split(File::SEPARATOR))
       end
 
       def assure_data_schema_table
-        config = ActiveRecord::Base.configurations[Rails.env || 'development'] || ENV["DATABASE_URL"]
-        ActiveRecord::Base.establish_connection(config)
+        ActiveRecord::Base.establish_connection(db_config)
         sm_table = DataMigrate::DataMigrator.schema_migrations_table_name
 
-        unless ActiveRecord::Base.connection.table_exists?(sm_table)
-          ActiveRecord::Base.connection.create_table(sm_table, :id => false) do |schema_migrations_table|
+        unless table_exists?(ActiveRecord::Base.connection, sm_table)
+          create_table(sm_table)
+        end
+      end
+
+      ##
+      # Compares the given filename with what we expect data migration
+      # filenames to be, eg the "20091231235959_some_name.rb" pattern
+      # @param (String) filename
+      # @return (MatchData)
+      def match(filename)
+        /(\d{14})_(.+)\.rb/.match(filename)
+      end
+
+      private
+
+      def create_table(sm_table)
+        ActiveRecord::Base.connection.create_table(sm_table, :id => false) do |schema_migrations_table|
             schema_migrations_table.column :version, :string, :null => false
           end
 
@@ -57,9 +87,24 @@ module DataMigrate
           ActiveRecord::Base.connection.add_index sm_table, :version,
             :unique => true,
             :name => index_name
+      end
+
+      def table_exists?(connection, table_name)
+        # Avoid the warning that table_exists? prints in Rails 5.0 due a
+        # change in behavior between Rails 5.0 and Rails 5.1 of this method
+        # with respect to database views.
+        if ActiveRecord.version >= Gem::Version.new("5.0") &&
+           ActiveRecord.version < Gem::Version.new("5.1")
+          connection.data_source_exists?(table_name)
+        else
+          connection.table_exists?(schema_migrations_table_name)
         end
       end
 
+      def db_config
+        ActiveRecord::Base.configurations[Rails.env || "development"] ||
+          ENV["DATABASE_URL"]
+      end
     end
   end
 end

data/lib/data_migrate/data_migrator_five.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+require "active_record"
+
+module DataMigrate
+  class DataMigrator < ActiveRecord::Migrator
+    self.migrations_paths = ["db/data"]
+
+    def self.assure_data_schema_table
+      DataMigrate::DataSchemaMigration.create_table
+    end
+
+    def initialize(direction, migrations, target_version = nil)
+      @direction         = direction
+      @target_version    = target_version
+      @migrated_versions = nil
+      @migrations        = migrations
+
+      validate(@migrations)
+
+      DataMigrate::DataSchemaMigration.create_table
+      ActiveRecord::InternalMetadata.create_table
+    end
+
+    def load_migrated
+      # Certain versions of the gem wrote data migration versions into
+      # schema_migrations table. After the fix, it was corrected to write into
+      # data_migrations. However, not to break anything we are going to
+      # get versions from both tables.
+      #
+      # This may cause some problems:
+      # Eg. rake data:versions will show version from the schema_migrations table
+      # which may be a version of actual schema migration and not data migration
+      @migrated_versions =
+        DataMigrate::DataSchemaMigration.normalized_versions.map(&:to_i).sort +
+        ActiveRecord::SchemaMigration.normalized_versions.map(&:to_i).sort
+    end
+
+    class << self
+      def current_version
+        DataMigrate::MigrationContext.new(migrations_paths).current_version
+      end
+
+      ##
+      # Compares the given filename with what we expect data migration
+      # filenames to be, eg the "20091231235959_some_name.rb" pattern
+      # @param (String) filename
+      # @return (MatchData)
+      def match(filename)
+        /(\d{14})_(.+)\.rb/.match(filename)
+      end
+
+      def migrations_status
+        DataMigrate::MigrationContext.new(migrations_paths).migrations_status
+      end
+
+      #TODO: this was added to be backward compatible, need to re-evaluate
+      def migrations(_migrations_paths)
+        DataMigrate::MigrationContext.new(migrations_paths).migrations
+      end
+
+      #TODO: this was added to be backward compatible, need to re-evaluate
+      def run(direction, migration_paths, version)
+        DataMigrate::MigrationContext.new(migration_paths).run(direction, version)
+      end
+
+      def rollback(migrations_path, steps)
+        DataMigrate::MigrationContext.new(migrations_path).rollback(steps)
+      end
+    end
+
+    private
+
+    def record_version_state_after_migrating(version)
+      if down?
+        migrated.delete(version)
+        DataMigrate::DataSchemaMigration.where(version: version.to_s).delete_all
+      else
+        migrated << version
+        DataMigrate::DataSchemaMigration.create!(version: version.to_s)
+      end
+    end
+  end
+end

data/lib/data_migrate/data_schema.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module DataMigrate
+  ##
+  # Provides the definition method for data_schema.rb
+  class Data < ActiveRecord::Schema
+    # This method is based on the following two methods
+    #   ActiveRecord::Schema#define
+    #   ActiveRecord::ConnectionAdapters::SchemaStatements
+    #     #assume_migrated_upto_version
+    def define(info)
+      DataMigrate::DataMigrator.assure_data_schema_table
+
+      return if info[:version].blank?
+
+      version = info[:version].to_i
+
+      unless migrated.include?(version)
+        execute "INSERT INTO #{sm_table} (version) VALUES ('#{version}')"
+      end
+
+      insert(version)
+    end
+
+    private
+
+    def migrated
+      @migrated ||= select_values("SELECT version FROM #{sm_table}").map(&:to_i)
+    end
+
+    def versions
+      @versions ||= begin
+        versions = []
+        Dir.foreach(DataMigrate::DataMigrator.full_migrations_path) do |file|
+          match_data = DataMigrate::DataMigrator.match(file)
+          versions << match_data[1].to_i if match_data
+        end
+        versions
+      end
+    end
+
+    def insert(version)
+      inserted = Set.new
+      (versions - migrated).each do |v|
+        if inserted.include?(v)
+          raise "Duplicate data migration #{v}. Please renumber your data " \
+            "migrations to resolve the conflict."
+        elsif v < version
+          execute "INSERT INTO #{sm_table} (version) VALUES ('#{v}')"
+          inserted << v
+        end
+      end
+    end
+
+    def sm_table
+      quote_table_name(table_name)
+    end
+
+    def table_name
+      DataMigrate::DataSchemaMigration.table_name
+    end
+  end
+end

data/lib/data_migrate/data_schema_migration.rb

@@ -1,5 +1,6 @@
 module DataMigrate
   class DataSchemaMigration < ::ActiveRecord::SchemaMigration
+
     class << self
       def table_name
         ActiveRecord::Base.table_name_prefix + 'data_migrations' + ActiveRecord::Base.table_name_suffix