data_migrate 8.3.0 → 11.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fbfb52ef4fcacc01bb90162dd1869dddf02d0f94fcabe7b3df3b6e23b0616bc6
-  data.tar.gz: da2ca8153fc78c898839a8309a242b96d84c656d33e28213f7e991118d068ca3
+  metadata.gz: 20f45c0a409f5eea659d56b8118d9625e3358cdd9d199978b56359d3416abe64
+  data.tar.gz: f553fd264b97b71671556ece98e9c79830223e960e7f41a46e383587a7b728c3
 SHA512:
-  metadata.gz: 9a1e6f412e15f2e597ebf7b6272dcc7e66f2aa3faeea4f8ff48e1991667d238bcaa33e1ff12d007fdf58ff7005a380c03f85babc7e39bd4ca205b6f522d091b0
-  data.tar.gz: 6555ff6611bc06d9c85b137d301a319b3a5f61974c92b14592b9cede75414ba8d02745808c179bd0dcc2126415ffd2ed2edf9a3d836423e676a51d2a53b4dc83
+  metadata.gz: 8ac579c9ca5323c12e413f4c2ad228f9f36cbeb7d84624ff6fa24160f350b5e91f3ca7a8f18e469ce8e25dd4390a9d76316f40257dd9ef5bf4324c60309dd55f
+  data.tar.gz: f2fe12d32734ba839ce736d6634d2e794aad151b2719ff6e4a9bf1c9a05ccfdc4ab2a2de6bed8636a9423e89d6dddf62442ccf2f4c91ede15cf66736f36ff64f

data/Changelog.md

@@ -1,5 +1,97 @@
 # Changelog
 
+# 11.3.0
+- Make table_name configurable https://github.com/ilyakatz/data-migrate/pull/361
+- Use lease_connection over deprecated connection for rails 8 https://github.com/ilyakatz/data-migrate/pull/353
+- Add Ruby 3.4 to CI matrix
+
+# 11.2.0
+- Remove committed Gemfile.lock, reduce bundled file list when running `gem install` https://github.com/ilyakatz/data-migrate/pull/351
+- [Bump actionpack from 7.1.3.4 to 7.1.4.1](https://github.com/ilyakatz/data-migrate/pull/348)
+- [Bump rexml from 3.3.6 to 3.3.9](https://github.com/ilyakatz/data-migrate/pull/349)
+- Fix db_config_with_versions arity change and backport https://github.com/ilyakatz/data-migrate/pull/337
+
+# 11.1.0
+- Allow multiple data migration paths https://github.com/ilyakatz/data-migrate/pull/331
+- Fix db:prepare:with_data task on [Rails 7.2](https://github.com/ilyakatz/data-migrate/pull/339)
+- Fix ConnectionPool deprecation warnings on [Rails 7.2](https://github.com/ilyakatz/data-migrate/pull/341)
+- Add Rails 8.0 to testing grounds https://github.com/ilyakatz/data-migrate/pull/341
+- [Bump webrick from 1.8.1 to 1.8.2](https://github.com/ilyakatz/data-migrate/pull/345)
+
+# 11.0.0
+- [Update rexml to 3.3.6](https://github.com/ilyakatz/data-migrate/pull/329)
+- Fixes a bug which ignored migrations on [Rails 7.1 and higher](https://github.com/ilyakatz/data-migrate/pull/326)
+
+# 11.0.0rc
+- Remove Ruby 3.0 from build matrix
+- Support Rails 7.2.0 https://github.com/ilyakatz/data-migrate/pull/312
+- Update gemfile.lock builds
+
+## 9.4.2
+- Fix db:prepare:with_data task
+
+## 9.4.1
+- Add db:prepare task
+
+## 9.4.0
+- Reset model schema cache before each data migration https://github.com/ilyakatz/data-migrate/pull/307
+- Run load_config rake task before db:migrate:with_data https://github.com/ilyakatz/data-migrate/pull/308
+
+## 9.3.0
+- Improve with_data Rake task for multiple database https://github.com/ilyakatz/data-migrate/pull/296
+
+## 9.2.0
+- Support Rails 7.1 https://github.com/ilyakatz/data-migrate/pull/278
+- Build and test against 7.1.0.rc1 https://github.com/ilyakatz/data-migrate/pull/286
+
+## 9.1.0
+
+- Fix a bug that caused `schema_sha1` in `ar_internal_metadata` to be reset to the `data_schema.rb` file. (#272)
+- Remove the need for empty data_schema files for non-primary databases. (#273)
+
+## [YANKED] 10.0.3.rc
+
+- Remove all travis references [leoarnold](https//:github.com/leoarnold)
+- Changing to rc because of ongoing discussion how to properly handle multiple database environments
+
+## [YANKED] 10.0.2
+
+Change "rails" dependencies to "railties"
+
+## [YANKED] 10.0.1
+
+- Bug fix for Rails 6 config [chaunce](https//:github.com/chaunce)
+- Railties bug fix by [opti](https://github.com/opti)
+
+## [YANKED] 10.0.0
+
+Releasing 10.0.0
+
+!!! Breaking changes !!!
+
+- This version introduces a breaking change which may lead to undesired
+behavior in multi-database environments. See https://github.com/ilyakatz/data-migrate/issues/181
+
+## [YANKED] 10.0.0.rc1
+
+- Changes by [chaunce](https//:github.com/chaunce)
+- Multiple databases support
+- Refactor to clean things up
+- Deprecate rails 5.2 support for real
+
+## 9.0.0
+
+Ruby 3.2 support [mehanoid](https://github.com/mehanoid)
+Rails 5.2 is no longer supported
+
+## 8.5.0
+
+Allow custom templates [bazay](https://github.com/bazay)
+
+## 8.4.0
+
+Avoid Globally Accessible Functions for All Rake Tasks [berniechiu](https://github.com/berniechiu)
+
 ## 8.3.0
 
 Add delegation to exists? for use by third parties [foxondo](https://github.com/foxondo)

data/README.md

@@ -2,7 +2,7 @@
 
 - [![Version](http://img.shields.io/gem/v/data_migrate.svg?style=flat-square)](https://rubygems.org/gems/data_migrate)
 - [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat-square)](http://opensource.org/licenses/MIT)
-- [![Travis](https://img.shields.io/travis/ilyakatz/data-migrate.svg)](https://travis-ci.org/ilyakatz/data-migrate)
+- ![.github/workflows/build.yml](https://github.com/ilyakatz/data-migrate/actions/workflows/.github/workflows/build.yml/badge.svg)
 - [![Reviewed by Hound](https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg)](https://houndci.com)
 
 Run data migrations alongside schema migrations.
@@ -12,7 +12,7 @@ migrations, except they should be reserved for data migrations. For
 instance, if you realize you need to titleize all your titles, this
 is the place to do it.
 
-![Travis](./screenshot.png)
+![directory tree example](./screenshot.png)
 
 ## Why should I use this?
 
@@ -36,8 +36,13 @@ table to track all migrations.
 
 ## Rails Support
 
-Support Rails 5.2 through 7.0
+Support Rails 6.1 through 8.0
 
+For **Rails 6.0** support, please use gem version `9.1.x`:
+
+```ruby
+gem 'data_migrate', '~> 9.1.0'
+```
 
 #### v1
 
@@ -86,6 +91,7 @@ You can generate a data migration as you would a schema migration:
     rake db:migrate:status:with_data               # Display status of data and schema migrations
     rake db:migrate:up:with_data                   # Runs the "up" for a given migration VERSION
     rake db:migrate:with_data                      # Migrate the database data and schema (options: VERSION=x, VERBOSE=false)
+    rake db:prepare:with_data                      # Runs setup if database does not exist, or runs data and schema migrations if it does
     rake db:rollback:with_data                     # Rolls the schema back to the previous version (specify steps w/ STEP=n)
     rake db:schema:load:with_data                  # Load both schema.rb and data_schema.rb file into the database
     rake db:structure:load:with_data               # Load both structure.sql and data_schema.rb file into the database
@@ -96,7 +102,9 @@ Tasks work as they would with the 'vanilla' db version. The 'with_data' addition
 
 With 'up' and 'down', you can specify the option 'BOTH', which defaults to false. Using true, will migrate both the data and schema (in the desired direction) if they both match the version provided. Again, going up, schema is given precedence. Down its data.
 
-`rake db:migrate:status:with_data` provides and additional column to indicate which type of migration.
+When using `rake db:migrate:with_data` migrations will be run in ascending order by their version. For example, if you have a data migration with version 20230410000000 and a schema migration with version 20230415000000, expect the data migration to run first.
+
+`rake db:migrate:status:with_data` provides an additional column to indicate which type of migration.
 
 ### Configuration
 
@@ -107,7 +115,9 @@ You can override this setting in `config/initializers/data_migrate.rb`
 
 ```ruby
 DataMigrate.configure do |config|
+  config.data_migrations_table_name = 'my_migrations_database_name'
   config.data_migrations_path = 'db/awesomepath/'
+  config.data_template_path = Rails.root.join("lib", "awesomepath", "custom_data_migration.rb")
   config.db_configuration = {
     'host' => '127.0.0.1',
     'database' => 'awesome_database',
@@ -135,19 +145,12 @@ From now on capistrano will run `rake db:migrate:with_data` in every deploy.
 ## Rails Engines support
 
 This gem also has a initial support for adding data migrations inside Rails engines.
-Inside the Engine's class initializer (the one that inherits from `Rails::Engine`, usually inside `engines/ENGINE_NAME/lib/engine.rb`) you need to add something like this:
-
+Just add your engines directory pattern to data_migrations initializer, for example
+in the case your engines are located in `engines` folder you can set it up like this:
 
 ```ruby
-module EngineName
-  class Engine < ::Rails::Engine
-    initializer :engine_name do |app|
-      ::DataMigrate.configure do |data_migrate|
-        default_path = ::DataMigrate::Config.new.data_migrations_path
-        data_migrate.data_migrations_path = [default_path, root.join('db', 'data')]
-      end
-    end
-  end
+DataMigrate.configure do |config|
+  config.data_migrations_path = ['db/data'] + Dir['engines/**/db/data']
 end
 ```
 
@@ -160,12 +163,22 @@ Then, in the Engine's `db/data` folder, you can add data migrations and run them
 Run tests for a specific version of Rails
 
 ```
-bundle exec appraisal install
-bundle exec appraisal rails-5.1 rspec
-bundle exec appraisal rails-5.2 rspec
-bundle exec appraisal rails-6.0 rspec
+bundle exec appraisal rails-6.1 rspec
+bundle exec appraisal rails-7.0 rspec
+bundle exec appraisal rails-7.1 rspec
+bundle exec appraisal rails-7.2 rspec
+bundle exec appraisal rails-8.0 rspec
 ```
 
+## Releasing new version
+
+1. Update version.rb file, run `bundle exec appraisal` to update the version in corresponding gemfile.lock
+2. Create a new tag, eg `git tag 9.4.1`
+3. Go to https://github.com/ilyakatz/data-migrate/tags
+4. Click "Create release" under 9.4.1
+5. CLick "Generate release notes"
+6. Click "Publish release"
+
 ## Thanks
 
 [Andrew J Vargo](http://github.com/ajvargo) Andrew was the original creator and maintainer of this project!

data/lib/data_migrate/config.rb

@@ -12,12 +12,22 @@ module DataMigrate
   end
 
   class Config
-    attr_accessor :data_migrations_path, :db_configuration, :spec_name
+    attr_accessor :data_migrations_table_name, :data_migrations_path, :data_template_path, :db_configuration, :spec_name
+
+    DEFAULT_DATA_TEMPLATE_PATH = "data_migration.rb"
 
     def initialize
+      @data_migrations_table_name = "data_migrations"
       @data_migrations_path = "db/data/"
+      @data_template_path = DEFAULT_DATA_TEMPLATE_PATH
       @db_configuration = nil
       @spec_name = nil
     end
+
+    def data_template_path=(value)
+      @data_template_path = value.tap do |path|
+        raise ArgumentError, "File not found: '#{path}'" unless path == DEFAULT_DATA_TEMPLATE_PATH || File.exist?(path)
+      end
+    end
   end
 end

data/lib/data_migrate/{data_migrator_five.rb → data_migrator.rb}

@@ -5,33 +5,20 @@ require "data_migrate/config"
 
 module DataMigrate
   class DataMigrator < ActiveRecord::Migrator
-    def self.migrations_paths
-      [DataMigrate.config.data_migrations_path]
-    end
-
-    def self.assure_data_schema_table
-      ActiveRecord::Base.establish_connection(db_config)
-      DataMigrate::DataSchemaMigration.create_table
-    end
-
-    def initialize(direction, migrations, target_version = nil)
-      @direction         = direction
-      @target_version    = target_version
-      @migrated_versions = nil
-      @migrations        = migrations
-
-      validate(@migrations)
-
-      DataMigrate::DataSchemaMigration.create_table
-      ActiveRecord::InternalMetadata.create_table
-    end
-
     def load_migrated
       @migrated_versions =
-        DataMigrate::DataSchemaMigration.normalized_versions.map(&:to_i).sort
+        DataMigrate::RailsHelper.data_schema_migration.normalized_versions.map(&:to_i).sort
     end
 
     class << self
+      def migrations_paths
+        Array.wrap(DataMigrate.config.data_migrations_path)
+      end
+
+      def create_data_schema_table
+        DataMigrate::RailsHelper.data_schema_migration.create_table
+      end
+
       def current_version
         DataMigrate::MigrationContext.new(migrations_paths).current_version
       end
@@ -61,28 +48,21 @@ module DataMigrate
 
       # TODO: this was added to be backward compatible, need to re-evaluate
       def migrations(_migrations_paths)
-        #DataMigrate::MigrationContext.new(migrations_paths).migrations
         DataMigrate::MigrationContext.new(_migrations_paths).migrations
       end
 
       #TODO: this was added to be backward compatible, need to re-evaluate
       def run(direction, migration_paths, version)
+        # Ensure all Active Record model cache is reset for each data migration
+        # As recommended in: https://github.com/rails/rails/blob/da21c2e9812e5eb0698fba4a9aa38632fc004432/activerecord/lib/active_record/migration.rb#L467-L470
+        ActiveRecord::Base.descendants.each(&:reset_column_information)
+
         DataMigrate::MigrationContext.new(migration_paths).run(direction, version)
       end
 
       def rollback(migrations_path, steps)
         DataMigrate::MigrationContext.new(migrations_path).rollback(steps)
       end
-
-      def db_config
-        env = Rails.env || "development"
-        ar_config = if (Rails::VERSION::MAJOR == 6 && Rails::VERSION::MINOR >= 1) || Rails::VERSION::MAJOR > 6
-                      ActiveRecord::Base.configurations.configs_for(env_name: env).first
-                    else
-                      ActiveRecord::Base.configurations[env]
-                    end
-        ar_config || ENV["DATABASE_URL"]
-      end
     end
 
     private
@@ -90,10 +70,10 @@ module DataMigrate
     def record_version_state_after_migrating(version)
       if down?
         migrated.delete(version)
-        DataMigrate::DataSchemaMigration.where(version: version.to_s).delete_all
+        DataMigrate::RailsHelper.data_schema_delete_version(version.to_s)
       else
         migrated << version
-        DataMigrate::DataSchemaMigration.create!(version: version.to_s)
+        DataMigrate::RailsHelper.data_schema_migration.create_version(version.to_s)
       end
     end
   end

data/lib/data_migrate/data_schema.rb

@@ -9,7 +9,7 @@ module DataMigrate
     #   ActiveRecord::ConnectionAdapters::SchemaStatements
     #     #assume_migrated_upto_version
     def define(info)
-      DataMigrate::DataMigrator.assure_data_schema_table
+      DataMigrate::DataMigrator.create_data_schema_table
 
       return if info[:version].blank?
 
@@ -29,13 +29,13 @@ module DataMigrate
     end
 
     def versions
-      @versions ||= begin
-        versions = []
-        Dir.foreach(DataMigrate::DataMigrator.full_migrations_path) do |file|
-          match_data = DataMigrate::DataMigrator.match(file)
-          versions << match_data[1].to_i if match_data
+      @versions ||= Set.new.tap do |versions|
+        DataMigrate::DataMigrator.migrations_paths.each do |path|
+          Dir.foreach(path) do |file|
+            match_data = DataMigrate::DataMigrator.match(file)
+            versions << match_data[1].to_i if match_data
+          end
         end
-        versions
       end
     end
 
@@ -57,7 +57,7 @@ module DataMigrate
     end
 
     def table_name
-      DataMigrate::DataSchemaMigration.table_name
+      DataMigrate::RailsHelper.data_schema_migration.table_name
     end
   end
 end

data/lib/data_migrate/data_schema_migration.rb

@@ -1,12 +1,29 @@
 module DataMigrate
-  class DataSchemaMigration
-    class << self
-      delegate :table_name, :primary_key, :create_table, :normalized_versions, :create, :create!, :table_exists?, :exists?, :where, to: :instance
+  class DataSchemaMigration < ActiveRecord::SchemaMigration
+    # In Rails 7.1+, ActiveRecord::SchemaMigration methods are instance methods
+    # So we only load the appropriate methods depending on Rails version.
+    if DataMigrate::RailsHelper.rails_version_equal_to_or_higher_than_7_1
+      def table_name
+        ActiveRecord::Base.table_name_prefix + DataMigrate.config.data_migrations_table_name + ActiveRecord::Base.table_name_suffix
+      end
+
+      def primary_key
+        "version"
+      end
+    else
+      class << self
+        def table_name
+          ActiveRecord::Base.table_name_prefix + DataMigrate.config.data_migrations_table_name + ActiveRecord::Base.table_name_suffix
+        end
+
+        def primary_key
+          "version"
+        end
 
-      def instance
-        @instance ||= Class.new(::ActiveRecord::SchemaMigration) do
-          define_singleton_method(:table_name) { ActiveRecord::Base.table_name_prefix + 'data_migrations' + ActiveRecord::Base.table_name_suffix }
-          define_singleton_method(:primary_key) { "version" }
+        def create_version(version)
+          # Note that SchemaMigration.create_version in Rails 7.1 does not
+          # raise an error if validations fail but we retain this behaviour for now.
+          create!(version: version)
         end
       end
     end

data/lib/data_migrate/database_configurations_wrapper.rb

@@ -0,0 +1,11 @@
+module DataMigrate
+  # This wrapper is used to differentiate between
+  # a data and schema db config when running migrations
+  class DatabaseConfigurationWrapper
+    attr_reader :db_config
+
+    def initialize(db_config)
+      @db_config = db_config
+    end
+  end
+end