data_migrate 9.4.1 → 11.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: aa16e421f2f5d046c5e78678b226574ddaeca7345bfcff5f8ab69091fb901f3c
-  data.tar.gz: 91d60c54724dc47e6aac6f7638275c005c0f27f052741c00fbed0f15fb8056b7
+  metadata.gz: 907383933e031d0a1da7bad6dd3a43807794c0abeb27313af27e59ed8d802f3a
+  data.tar.gz: 1ddf826962f7078972ca9a234a723646d7744a41990dbe38550155fc0276fca3
 SHA512:
-  metadata.gz: e0585dd9cc369cfbb980c609d1c7f1cc286a3109f799303d1518cca460f1e41b89df484154426e8e6798cd674f520241d19ab4bd2343bb0800e429de78b7d379
-  data.tar.gz: 52aeb1ce48bcf2644f8c4640b99d5dbb98bee62e10fa5f3a20fabff41eb3dc833c346c39b51245a128095eb75231d588ca734092056aedbff62bfa1df6512888
+  metadata.gz: e9e79dd56bda8e5f2afecb79f940eb17d7b5439c1d07b38d67943d58fc0ebaf159283d4478ac13925a2ee399097f6825b519052d3fea9f7b04c659a98647e75f
+  data.tar.gz: '0959a1bcf969cbc7834c839eca029d1416ff287ea2927a8ae0493a7880fb6910adea4baf5a2d162bd4cce35f80b610dc2f9d671cf77cf6583a699a96196871de'

data/Changelog.md

@@ -1,7 +1,33 @@
 # Changelog
 
+# 11.2.0
+
+- Remove committed Gemfile.lock, reduce bundled file list when running `gem install` https://github.com/ilyakatz/data-migrate/pull/351
+- [Bump actionpack from 7.1.3.4 to 7.1.4.1](https://github.com/ilyakatz/data-migrate/pull/348)
+- [Bump rexml from 3.3.6 to 3.3.9](https://github.com/ilyakatz/data-migrate/pull/349)
+- Fix db_config_with_versions arity change and backport https://github.com/ilyakatz/data-migrate/pull/337
+
+# 11.1.0
+- Allow multiple data migration paths https://github.com/ilyakatz/data-migrate/pull/331
+- Fix db:prepare:with_data task on [Rails 7.2](https://github.com/ilyakatz/data-migrate/pull/339)
+- Fix ConnectionPool deprecation warnings on [Rails 7.2](https://github.com/ilyakatz/data-migrate/pull/341)
+- Add Rails 8.0 to testing grounds https://github.com/ilyakatz/data-migrate/pull/341
+- [Bump webrick from 1.8.1 to 1.8.2](https://github.com/ilyakatz/data-migrate/pull/345)
+
+# 11.0.0
+- [Update rexml to 3.3.6](https://github.com/ilyakatz/data-migrate/pull/329)
+- Fixes a bug which ignored migrations on [Rails 7.1 and higher](https://github.com/ilyakatz/data-migrate/pull/326)
+
+# 11.0.0rc
+- Remove Ruby 3.0 from build matrix
+- Support Rails 7.2.0 https://github.com/ilyakatz/data-migrate/pull/312
+- Update gemfile.lock builds
+
+## 9.4.2
+- Fix db:prepare:with_data task
+
 ## 9.4.1
-  - Add db:prepare task
+- Add db:prepare task
 
 ## 9.4.0
 - Reset model schema cache before each data migration https://github.com/ilyakatz/data-migrate/pull/307

data/README.md

@@ -36,8 +36,13 @@ table to track all migrations.
 
 ## Rails Support
 
-Support Rails 6.1 through 7.1
+Support Rails 6.1 through 7.2
 
+For **Rails 6.0** support, please use gem version `9.1.x`:
+
+```ruby
+gem 'data_migrate', '~> 9.1.0'
+```
 
 #### v1
 
@@ -86,6 +91,7 @@ You can generate a data migration as you would a schema migration:
     rake db:migrate:status:with_data               # Display status of data and schema migrations
     rake db:migrate:up:with_data                   # Runs the "up" for a given migration VERSION
     rake db:migrate:with_data                      # Migrate the database data and schema (options: VERSION=x, VERBOSE=false)
+    rake db:prepare:with_data                      # Runs setup if database does not exist, or runs data and schema migrations if it does
     rake db:rollback:with_data                     # Rolls the schema back to the previous version (specify steps w/ STEP=n)
     rake db:schema:load:with_data                  # Load both schema.rb and data_schema.rb file into the database
     rake db:structure:load:with_data               # Load both structure.sql and data_schema.rb file into the database
@@ -138,19 +144,12 @@ From now on capistrano will run `rake db:migrate:with_data` in every deploy.
 ## Rails Engines support
 
 This gem also has a initial support for adding data migrations inside Rails engines.
-Inside the Engine's class initializer (the one that inherits from `Rails::Engine`, usually inside `engines/ENGINE_NAME/lib/engine.rb`) you need to add something like this:
-
+Just add your engines directory pattern to data_migrations initializer, for example 
+in the case your engines are located in `engines` folder you can set it up like this:
 
 ```ruby
-module EngineName
-  class Engine < ::Rails::Engine
-    initializer :engine_name do |app|
-      ::DataMigrate.configure do |data_migrate|
-        default_path = ::DataMigrate::Config.new.data_migrations_path
-        data_migrate.data_migrations_path = [default_path, root.join('db', 'data')]
-      end
-    end
-  end
+DataMigrate.configure do |config|
+  config.data_migrations_path = ['db/data'] + Dir['engines/**/db/data']
 end
 ```
 
@@ -163,12 +162,20 @@ Then, in the Engine's `db/data` folder, you can add data migrations and run them
 Run tests for a specific version of Rails
 
 ```
-bundle exec appraisal install
 bundle exec appraisal rails-6.1 rspec
 bundle exec appraisal rails-7.0 rspec
 bundle exec appraisal rails-7.1 rspec
+bundle exec appraisal rails-7.2 rspec
 ```
 
+## Releasing new version
+
+1. Create a new tag, eg `git tag 9.4.1`
+1. Go to https://github.com/ilyakatz/data-migrate/tags
+1. Click "Create release" under 9.4.1
+1. CLick "Generate release notes"
+1. Click "Publish release"
+
 ## Thanks
 
 [Andrew J Vargo](http://github.com/ajvargo) Andrew was the original creator and maintainer of this project!

data/lib/data_migrate/data_migrator.rb

@@ -12,7 +12,7 @@ module DataMigrate
 
     class << self
       def migrations_paths
-        [DataMigrate.config.data_migrations_path]
+        Array.wrap(DataMigrate.config.data_migrations_path)
       end
 
       def create_data_schema_table

data/lib/data_migrate/data_schema.rb

@@ -29,13 +29,13 @@ module DataMigrate
     end
 
     def versions
-      @versions ||= begin
-        versions = []
-        Dir.foreach(DataMigrate::DataMigrator.full_migrations_path) do |file|
-          match_data = DataMigrate::DataMigrator.match(file)
-          versions << match_data[1].to_i if match_data
+      @versions ||= Set.new.tap do |versions|
+        DataMigrate::DataMigrator.migrations_paths.each do |path|
+          Dir.foreach(path) do |file|
+            match_data = DataMigrate::DataMigrator.match(file)
+            versions << match_data[1].to_i if match_data
+          end
         end
-        versions
       end
     end
 

data/lib/data_migrate/database_tasks.rb

@@ -10,21 +10,21 @@ module DataMigrate
     extend self
 
     # These method are only introduced in Rails 7.1
-    unless respond_to?(:with_temporary_connection_for_each)
-      def with_temporary_connection_for_each(env: ActiveRecord::Tasks::DatabaseTasks.env, name: nil, &block) # :nodoc:
+    unless respond_to?(:with_temporary_pool_for_each)
+      def with_temporary_pool_for_each(env: ActiveRecord::Tasks::DatabaseTasks.env, name: nil, &block) # :nodoc:
         if name
           db_config = ActiveRecord::Base.configurations.configs_for(env_name: env, name: name)
-          with_temporary_connection(db_config, &block)
+           with_temporary_pool(db_config, &block)
         else
           ActiveRecord::Base.configurations.configs_for(env_name: env, name: name).each do |db_config|
-            with_temporary_connection(db_config, &block)
+             with_temporary_pool(db_config, &block)
           end
         end
       end
 
-      def with_temporary_connection(db_config) # :nodoc:
+      def with_temporary_connection(db_config, &block) # :nodoc:
         with_temporary_pool(db_config) do |pool|
-          yield pool.connection
+          pool.with_connection(&block)
         end
       end
 
@@ -49,8 +49,8 @@ module DataMigrate
     def db_configs_with_versions
       db_configs_with_versions = Hash.new { |h, k| h[k] = [] }
 
-      with_temporary_connection_for_each do |conn|
-        db_config = conn.pool.db_config
+      with_temporary_pool_for_each do |pool|
+        db_config = pool.db_config
         if db_config.primary?
           versions_to_run = DataMigrate::DatabaseTasks.pending_data_migrations.map { |m| m[:version] }
           target_version = ActiveRecord::Tasks::DatabaseTasks.target_version
@@ -179,9 +179,16 @@ module DataMigrate
 
       ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
 
-      db_configs = ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env)
+      # 7.2 removes the param for db_configs_with_versions in https://github.com/rails/rails/commit/9572fcb4a0bd5396436689a6a42613886871cd81
+      # 7.1 stable backported the change in https://github.com/rails/rails/commit/c53ec4b60980036b43528829d4b0b7457f759224
+      schema_mapped_versions = if Gem::Dependency.new("railties", ">= 7.1.4").match?("railties", Gem.loaded_specs["railties"].version, true)
+        ActiveRecord::Tasks::DatabaseTasks.db_configs_with_versions
+      else
+        db_configs = ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env)
+
+        ActiveRecord::Tasks::DatabaseTasks.db_configs_with_versions(db_configs)
+      end
 
-      schema_mapped_versions = ActiveRecord::Tasks::DatabaseTasks.db_configs_with_versions(db_configs)
       data_mapped_versions = DataMigrate::DatabaseTasks.db_configs_with_versions
 
       mapped_versions = schema_mapped_versions.merge(data_mapped_versions) do |_key, schema_db_configs, data_db_configs|
@@ -204,5 +211,72 @@ module DataMigrate
         end
       end
     end
+
+    def self.prepare_all_with_data
+      seed = false
+
+      each_current_configuration(env) do |db_config|
+        next unless primary?(db_config)
+
+        with_temporary_pool(db_config) do |pool|
+          unless database_exists?(pool.connection)
+            create(db_config)
+            if File.exist?(schema_dump_path(db_config))
+              load_schema(db_config, schema_format, nil)
+              load_schema_current(
+                :ruby,
+                ENV["DATA_SCHEMA"]
+              )
+            end
+
+            seed = true
+          end
+
+          migrate_with_data
+          if dump_schema_after_migration?
+            dump_schema(db_config)
+            DataMigrate::Tasks::DataMigrateTasks.dump
+          end
+        end
+      end
+
+      load_seed if seed
+    end
+
+    private
+
+    def database_exists?(connection)
+      if connection.respond_to?(:database_exists?)  # Rails 7.1+
+        connection.database_exists?
+      else
+        connection.table_exists?(ActiveRecord::SchemaMigration.table_name)
+      end
+    rescue ActiveRecord::NoDatabaseError
+      false
+    end
+
+    def primary?(db_config)
+      if db_config.respond_to?(:primary?)  # Rails 7.0+
+        db_config.primary?
+      else
+        db_config.name == "primary"
+      end
+    end
+
+    def dump_schema_after_migration?
+      if ActiveRecord.respond_to?(:dump_schema_after_migration)
+        ActiveRecord.dump_schema_after_migration
+      else
+        ActiveRecord::Base.dump_schema_after_migration
+      end
+    end
+
+    def schema_format
+      if ActiveRecord.respond_to?(:schema_format)
+        ActiveRecord.schema_format
+      else
+        ActiveRecord::Base.schema_format
+      end
+    end
   end
 end

data/lib/data_migrate/rails_helper.rb

@@ -1,6 +1,12 @@
 module DataMigrate
   class RailsHelper
     class << self
+      def rails_version_equal_to_or_higher_than_7_2
+        return @equal_to_or_higher_than_7_2 if defined?(@equal_to_or_higher_than_7_2)
+
+        @equal_to_or_higher_than_7_2 = Gem::Dependency.new("railties", ">= 7.2.0.alpha").match?("railties", Gem.loaded_specs["railties"].version, true)
+      end
+
       def rails_version_equal_to_or_higher_than_7_1
         return @equal_to_or_higher_than_7_1 if defined?(@equal_to_or_higher_than_7_1)
 
@@ -14,7 +20,9 @@ module DataMigrate
       end
 
       def internal_metadata
-        if rails_version_equal_to_or_higher_than_7_1
+        if rails_version_equal_to_or_higher_than_7_2
+          ActiveRecord::Base.connection_pool.internal_metadata
+        elsif rails_version_equal_to_or_higher_than_7_1
           ActiveRecord::Base.connection.internal_metadata
         else
           ActiveRecord::InternalMetadata
@@ -22,7 +30,9 @@ module DataMigrate
       end
 
       def schema_migration
-        if rails_version_equal_to_or_higher_than_7_1
+        if rails_version_equal_to_or_higher_than_7_2
+          ActiveRecord::Base.connection_pool.schema_migration
+        elsif rails_version_equal_to_or_higher_than_7_1
           ActiveRecord::Base.connection.schema_migration
         else
           ActiveRecord::SchemaMigration
@@ -54,7 +64,9 @@ module DataMigrate
       end
 
       def data_schema_migration
-        if rails_version_equal_to_or_higher_than_7_1
+        if rails_version_equal_to_or_higher_than_7_2
+          DataMigrate::DataSchemaMigration.new(ActiveRecord::Tasks::DatabaseTasks.migration_connection_pool)
+        elsif rails_version_equal_to_or_higher_than_7_1
           DataMigrate::DataSchemaMigration.new(ActiveRecord::Tasks::DatabaseTasks.migration_connection)
         else
           DataMigrate::DataSchemaMigration

data/lib/data_migrate/tasks/data_migrate_tasks.rb

@@ -6,7 +6,7 @@ module DataMigrate
       extend self
 
       def migrations_paths
-        @migrations_paths ||= DataMigrate.config.data_migrations_path
+        @migrations_paths ||= Array.wrap(DataMigrate.config.data_migrations_path)
       end
 
       def dump
@@ -55,11 +55,13 @@ module DataMigrate
         db_list_schema = DataMigrate::RailsHelper.schema_migration_versions
         file_list = []
 
-        Dir.foreach(File.join(Rails.root, migrations_paths)) do |file|
-          # only files matching "20091231235959_some_name.rb" pattern
-          if match_data = /(\d{14})_(.+)\.rb/.match(file)
-            status = db_list_data.delete(match_data[1]) ? 'up' : 'down'
-            file_list << [status, match_data[1], match_data[2], 'data']
+        migrations_paths.each do |path|
+          Dir.foreach(File.join(Rails.root, path)) do |file|
+            # only files matching "20091231235959_some_name.rb" pattern
+            if match_data = /(\d{14})_(.+)\.rb/.match(file)
+              status = db_list_data.delete(match_data[1]) ? 'up' : 'down'
+              file_list << [status, match_data[1], match_data[2], 'data']
+            end
           end
         end
 

data/lib/data_migrate/version.rb

@@ -1,3 +1,3 @@
 module DataMigrate
-  VERSION = "9.4.1".freeze
+  VERSION = "11.2.0".freeze
 end

data/lib/generators/data_migration/data_migration_generator.rb

@@ -38,8 +38,9 @@ module DataMigrate
         File.join(data_migrations_path, "#{file_name}.rb")
       end
 
+      # Use the first path in the data_migrations_path as the target directory
       def data_migrations_path
-        DataMigrate.config.data_migrations_path
+        Array.wrap(DataMigrate.config.data_migrations_path).first
       end
     end
   end

data/tasks/databases.rake

@@ -146,6 +146,13 @@ namespace :db do
       end
     end
   end
+
+  namespace :prepare do
+    desc "Runs setup if database does not exist, or runs data and schema migrations if it does"
+    task with_data: :environment do
+      DataMigrate::DatabaseTasks.prepare_all_with_data
+    end
+  end
 end
 
 namespace :data do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_migrate
 version: !ruby/object:Gem::Version
-  version: 9.4.1
+  version: 11.2.0
 platform: ruby
 authors:
 - Andrew J Vargo
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-08-13 00:00:00.000000000 Z
+date: 2024-11-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -189,29 +189,9 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/workflows/build.yml"
-- ".github/workflows/gempush.yml"
-- ".gitignore"
-- ".hound.yml"
-- ".overcommit.yml"
-- ".rbenv-gemsets"
-- ".rspec"
-- ".rubocop.yml"
-- ".ruby-style.yml"
-- Appraisals
 - Changelog.md
-- Gemfile
-- Gemfile.lock
 - LICENSE
 - README.md
-- Rakefile
-- data_migrate.gemspec
-- gemfiles/rails_6.1.gemfile
-- gemfiles/rails_6.1.gemfile.lock
-- gemfiles/rails_7.0.gemfile
-- gemfiles/rails_7.0.gemfile.lock
-- gemfiles/rails_7.1.gemfile
-- gemfiles/rails_7.1.gemfile.lock
 - lib/capistrano/data_migrate.rb
 - lib/capistrano/data_migrate/migrate.rb
 - lib/data_migrate.rb
@@ -228,36 +208,11 @@ files:
 - lib/data_migrate/schema_migration.rb
 - lib/data_migrate/status_service.rb
 - lib/data_migrate/tasks/data_migrate_tasks.rb
-- lib/data_migrate/test.rb
 - lib/data_migrate/version.rb
 - lib/generators/data_migrate.rb
 - lib/generators/data_migration/data_migration_generator.rb
 - lib/generators/data_migration/templates/data_migration.rb
 - lib/generators/data_migration/templates/migration.rb
-- screenshot.png
-- spec/data_migrate/config_spec.rb
-- spec/data_migrate/data_migrator_spec.rb
-- spec/data_migrate/data_schema_migration_spec.rb
-- spec/data_migrate/data_spec.rb
-- spec/data_migrate/database_tasks_spec.rb
-- spec/data_migrate/migration.rb
-- spec/data_migrate/migration_context_spec.rb
-- spec/data_migrate/schema_dumper_spec.rb
-- spec/data_migrate/schema_migration_spec.rb
-- spec/data_migrate/status_service_spec.rb
-- spec/data_migrate/tasks/data_migrate_tasks_spec.rb
-- spec/db/data/20091231235959_some_name.rb
-- spec/db/data/20171231235959_super_update.rb
-- spec/db/data/20181128000207_excluded_file.rb.other_ext
-- spec/db/data/partial_schema/data_schema.rb
-- spec/db/data/partial_schema/test_data_schema.rb
-- spec/db/data/schema/data_schema.rb
-- spec/db/data/schema/test_data_schema.rb
-- spec/db/migrate/20131111111111_late_migration.rb
-- spec/db/migrate/20202020202011_db_migration.rb
-- spec/generators/data_migration/data_migration_generator_spec.rb
-- spec/spec_helper.rb
-- tasks/.gitkeep
 - tasks/databases.rake
 homepage: https://github.com/ilyakatz/data-migrate
 licenses:

data/.github/workflows/build.yml

@@ -1,35 +0,0 @@
-name: CI
-
-on:
-  - push
-  - pull_request
-
-jobs:
-  test:
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ "ubuntu-20.04" ]
-        ruby:
-          - '3.0'
-          - '3.1'
-          - '3.2'
-          - '3.3'
-        gemfile:
-          - gemfiles/rails_6.1.gemfile
-          - gemfiles/rails_7.0.gemfile
-          - gemfiles/rails_7.1.gemfile
-    runs-on: ubuntu-latest
-    env:
-      BUNDLE_GEMFILE: ${{ matrix.gemfile }}
-      RAILS_ENV: test
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v3
-      - name: Set up Ruby
-        uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: ${{ matrix.ruby }}
-          bundler-cache: true
-      - name: Run tests
-        run: bundle exec rspec

data/.github/workflows/gempush.yml

@@ -1,28 +0,0 @@
-name: Release Gem
-
-on:
-  release:
-    types: [released]
-
-jobs:
-  build:
-    name: Build + Publish
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@master
-      - name: Set up Ruby 3.2
-        uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: '3.2'
-
-      - name: Publish to RubyGems
-        run: |
-          mkdir -p $HOME/.gem
-          touch $HOME/.gem/credentials
-          chmod 0600 $HOME/.gem/credentials
-          printf -- "---\n:rubygems_api_key: ${GEM_HOST_API_KEY}\n" > $HOME/.gem/credentials
-          gem build *.gemspec
-          gem push *.gem
-        env:
-          GEM_HOST_API_KEY: ${{secrets.RUBYGEMS_AUTH_TOKEN}}

data/.gitignore

@@ -1,12 +0,0 @@
-/.rvmrc
-*.gem
-gemfiles/.bundle
-spec/db/test.db
-spec/db/other_test.db
-spec/db/data_schema.rb
-.vscode/
-.DS_Store
-.ruby-gemset
-.ruby-version
-.idea/
-vendor/

data/.hound.yml

@@ -1,4 +0,0 @@
-fail_on_violations: true
-
-ruby:
-  config_file: .rubocop.yml

data/.overcommit.yml

@@ -1,21 +0,0 @@
-# Use this file to configure the Overcommit hooks you wish to use. This will
-# extend the default configuration defined in:
-# https://github.com/brigade/overcommit/blob/master/config/default.yml
-#
-# At the topmost level of this YAML file is a key representing type of hook
-# being run (e.g. pre-commit, commit-msg, etc.). Within each type you can
-# customize each hook, such as whether to only run it on certain files (via
-# `include`), whether to only display output if it fails (via `quiet`), etc.
-#
-# For a complete list of hooks, see:
-# https://github.com/brigade/overcommit/tree/master/lib/overcommit/hook
-#
-# For a complete list of options that you can use to customize hooks, see:
-# https://github.com/brigade/overcommit#configuration
-#
-# Uncomment the following lines to make the configuration take effect.
-
-PreCommit:
-  RuboCop:
-    enabled: true
-    on_warn: fail # Treat all warnings as failures

data/.rbenv-gemsets

@@ -1,2 +0,0 @@
-data-migrate
--global

data/.rspec

@@ -1,3 +0,0 @@
---color
---format d
-

data/.rubocop.yml

@@ -1,2 +0,0 @@
-# Use same configration as used by HoundCI
-inherit_from: .ruby-style.yml