data_customs 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 7d7c7b4003b90a683a9ea760719dc9f3fee13c2e41d8e913904d0c25d05cfb3b
+  data.tar.gz: 23abed296ee83d222794a3467f9fba98430cbb15463505f21ce7be11ff1b4c74
+SHA512:
+  metadata.gz: f2f015a1096f50f09f5f535dcdd7cfe5d5d2c2093763af254cc69743d542cfe21fe82f90f8d42c40d11a38951ce80046ec413c9944ec02ee5107df4207f6e4c3
+  data.tar.gz: a50e40759553bebca3718c77d3095417798ae8c946524a45216d0bf331fc3a9cf08bcaf3e74a53b015b0d7f4523dcadcbfa15c73ec3918cbce4d97b7f46a9783

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2025-09-04
+
+- Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,6 @@
+# Code of conduct
+
+By participating in this project, you agree to abide by the
+[thoughtbot code of conduct][1].
+
+[1]: https://thoughtbot.com/open-source-code-of-conduct

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Matheus Richard
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,216 @@
+# 🛃 Data Customs
+
+> [!WARNING]
+> This project is in early development. The API may change before reaching
+> version 1.0.0.
+
+A simple gem to help you perform data migrations in your Rails app. The premise
+is simple: bundle the migration code along with a verification test to assert
+that the migration achieves the desired effect. If either the migration or the
+verification step fails, the entire migration is rolled back.
+
+> Can't I just test my rake task or migration script?
+
+Great question. Yes, you should test your migration code (including the Data
+Customs migrations). The problem is that **production data is always different
+from your test data**, and **in unexpected ways**. This means that even if your
+tests pass, the migration might still fail when run in production.
+
+Data Customs provides a safety net by ensuring that if the migration doesn't do
+what you expect it to do, the changes are rolled back. This way, you can
+investigate the issue and fix it without leaving your data in a half-migrated
+(or bad!) state.
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+```bash
+bundle add data_customs
+```
+
+If bundler is not being used to manage dependencies, install the gem by
+executing:
+
+```bash
+gem install data_customs
+```
+
+## Usage
+
+### Generating a data migration
+
+You can create a new data migration by running:
+
+```bash
+rails generate data_migration MigrationName
+```
+
+That will create a file at `db/data_migrations/migration_name.rb`.
+
+### Writing a data migration
+
+After generating a migration, implement the `up` method with the code that
+performs the data migration, and the `verify!` method with the code that
+asserts that the migration was successful.
+
+```ruby
+class AddDefaultUsername < DataCustoms::Migration
+  def up
+    User.where.missing(:username).update_all(username: "guest")
+  end
+
+  def verify!
+    if User.exists?(username: nil)
+      raise "Some users still have no usernames!"
+    end
+  end
+end
+
+AddDefaultUsername.run # runs the migration and rolls back if necessary
+```
+
+Use any exception to indicate failure in the `verify!` method.
+
+> [!TIP]
+> Include modules like `RSpec::Matchers` or `Minitest::Assertions` in your
+> migration class to use your favorite assertions and matchers inside `verify!`.
+
+If you need to pass arguments to the data migration, you can use an initializer
+and `run` will forward any arguments to it.
+
+```ruby
+class AddDefaultUsername < DataCustoms::Migration
+  def initialize(default_username "guest")
+    @default_username = default_username
+  end
+
+  def up
+    User.where.missing(:username).update_all(username: @default_username)
+  end
+
+  def verify!
+    if User.exists?(username: nil)
+      raise "Some users still have no usernames!"
+    end
+  end
+end
+
+AddDefaultUsername.run("anonymous")
+```
+
+#### Dealing with large datasets
+
+The migration code runs inside a transaction, so be careful when dealing with
+large datasets, as [it will block the database][blocking] for the duration of
+the transaction.
+
+Data Customs provides a few helpers to make working in batches easier and
+automatically throttles the migration to avoid overwhelming the database.
+
+```ruby
+class AddDefaultUsername < DataCustoms::Migration
+  def up
+    batch(User.where.missing(:username)) do |relation|
+      relation.update_all(username: "guest")
+    end
+  end
+end
+```
+
+Or, if you need access to each individual record:
+
+```ruby
+class AddDefaultUsername < DataCustoms::Migration
+  def up
+    find_each(User.where.missing(:username)) do |record|
+      # do something with record
+    end
+  end
+end
+```
+
+For both methods, you can configure the batch size and the pause between batches:
+
+```ruby
+class LongMigration < DataCustoms::Migration
+  def up
+    batch(records, batch_size: 500, throttle_seconds: 0.1) do |relation|
+      # ...
+    end
+
+    find_each(records, batch_size: 500, throttle_seconds: 0.1) do |record|
+      # ...
+    end
+  end
+end
+```
+
+### Running a data migration in the command line
+
+These migrations don't run automatically. You need to invoke them manually.
+Since they might take some time, you can choose the best time to run them.
+
+If you want to run a data migration from the command line, you can use the
+`data_customs:run` task. It accepts the migration class name (either in
+PascalCase or snake_case) as an argument:
+
+```bash
+rails data_customs:run NAME=AddDefaultUsername
+# or
+rake data_customs:run NAME=add_default_username
+```
+
+If you need to pass arguments to the migration, you can do so by passing them as
+additional arguments to the task:
+
+```bash
+rails data_customs:run NAME=AddDefaultUsername ARGS="anonymous"
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run
+`rake spec` to run the tests. You can also run `bin/console` for an interactive
+prompt that will allow you to experiment.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+<https://github.com/thoughtbot/data_customs>.
+
+This project is intended to be a safe, welcoming space for collaboration, and
+contributors are expected to adhere to the [code of
+conduct](https://github.com/thoughtbot/data_customs/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+Open source templates are Copyright (c) thoughtbot, inc. It contains free
+software that may be redistributed under the terms specified in the
+[LICENSE](https://github.com/thoughtbot/data_customs/blob/main/LICENSE.txt)
+file.
+
+## Code of Conduct
+
+Everyone interacting in the DataCustoms project's codebases, issue trackers,
+chat rooms and mailing lists is expected to follow the [code of
+conduct](https://github.com/thoughtbot/data_customs/blob/main/CODE_OF_CONDUCT.md).
+
+<!-- START /templates/footer.md -->
+
+## About thoughtbot
+
+![thoughtbot](https://thoughtbot.com/thoughtbot-logo-for-readmes.svg)
+
+This repo is maintained and funded by thoughtbot, inc. The names and logos for
+thoughtbot are trademarks of thoughtbot, inc.
+
+We love open source software! See [our other projects][community]. We are
+[available for hire][hire].
+
+[community]: https://thoughtbot.com/community?utm_source=github
+[hire]: https://thoughtbot.com/hire-us?utm_source=github
+
+<!-- END /templates/footer.md -->
+
+[blocking]: https://github.com/ankane/strong_migrations?tab=readme-ov-file#backfilling-data

data/Rakefile

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec
+task build: :spec

data/lib/data_customs/migration.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module DataCustoms
+  class Migration
+    DEFAULT_BATCH_SIZE = 1000
+    DEFAULT_THROTTLE = 0.01
+
+    def self.run(...) = new(...).run
+
+    def up = raise NotImplementedError
+
+    def verify! = raise NotImplementedError
+
+    def run
+      ActiveRecord::Base.transaction do
+        up
+        verify!
+        puts "🛃 Data migration ran successfully!"
+      rescue => e
+        warn "🛃 Data migration failed"
+        raise e
+      end
+    end
+
+    private
+
+    def batch(scope, batch_size: DEFAULT_BATCH_SIZE, throttle_seconds: DEFAULT_THROTTLE)
+      scope.in_batches(of: batch_size) do |relation|
+        yield relation
+        sleep(throttle_seconds) if throttle_seconds.positive?
+      end
+    end
+
+    def find_each(scope, **, &)
+      batch(scope, **) do |relation|
+        relation.each(&)
+      end
+    end
+  end
+end

data/lib/data_customs/railtie.rb

@@ -0,0 +1,13 @@
+require "rails/railtie"
+
+module DataCustoms
+  class Railtie < Rails::Railtie
+    rake_tasks do
+      load File.expand_path("tasks/data_customs.rake", __dir__)
+    end
+
+    generators do
+      require "generators/data_migration/data_migration_generator"
+    end
+  end
+end

data/lib/data_customs/tasks/data_customs.rake

@@ -0,0 +1,18 @@
+namespace :data_customs do
+  desc 'Run a single data migration from db/data_migrations'
+  task run: :environment do
+    name = ENV['NAME']
+    abort '❌ Missing migration name (e.g. `rake data_customs:run NAME=fix_users`)' unless name
+
+    path = Rails.root.join('db', 'data_migrations', "#{name.underscore}.rb")
+    abort "❌ Migration not found: #{path}" unless File.exist?(path)
+
+    require path
+    migration_class = name.camelize.constantize
+    if args = ENV['ARGS']
+      migration_class.run(args.split(','))
+    else
+      migration_class.run
+    end
+  end
+end

data/lib/data_customs/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module DataCustoms
+  VERSION = "0.1.0"
+end

data/lib/data_customs.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require_relative "data_customs/migration"
+require_relative "data_customs/railtie"
+require_relative "data_customs/version"
+
+module DataCustoms
+end

data/lib/generators/data_migration/data_migration_generator.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/named_base"
+
+module DataMigration
+  class DataMigrationGenerator < Rails::Generators::NamedBase
+    source_root File.expand_path("templates", __dir__)
+
+    def create_data_migration_file
+      template "data_migration.rb.tt", File.join("db/data_migrations", "#{file_name.underscore}.rb")
+    end
+  end
+end

data/lib/generators/data_migration/templates/data_migration.rb.tt

@@ -0,0 +1,10 @@
+class <%= class_name %> < DataCustoms::Migration
+  def up
+    # Your migration code goes here.
+  end
+
+  def verify!
+    # Your verification code goes here.
+    # Raise an exception to cause the migration to fail.
+  end
+end

metadata

@@ -0,0 +1,71 @@
+--- !ruby/object:Gem::Specification
+name: data_customs
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Matheus Richard
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+description: A simple gem to help you perform data migrations in your Rails app. Define
+  the work and a verification step steps, if any of them fail, your migration will
+  be rolled back.
+email:
+- matheusrichardt@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/data_customs.rb
+- lib/data_customs/migration.rb
+- lib/data_customs/railtie.rb
+- lib/data_customs/tasks/data_customs.rake
+- lib/data_customs/version.rb
+- lib/generators/data_migration/data_migration_generator.rb
+- lib/generators/data_migration/templates/data_migration.rb.tt
+homepage: https://github.com/thoughtbot/data_customs
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/thoughtbot/data_customs
+  source_code_uri: https://github.com/thoughtbot/data_customs
+  changelog_uri: https://github.com/thoughtbot/data_customs/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.9
+specification_version: 4
+summary: A simple gem to help you perform data migrations in your Rails app.
+test_files: []