data_migration_for_rails 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 49026939624c298b1cb497970e5768eb0c8494c2803e601736c7df95c5a9cec1
+  data.tar.gz: a8f16d45bfef5ce2e926bae3e558d33e0167b99640904c5b66a1e5731810e87b
+SHA512:
+  metadata.gz: 87560a148c1981e1783c68848e9919314fd199e46868960e308e6f3ed2fab53e7d48fa94e0db9712a3218f9ba18c42f1bc3cbb134dbcd86568df6322b732ab39
+  data.tar.gz: 91057070c7092b2d766f5f4f2a0bda8b5e88db334108bd5ab989cecebc6cf4f71e84889c9d59f9ce4a2543e948251e57b65a1f1d934e0b848ade659713421cda

data/LICENSE

@@ -0,0 +1,17 @@
+GNU GENERAL PUBLIC LICENSE
+Version 3, 29 June 2007
+
+Copyright (C) 2025 Vaibhav Rokkam
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.

data/README.md

@@ -0,0 +1,196 @@
+# Data Migration - Rails Engine
+
+A mountable Rails engine for migrating data between Rails application environments with full audit trails, role-based access control, and dependency management.
+
+## Features
+
+- **Role-Based Access Control** - Admin, Operator, and Viewer roles
+- **Migration Plans** - Reusable migration configurations with execution order
+- **Plan Configuration Import/Export** - Share migration plans across environments as JSON
+- **Dynamic Filtering** - ActiveRecord queries with runtime parameter substitution
+- **Dependency Management** - Maintain referential integrity across related models
+- **Association Handling** - Remap foreign keys and handle polymorphic associations
+- **Attachment Handling** - Export/import Active Storage attachments with URL or raw data modes
+- **Background Processing** - Sidekiq-based async exports and imports
+- **Audit Trail** - Complete execution history and detailed record-level tracking
+- **Real-time Progress** - ActionCable integration for live updates
+
+![Landing Page](screenshots/landing_page.png)
+
+---
+
+## Installation
+
+### 1. Add to Gemfile
+
+```ruby
+gem "data_migration_for_rails", "~> 0.1.1"
+```
+
+### 2. Install and Migrate
+
+```bash
+bundle install
+bundle exec rake data_migration_engine:install:migrations
+bundle exec rake db:migrate
+```
+
+### 3. Mount Routes
+
+In `config/routes.rb`:
+
+```ruby
+Rails.application.routes.draw do
+  mount DataMigration::Engine, at: "/data_migration"
+end
+```
+
+### 4. Seed Initial Admin User
+
+```bash
+bundle exec rake data_migration:seed
+```
+
+This creates an admin user with independent authentication:
+- **Email:** admin@datamigration.local
+- **Password:** password
+
+**Change the password immediately** after first login! Admin users can create additional users via the "Users" menu.
+
+### 5. Configure Sidekiq
+
+In `config/application.rb`:
+
+```ruby
+config.active_job.queue_adapter = :sidekiq
+```
+
+Start services:
+```bash
+redis-server
+bundle exec sidekiq
+```
+
+---
+
+## Prerequisites
+
+- Ruby 3.0+
+- Rails 7.0+
+- Redis
+- Devise (for authentication)
+- Pundit (for authorization)
+- Active Storage (optional, for attachment handling)
+
+---
+
+## Usage
+
+### 1. Create Migration Plan
+
+Navigate to `/data_migration/migration_plans` and create a new plan.
+
+![Migration Plan Page](screenshots/migration_plan_page.png)
+
+### 2. Add Migration Steps
+
+Configure each step with:
+
+- **Model Name**: Select from dropdown (shows fields and attachments)
+- **Sequence**: Execution order
+- **Filter Query** (optional): ActiveRecord query with optional placeholders
+
+Examples:
+```ruby
+where(active: true).limit(100)
+where("created_at > ?", "{{cutoff_date}}")
+```
+
+- **Column Overrides**: Export association attributes
+- **Association ID Mappings**: Remap foreign keys on import
+- **Dependee Attribute Mapping**: Filter based on parent step's exported records
+- **Attachment Export Mode**: Choose how to handle Active Storage attachments:
+  - **Ignore** - Skip attachments
+  - **URL** - Export attachment URLs (for cloud storage)
+  - **Raw Data** - Export actual files in archive (for local storage)
+
+![Migration Step Form](screenshots/migration_step_form.png)
+
+### 3. Export
+
+Click **Export**, fill in any filter parameters, and download the `.tar.gz` archive.
+
+![Export Execution Form](screenshots/export_execution_form.png)
+
+![Executed Export Page](screenshots/executed_export_page.png)
+
+### 4. Import
+
+On target environment, click **Import**, upload the archive, and select conflict resolution strategy.
+
+![Import Execution Form](screenshots/import_execution_form.png)
+
+![Executed Import Page](screenshots/executed_import_page.png)
+
+---
+
+## Advanced Features
+
+### Dynamic Filter Parameters
+
+Use `{{placeholder}}` syntax in filter queries. Values are collected at export time and validated before execution.
+
+### Referential Integrity
+
+Set **Depends On Step** and **Dependee Attribute Mapping** to ensure child records only include those referencing exported parent records.
+
+Example: Export only employees whose companies were exported.
+
+### Polymorphic Associations
+
+Configure polymorphic associations with type-specific lookup attributes for proper ID remapping on import.
+
+### Plan Configuration Import/Export
+
+Share migration plan configurations across environments:
+
+1. **Export**: Click "📋 Export Config" on any plan → Download JSON file
+2. **Import**: Click "📋 Import Config" → Upload JSON → Plan and steps created/updated automatically
+
+Perfect for deploying migration plans to production or sharing with team members.
+
+---
+
+## Security
+
+- Pundit-based authorization on all actions
+- Strong parameter validation
+- Safe ActiveRecord query evaluation
+- Multi-layer input validation
+- Complete audit trail
+
+---
+
+## Troubleshooting
+
+**Sidekiq not processing:**
+- Check Redis: `redis-cli ping`
+- Verify `config.active_job.queue_adapter = :sidekiq`
+- Check logs: `tail -f log/sidekiq.log`
+
+**Filter parameter errors:**
+- Ensure placeholders are inside quotes: `"{{param}}"` not `{{param}}`
+- Provide all required parameter values before export
+
+**Model not found:**
+- Verify exact ActiveRecord class name (case-sensitive, singular form)
+  - Example: Use "Company" not "company" or "Companies"
+- Check lookup attributes exist on target models
+
+---
+
+## License
+
+This project is licensed under the [GNU General Public License v3.0](LICENSE).
+
+Copyright © 2025 Vaibhav Rokkam.

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+# Add your own tasks in files placed in lib/tasks ending in .rake,
+# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
+
+require_relative 'config/application'
+
+Rails.application.load_tasks

data/app/assets/config/manifest.js

@@ -0,0 +1,2 @@
+//= link_tree ../images
+//= link_directory ../stylesheets .css

data/app/assets/stylesheets/application.css

@@ -0,0 +1,15 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS (and SCSS, if configured) file within this directory, lib/assets/stylesheets, or any plugin's
+ * vendor/assets/stylesheets directory can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the bottom of the
+ * compiled file so the styles you add here take precedence over styles defined in any other CSS
+ * files in this directory. Styles in this file should be added after the last require_* statement.
+ * It is generally better to create a new file per style scope.
+ *
+ *= require_tree .
+ *= require_self
+ */

data/app/channels/application_cable/channel.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module ApplicationCable
+  class Channel < ActionCable::Channel::Base
+  end
+end

data/app/channels/application_cable/connection.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module ApplicationCable
+  class Connection < ActionCable::Connection::Base
+  end
+end

data/app/controllers/concerns/data_migration/pundit_authorization.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+module DataMigration
+  module PunditAuthorization
+    extend ActiveSupport::Concern
+
+    included do
+      after_action :verify_authorized, except: :index
+      after_action :verify_policy_scoped, only: :index
+    end
+  end
+end

data/app/controllers/data_migration/application_controller.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module DataMigration
+  class ApplicationController < ActionController::Base
+    protect_from_forgery with: :null_session, prepend: true
+
+    # Devise authentication
+    before_action :authenticate_user!, unless: :devise_controller?
+
+    # Pundit authorization
+    include Pundit::Authorization
+
+    rescue_from Pundit::NotAuthorizedError, with: :user_not_authorized
+
+    # Layout
+    layout 'data_migration'
+
+    # Make engine route helpers available in views
+    helper DataMigration::Engine.routes.url_helpers
+
+    # Make model registry available to all views
+    helper_method :model_registry
+
+    private
+
+    def model_registry
+      @model_registry ||= DataMigration::ModelRegistry.all_models
+    end
+
+    def user_not_authorized
+      flash[:alert] = 'You are not authorized to perform this action.'
+      redirect_to(request.referrer || '/data_migration/migration_plans')
+    end
+
+    # Devise helpers are automatically available as:
+    # - current_user (from devise_for :users)
+    # - user_signed_in? (from devise_for :users)
+    # No need to override since resource name is :users
+
+    # Override Devise redirect after sign in
+    def after_sign_in_path_for(_resource)
+      '/data_migration/migration_plans'
+    end
+
+    # Handle CSRF token verification failures for Devise controllers
+    # This prevents errors when Devise's FailureApp redirects after failed login
+    def handle_unverified_request
+      if devise_controller?
+        if controller_name == 'sessions'
+          # For login forms, redirect back to login to get fresh token
+          flash[:alert] = 'Your session has expired. Please try signing in again.'
+        else
+          # For other Devise controllers, sign out and redirect
+          sign_out if user_signed_in?
+          flash[:alert] = 'Session expired. Please sign in again.'
+        end
+        redirect_to new_session_path(:user)
+      else
+        super
+      end
+    end
+  end
+end

data/app/controllers/data_migration/exports_controller.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module DataMigration
+  class ExportsController < ApplicationController
+    before_action :set_migration_plan
+
+    def new
+      authorize @migration_plan, :execute?
+      @filter_params = extract_placeholders_from_plan
+    end
+
+    def create
+      authorize @migration_plan, :execute?
+
+      # Validate that all required filter parameters are provided
+      required_params = extract_placeholders_from_plan
+
+      # Permit the filter_params dynamically based on what's required
+      provided_params = if params[:filter_params].present? && required_params.any?
+                          params.require(:filter_params).permit(*required_params.keys)
+                        else
+                          {}
+                        end
+
+      missing_params = required_params.keys.select do |param|
+        provided_params[param].blank?
+      end
+
+      if missing_params.any?
+        flash[:alert] = "Please provide values for: #{missing_params.join(', ')}"
+        @filter_params = required_params
+        render :new and return
+      end
+
+      execution = MigrationExecution.create!(
+        migration_plan: @migration_plan,
+        user: current_user,
+        execution_type: :export,
+        status: :pending,
+        filter_params: provided_params.to_h
+      )
+
+      ExportJob.perform_later(execution.id)
+
+      redirect_to "/data_migration/migration_executions/#{execution.id}",
+                  notice: 'Export started. You will be notified when it completes.'
+    end
+
+    private
+
+    def set_migration_plan
+      @migration_plan = MigrationPlan.find(params[:migration_plan_id])
+    end
+
+    def extract_placeholders_from_plan
+      placeholders = {}
+      @migration_plan.migration_steps.each do |step|
+        next if step.filter_query.blank?
+
+        # Extract placeholders like {{param_name}} from filter query
+        step.filter_query.scan(/\{\{(\w+)\}\}/).flatten.each do |param|
+          placeholders[param] = nil
+        end
+      end
+      placeholders
+    end
+  end
+end

data/app/controllers/data_migration/imports_controller.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+module DataMigration
+  class ImportsController < ApplicationController
+    before_action :set_migration_plan
+
+    def new
+      authorize @migration_plan, :execute?
+      @filter_params = extract_placeholders_from_plan
+      @last_export = @migration_plan.migration_executions.export.completed.order(created_at: :desc).first
+    end
+
+    def create
+      authorize @migration_plan, :execute?
+
+      unless params[:archive_file].present?
+        redirect_to "/data_migration/migration_plans/#{@migration_plan.id}/import/new",
+                    alert: 'Please select a file to upload.'
+        return
+      end
+
+      uploaded_file = params[:archive_file]
+
+      # Save uploaded file to tmp directory
+      file_path = save_uploaded_file(uploaded_file)
+
+      # Use provided filter params or default to empty hash
+      filter_params = params[:filter_params].present? ? params[:filter_params] : {}
+
+      execution = MigrationExecution.create!(
+        migration_plan: @migration_plan,
+        user: current_user,
+        execution_type: :import,
+        status: :pending,
+        file_path: file_path,
+        filter_params: filter_params
+      )
+
+      ImportJob.perform_later(execution.id)
+
+      redirect_to "/data_migration/migration_executions/#{execution.id}",
+                  notice: 'Import started. You will be notified when it completes.'
+    end
+
+    private
+
+    def set_migration_plan
+      @migration_plan = MigrationPlan.find(params[:migration_plan_id])
+    end
+
+    def extract_placeholders_from_plan
+      placeholders = {}
+      @migration_plan.migration_steps.each do |step|
+        next if step.filter_query.blank?
+
+        # Extract placeholders like {{param_name}} from filter query
+        step.filter_query.scan(/\{\{(\w+)\}\}/).flatten.each do |param|
+          placeholders[param] = nil
+        end
+      end
+      placeholders
+    end
+
+    def save_uploaded_file(uploaded_file)
+      timestamp = Time.current.strftime('%Y%m%d_%H%M%S')
+      filename = "import_#{timestamp}_#{uploaded_file.original_filename}"
+      file_path = Rails.root.join('tmp', 'imports', filename)
+
+      FileUtils.mkdir_p(File.dirname(file_path))
+
+      File.open(file_path, 'wb') do |file|
+        file.write(uploaded_file.read)
+      end
+
+      file_path.to_s
+    end
+  end
+end

data/app/controllers/data_migration/migration_executions_controller.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+module DataMigration
+  class MigrationExecutionsController < ApplicationController
+    include DataMigration::PunditAuthorization
+
+    before_action :set_execution, only: %i[show download]
+
+    def index
+      @executions = policy_scope(MigrationExecution).recent.includes(:migration_plan, :user)
+    end
+
+    def show
+      authorize @execution
+
+      # Initialize variables
+      @migration_records = []
+      @model_names = []
+      @action_counts = {}
+
+      # Only load migration records for import executions
+      return unless @execution.import?
+
+      # Reload association to ensure fresh data
+      @execution.reload
+
+      # Filter and paginate migration records
+      @migration_records = @execution.migration_records.order(created_at: :desc)
+
+      # Apply filters if provided
+      @migration_records = @migration_records.by_model(params[:model]) if params[:model].present?
+
+      @migration_records = @migration_records.by_action(params[:filter_action]) if params[:filter_action].present?
+
+      # Limit to 500 records for display (can be customized)
+      @limit = (params[:limit] || 500).to_i
+      @migration_records = @migration_records.limit(@limit).to_a
+
+      # Get unique model names for filter dropdown
+      @model_names = @execution.migration_records.distinct.pluck(:migrated_model_name).sort
+
+      # Get total counts by action (convert enum integers to string keys)
+      raw_counts = @execution.migration_records.group(:action).count
+      raw_counts.each do |action_value, count|
+        action_name = MigrationRecord.actions.key(action_value)
+        @action_counts[action_name] = count if action_name
+      end
+    end
+
+    def download
+      authorize @execution, :download?
+
+      unless @execution.completed? && @execution.export? && @execution.file_path.present?
+        redirect_to @execution, alert: 'Export file not available.'
+        return
+      end
+
+      unless File.exist?(@execution.file_path)
+        redirect_to @execution, alert: 'Export file not found.'
+        return
+      end
+
+      send_file @execution.file_path,
+                filename: File.basename(@execution.file_path),
+                type: 'application/gzip',
+                disposition: 'attachment'
+    end
+
+    private
+
+    def set_execution
+      @execution = MigrationExecution.find(params[:id])
+    end
+  end
+end

data/app/controllers/data_migration/migration_plans_controller.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+module DataMigration
+  class MigrationPlansController < ApplicationController
+    include DataMigration::PunditAuthorization
+
+    before_action :set_migration_plan, only: %i[show edit update destroy export_config]
+
+    def index
+      @migration_plans = policy_scope(MigrationPlan).ordered_by_name
+    end
+
+    def show
+      authorize @migration_plan
+    end
+
+    def new
+      @migration_plan = MigrationPlan.new
+      authorize @migration_plan
+    end
+
+    def create
+      @migration_plan = MigrationPlan.new(migration_plan_params)
+      authorize @migration_plan
+
+      if @migration_plan.save
+        redirect_to "/data_migration/migration_plans/#{@migration_plan.id}",
+                    notice: 'Migration plan was successfully created.'
+      else
+        render :new, status: :unprocessable_entity
+      end
+    end
+
+    def edit
+      authorize @migration_plan
+    end
+
+    def update
+      authorize @migration_plan
+
+      if @migration_plan.update(migration_plan_params)
+        redirect_to "/data_migration/migration_plans/#{@migration_plan.id}",
+                    notice: 'Migration plan was successfully updated.'
+      else
+        render :edit, status: :unprocessable_entity
+      end
+    end
+
+    def destroy
+      authorize @migration_plan
+
+      if @migration_plan.destroy
+        redirect_to '/data_migration/migration_plans', notice: 'Migration plan was successfully destroyed.'
+      else
+        redirect_to "/data_migration/migration_plans/#{@migration_plan.id}",
+                    alert: 'Cannot delete migration plan with existing executions.'
+      end
+    end
+
+    def export_config
+      authorize @migration_plan
+
+      service = MigrationPlans::ExportConfigService.new(@migration_plan)
+      json_config = service.call
+
+      send_data json_config,
+                filename: "#{@migration_plan.name.parameterize}_config.json",
+                type: 'application/json',
+                disposition: 'attachment'
+    end
+
+    def import_config
+      authorize MigrationPlan.new
+
+      unless params[:config_file].present?
+        redirect_to '/data_migration/migration_plans', alert: 'Please select a configuration file to import.'
+        return
+      end
+
+      config_json = params[:config_file].read
+      service = MigrationPlans::ImportConfigService.new(config_json, current_user)
+
+      imported_plan = service.call
+
+      if service.success? && imported_plan
+        redirect_to "/data_migration/migration_plans/#{imported_plan.id}",
+                    notice: 'Migration plan configuration imported successfully!'
+      else
+        redirect_to '/data_migration/migration_plans', alert: "Import failed: #{service.errors.join(', ')}"
+      end
+    end
+
+    private
+
+    def set_migration_plan
+      @migration_plan = MigrationPlan.find(params[:id])
+    end
+
+    def migration_plan_params
+      params.require(:migration_plan).permit(:name, :description, settings: [])
+    end
+  end
+end