data_porter 0.5.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e99d5974ba60bb61902d1dcf98ad74d940fd82344ec88404fa55e2414cee3fc0
-  data.tar.gz: 7166547614a1a5942648e85d900d30abe053b4ba4f066811bb71389b83359531
+  metadata.gz: 20e2b579cf4078a611095abd155090d7754e1e33e044a022136c846edb793bcb
+  data.tar.gz: 58e1bcad055c198aab6ccb71ddbbb5b219d5b82a7a94a34fb8c064f109d37878
 SHA512:
-  metadata.gz: 66860e2fb095ccd684e9899d8209fa4ae26b4c79595792bc3ebc4714a34b4a0605fab7e8c1e3a3e7698ad2b40e1cfbd06b58e7b80c995fab50091c88e547a2f6
-  data.tar.gz: 3864e610fbfce43d2ed67fdc5eab5b8d41920f6f79644fa549875f9bac8a96d2eae88cc358af8b74c85c2b374dc7486c728da9afb9ee786027de327d726e1b61
+  metadata.gz: 9929013331242330b53fe9f5e4b4940600a339c7e2d723551e864a969406cd2c516e70a10c95eff25c820b7f7e5904b63da4156b473d140e261d3a65ed9811eb
+  data.tar.gz: c9be1bbff65d0da5b36b5a837f590514ed29eb911a4ba527baa3e2f228f2d27252a4f4ff9bff3af579ac6674b6be19b32f0f4bf23aab992d42291877ea019e56

data/CHANGELOG.md

@@ -5,6 +5,35 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-02-07
+
+### Added
+
+- **Import params DSL** -- Targets declare extra form fields via `params { param :hotel_id, type: :select, ... }` with support for `:select`, `:text`, `:number`, and `:hidden` types
+- **`DSL::Param` struct** -- Stores param definitions (name, type, required, label, collection, default) with type validation
+- **`import_params` accessor** -- Available in all target instance methods (`persist`, `transform`, `validate`, `after_import`, `on_error`), defaults to `{}`
+- **Dynamic form rendering** -- Stimulus controller and inline JS dynamically render param fields when a target is selected, with default values and required indicators
+- **Required params validation** -- Controller validates required import params on create, with error messages per missing field
+- **Registry params serialization** -- `Registry.available` includes serialized param definitions with collection lambdas evaluated at call time
+
+### Changed
+
+- Controller concerns extracted into `ImportValidation`, `MappingManagement`, `RecordPagination`
+- Orchestrator logic extracted into `RecordBuilder`, `Importer`, `DryRunner` modules
+- Inline JS in `new.html.erb` extracted for consistency
+- 354 RSpec examples (up from 313), 0 failures
+
+## [0.6.0] - 2026-02-07
+
+### Added
+
+- **Records pagination** -- Preview and completed pages paginate records at 50 per page with Previous/Next navigation
+- **Pagination component** -- Phlex `Shared::Pagination` with disabled states on first/last page and anchor scrolling to keep viewport on the table
+
+### Changed
+
+- 313 RSpec examples (up from 300), 0 failures
+
 ## [0.5.0] - 2026-02-07
 
 ### Added

data/README.md

@@ -1,7 +1,8 @@
 # DataPorter
 
-> [!WARNING]
-> This gem is under active development and not yet production-ready. APIs and features may change without notice.
+> [!CAUTION]
+> **This gem is under active development and not yet production-ready.**
+> APIs and features may change without notice. Use at your own risk.
 
 A mountable Rails engine for data import workflows: **Upload**, **Map**, **Preview**, **Import**.
 
@@ -23,9 +24,12 @@ Supports CSV, JSON, XLSX, and API sources with a declarative DSL for defining im
 - **4 source types** -- CSV, XLSX, JSON, and API with a unified parsing pipeline
 - **Interactive column mapping** -- Drag-free UI to match file headers to target fields ([docs](docs/MAPPING.md))
 - **Mapping templates** -- Save and reuse column mappings across imports ([docs](docs/MAPPING.md#mapping-templates))
-- **Real-time progress** -- ActionCable updates with polling fallback
+- **Real-time progress** -- JSON polling with animated progress bar, no ActionCable required
 - **Dry run mode** -- Validate against the database without persisting
 - **Standalone UI** -- Self-contained layout with Turbo Drive and Stimulus, no host app dependencies
+- **Import params** -- Declare extra form fields (select, text, number, hidden) per target for scoped imports ([docs](docs/TARGETS.md#params--))
+- **Per-target source filtering** -- Each target declares its allowed sources, the UI filters accordingly
+- **Import deletion & auto-purge** -- Delete imports from the UI, or schedule `rake data_porter:purge` for automatic cleanup
 - **Declarative Target DSL** -- One class per import type, zero boilerplate ([docs](docs/TARGETS.md))
 
 ## Requirements
@@ -33,7 +37,6 @@ Supports CSV, JSON, XLSX, and API sources with a declarative DSL for defining im
 - Ruby >= 3.2
 - Rails >= 7.0
 - ActiveStorage (for file uploads)
-- ActionCable (optional, for real-time progress)
 
 ## Installation
 
@@ -54,7 +57,7 @@ The generator creates:
 Generate a target:
 
 ```bash
-bin/rails generate data_porter:target Product name:string:required price:integer sku:string
+bin/rails generate data_porter:target Product name:string:required price:integer sku:string --sources csv xlsx
 ```
 
 Implement `persist` in `app/importers/product_target.rb`:
@@ -110,6 +113,7 @@ pending -> parsing -> previewing -> importing -> completed
 | [Targets](docs/TARGETS.md) | DSL reference, columns, hooks, generator |
 | [Sources](docs/SOURCES.md) | CSV, JSON, XLSX, API setup and examples |
 | [Column Mapping](docs/MAPPING.md) | Interactive mapping, templates, priority order |
+| [Roadmap](docs/ROADMAP.md) | v1.0 plan and progress |
 
 ## Routes
 
@@ -118,6 +122,8 @@ pending -> parsing -> previewing -> importing -> completed
 | GET | `/imports` | List imports |
 | POST | `/imports` | Create import |
 | GET | `/imports/:id` | Show import |
+| DELETE | `/imports/:id` | Delete import |
+| GET | `/imports/:id/status` | JSON progress polling |
 | PATCH | `/imports/:id/update_mapping` | Save column mapping |
 | POST | `/imports/:id/parse` | Parse source |
 | POST | `/imports/:id/confirm` | Run import |
@@ -131,7 +137,7 @@ pending -> parsing -> previewing -> importing -> completed
 git clone https://github.com/SerylLns/data_porter.git
 cd data_porter
 bin/setup
-bundle exec rspec     # 280 specs
+bundle exec rspec     # 354 specs
 bundle exec rubocop   # 0 offenses
 ```
 

data/app/assets/javascripts/data_porter/import_form_controller.js

@@ -0,0 +1,126 @@
+import { Controller } from "@hotwired/stimulus"
+
+export default class extends Controller {
+  static targets = ["targetSelect", "sourceSelect", "fileField", "fileInput", "dropzone", "fileName", "modal", "paramsContainer"]
+  static values = { sources: Object, params: Object }
+
+  connect() {
+    this.filterSources()
+    this.renderParams()
+  }
+
+  filterSources() {
+    if (!this.hasSourceSelectTarget || !this.hasTargetSelectTarget) return
+    var allowed = this.sourcesValue[this.targetSelectTarget.value]
+    var options = this.sourceSelectTarget.options
+    for (var i = 1; i < options.length; i++) {
+      options[i].style.display = allowed && allowed.indexOf(options[i].value) === -1 ? "none" : ""
+    }
+    if (allowed && this.sourceSelectTarget.selectedIndex > 0 && allowed.indexOf(this.sourceSelectTarget.value) === -1) {
+      this.sourceSelectTarget.selectedIndex = 0
+      this.fileFieldTarget.style.display = ""
+    }
+    this.renderParams()
+  }
+
+  renderParams() {
+    if (!this.hasParamsContainerTarget || !this.hasTargetSelectTarget) return
+    this.paramsContainerTarget.innerHTML = ""
+    var defs = this.paramsValue[this.targetSelectTarget.value] || []
+    var self = this
+    defs.forEach(function(p) { self.paramsContainerTarget.appendChild(self.buildParamField(p)) })
+  }
+
+  buildParamField(p) {
+    var div = document.createElement("div")
+    div.className = "dp-field"
+    div.appendChild(this.buildLabel(p))
+    div.appendChild(this.buildInput(p))
+    return div
+  }
+
+  buildLabel(p) {
+    var label = document.createElement("label")
+    label.className = "dp-label"
+    label.textContent = p.label + (p.required ? " *" : "")
+    return label
+  }
+
+  buildInput(p) {
+    if (p.type === "select" && p.collection) return this.buildSelect(p)
+    return this.buildTextField(p)
+  }
+
+  buildSelect(p) {
+    var select = document.createElement("select")
+    select.className = "dp-select"
+    select.name = "data_import[config][import_params][" + p.name + "]"
+    if (p.required) select.required = true
+    var blank = document.createElement("option")
+    blank.value = ""
+    blank.textContent = "Select..."
+    select.appendChild(blank)
+    p.collection.forEach(function(opt) {
+      var o = document.createElement("option")
+      o.textContent = opt[0]
+      o.value = opt[1]
+      if (p["default"] && String(opt[1]) === String(p["default"])) o.selected = true
+      select.appendChild(o)
+    })
+    return select
+  }
+
+  buildTextField(p) {
+    var input = document.createElement("input")
+    input.className = "dp-input"
+    input.type = p.type === "number" ? "number" : (p.type === "hidden" ? "hidden" : "text")
+    input.name = "data_import[config][import_params][" + p.name + "]"
+    if (p["default"]) input.value = p["default"]
+    if (p.required) input.required = true
+    return input
+  }
+
+  toggleFileField() {
+    this.fileFieldTarget.style.display = this.sourceSelectTarget.value === "api" ? "none" : ""
+  }
+
+  handleFile() {
+    if (this.fileInputTarget.files.length > 0) {
+      this.fileNameTarget.textContent = this.fileInputTarget.files[0].name
+      this.fileNameTarget.style.display = ""
+      this.dropzoneTarget.classList.add("dp-dropzone--has-file")
+    }
+  }
+
+  dragover(event) {
+    event.preventDefault()
+    this.dropzoneTarget.classList.add("dp-dropzone--dragover")
+  }
+
+  dragleave() {
+    this.dropzoneTarget.classList.remove("dp-dropzone--dragover")
+  }
+
+  drop(event) {
+    event.preventDefault()
+    this.dropzoneTarget.classList.remove("dp-dropzone--dragover")
+    if (event.dataTransfer.files.length > 0) {
+      this.fileInputTarget.files = event.dataTransfer.files
+      this.fileInputTarget.dispatchEvent(new Event("change"))
+    }
+  }
+
+  closeModal(event) {
+    if (event.key === "Escape") {
+      this.modalTarget.classList.remove("dp-modal--open")
+    }
+  }
+
+  openModal() {
+    this.modalTarget.classList.add("dp-modal--open")
+  }
+
+  closeModalClick() {
+    this.modalTarget.classList.remove("dp-modal--open")
+  }
+}

data/app/assets/stylesheets/data_porter/table.css

@@ -49,3 +49,48 @@
   align-items: center;
   gap: 0.5rem;
 }
+
+.dp-pagination {
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  gap: 1rem;
+  margin: 1.5rem 0;
+}
+
+.dp-pagination__btn {
+  display: inline-flex;
+  align-items: center;
+  padding: 0.5rem 1rem;
+  font-size: 0.875rem;
+  font-weight: 500;
+  color: var(--dp-primary);
+  background: white;
+  border: 1px solid var(--dp-gray-300);
+  border-radius: var(--dp-radius-md);
+  text-decoration: none;
+  cursor: pointer;
+  transition: all 0.15s ease;
+}
+
+.dp-pagination__btn:hover {
+  background: var(--dp-gray-50);
+  border-color: var(--dp-primary);
+}
+
+.dp-pagination__btn--disabled {
+  color: var(--dp-gray-400);
+  cursor: default;
+  pointer-events: none;
+}
+
+.dp-pagination__btn--disabled:hover {
+  background: white;
+  border-color: var(--dp-gray-300);
+}
+
+.dp-pagination__info {
+  font-size: 0.875rem;
+  color: var(--dp-gray-500);
+  font-weight: 500;
+}

data/app/controllers/data_porter/concerns/import_validation.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Concerns
+    module ImportValidation
+      extend ActiveSupport::Concern
+
+      private
+
+      def valid_source_for_target?
+        target = DataPorter::Registry.find(@import.target_key)
+        allowed = target._sources || DataPorter.configuration.enabled_sources
+        return true if allowed.map(&:to_s).include?(@import.source_type.to_s)
+
+        @import.errors.add(:source_type, "#{@import.source_type} is not available for this target")
+        false
+      end
+
+      def valid_file_presence?
+        return true unless %w[csv json xlsx].include?(@import.source_type)
+        return true if @import.file.attached?
+
+        @import.errors.add(:file, "must be attached for #{@import.source_type.upcase} imports")
+        false
+      end
+
+      def valid_import_params?
+        missing = missing_required_params
+        return true if missing.empty?
+
+        missing.each { |p| @import.errors.add(:base, "#{p.label} is required") }
+        false
+      end
+
+      def missing_required_params
+        target = DataPorter::Registry.find(@import.target_key)
+        required = (target._params || []).select(&:required)
+        values = import_param_values
+        required.reject { |p| values[p.name.to_s].present? }
+      end
+
+      def import_param_values
+        (@import.config || {}).fetch("import_params", {})
+      end
+    end
+  end
+end

data/app/controllers/data_porter/concerns/mapping_management.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Concerns
+    module MappingManagement
+      extend ActiveSupport::Concern
+
+      private
+
+      def load_mapping_data
+        target = @import.target_class
+        columns = target._columns || []
+        @file_headers = @import.config["file_headers"] || []
+        @target_columns = columns.map { |c| [c.label, c.name.to_s, c.required] }
+        @default_mapping = (target._csv_mappings || {}).transform_values(&:to_s)
+        @templates = load_templates
+      end
+
+      def load_templates
+        return [] unless defined?(DataPorter::MappingTemplate)
+
+        DataPorter::MappingTemplate.for_target(@import.target_key)
+      end
+
+      def save_column_mapping
+        mapping = params.require(:column_mapping).permit!.to_h
+        merged = (@import.config || {}).merge("column_mapping" => mapping)
+        @import.update!(config: merged, status: :pending)
+      end
+
+      def save_template_if_requested
+        return unless params[:save_template] == "1"
+        return unless defined?(DataPorter::MappingTemplate)
+
+        mapping = params.require(:column_mapping).permit!.to_h
+        DataPorter::MappingTemplate.find_or_initialize_by(
+          target_key: @import.target_key,
+          name: params[:template_name].presence || "Default"
+        ).update!(mapping: mapping)
+      end
+    end
+  end
+end

data/app/controllers/data_porter/concerns/record_pagination.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Concerns
+    module RecordPagination
+      extend ActiveSupport::Concern
+
+      private
+
+      def paginate_records
+        per_page = 50
+        @total_pages = (@records.size.to_f / per_page).ceil
+        @total_pages = 1 if @total_pages.zero?
+        @page = (params[:page] || 1).to_i.clamp(1, @total_pages)
+        @records = @records.slice((@page - 1) * per_page, per_page) || []
+      end
+    end
+  end
+end

data/app/controllers/data_porter/imports_controller.rb

@@ -2,6 +2,10 @@
 
 module DataPorter
   class ImportsController < DataPorter.configuration.parent_controller.constantize
+    include Concerns::ImportValidation
+    include Concerns::MappingManagement
+    include Concerns::RecordPagination
+
     layout "data_porter/application"
 
     before_action :set_import, only: %i[show parse confirm cancel dry_run update_mapping status destroy]
@@ -18,7 +22,7 @@ module DataPorter
     def create
       build_import
 
-      if valid_source_for_target? && valid_file_presence? && @import.save
+      if valid_source_for_target? && valid_file_presence? && valid_import_params? && @import.save
         enqueue_after_create
         redirect_to import_path(@import)
       else
@@ -30,6 +34,7 @@ module DataPorter
       @target = @import.target_class
       @records = @import.records
       @grouped = @records.group_by(&:status)
+      paginate_records
       load_mapping_data if @import.mapping?
     end
 
@@ -91,24 +96,17 @@ module DataPorter
     end
 
     def import_params
-      params.require(:data_import).permit(:target_key, :source_type, :file, config: {})
-    end
-
-    def valid_source_for_target?
-      target = DataPorter::Registry.find(@import.target_key)
-      allowed = target._sources || DataPorter.configuration.enabled_sources
-      return true if allowed.map(&:to_s).include?(@import.source_type.to_s)
-
-      @import.errors.add(:source_type, "#{@import.source_type} is not available for this target")
-      false
+      permitted = params.require(:data_import).permit(:target_key, :source_type, :file, config: {})
+      merge_import_params(permitted)
     end
 
-    def valid_file_presence?
-      return true unless %w[csv json xlsx].include?(@import.source_type)
-      return true if @import.file.attached?
+    def merge_import_params(permitted)
+      nested = params.dig(:data_import, :config, :import_params)
+      return permitted unless nested
 
-      @import.errors.add(:file, "must be attached for #{@import.source_type.upcase} imports")
-      false
+      config = permitted[:config] || {}
+      config["import_params"] = nested.permit!.to_h
+      permitted.merge(config: config)
     end
 
     def enqueue_after_create
@@ -118,37 +116,5 @@ module DataPorter
         DataPorter::ParseJob.perform_later(@import.id)
       end
     end
-
-    def load_mapping_data
-      target = @import.target_class
-      columns = target._columns || []
-      @file_headers = @import.config["file_headers"] || []
-      @target_columns = columns.map { |c| [c.label, c.name.to_s, c.required] }
-      @default_mapping = (target._csv_mappings || {}).transform_values(&:to_s)
-      @templates = load_templates
-    end
-
-    def load_templates
-      return [] unless defined?(DataPorter::MappingTemplate)
-
-      DataPorter::MappingTemplate.for_target(@import.target_key)
-    end
-
-    def save_column_mapping
-      mapping = params.require(:column_mapping).permit!.to_h
-      merged = (@import.config || {}).merge("column_mapping" => mapping)
-      @import.update!(config: merged, status: :pending)
-    end
-
-    def save_template_if_requested
-      return unless params[:save_template] == "1"
-      return unless defined?(DataPorter::MappingTemplate)
-
-      mapping = params.require(:column_mapping).permit!.to_h
-      DataPorter::MappingTemplate.find_or_initialize_by(
-        target_key: @import.target_key,
-        name: params[:template_name].presence || "Default"
-      ).update!(mapping: mapping)
-    end
   end
 end