data_porter 0.1.0 → 0.4.0

data/docs/CONFIGURATION.md

@@ -0,0 +1,81 @@
+# Configuration
+
+All options are set in `config/initializers/data_porter.rb`:
+
+```ruby
+DataPorter.configure do |config|
+  # Parent controller for the engine's controllers to inherit from.
+  # Controls authentication, layouts, and helpers.
+  config.parent_controller = "ApplicationController"
+
+  # ActiveJob queue name for import jobs.
+  config.queue_name = :imports
+
+  # ActiveStorage service for uploaded files.
+  config.storage_service = :local
+
+  # ActionCable channel prefix.
+  config.cable_channel_prefix = "data_porter"
+
+  # Context builder: inject business data into targets.
+  # Receives the current controller instance.
+  config.context_builder = ->(controller) {
+    OpenStruct.new(user: controller.current_user)
+  }
+
+  # Maximum number of records displayed in preview.
+  config.preview_limit = 500
+
+  # Enabled source types.
+  config.enabled_sources = %i[csv json api xlsx]
+end
+```
+
+## Options reference
+
+| Option | Default | Description |
+|---|---|---|
+| `parent_controller` | `"ApplicationController"` | Controller class the engine inherits from |
+| `queue_name` | `:imports` | ActiveJob queue for import jobs |
+| `storage_service` | `:local` | ActiveStorage service name |
+| `cable_channel_prefix` | `"data_porter"` | ActionCable stream prefix |
+| `context_builder` | `nil` | Lambda receiving the controller, returns context passed to target methods |
+| `preview_limit` | `500` | Max records shown in the preview step |
+| `enabled_sources` | `%i[csv json api xlsx]` | Source types available in the UI |
+
+## Authentication
+
+The engine inherits authentication from `parent_controller`. Set it to your authenticated base controller:
+
+```ruby
+config.parent_controller = "Admin::BaseController"
+```
+
+All engine routes will require the same authentication as your base controller.
+
+## Context builder
+
+The `context_builder` lambda lets you inject business data (current user, tenant, permissions) into target methods (`persist`, `after_import`, `on_error`):
+
+```ruby
+config.context_builder = ->(controller) {
+  OpenStruct.new(
+    user: controller.current_user,
+    organization: controller.current_organization
+  )
+}
+```
+
+The returned object is available as `context` in all target instance methods.
+
+## Real-time updates
+
+DataPorter broadcasts import progress via ActionCable. The channel streams on:
+
+```
+#{cable_channel_prefix}/imports/#{import_id}
+```
+
+The default prefix is `data_porter`, so a typical stream name is `data_porter/imports/42`.
+
+The engine ships with a Stimulus controller that automatically subscribes to the channel and updates a progress bar during parsing and importing. If ActionCable is unavailable, it falls back to polling every 3 seconds.

data/docs/MAPPING.md

@@ -0,0 +1,44 @@
+# Column Mapping
+
+For file-based sources (CSV/XLSX), DataPorter adds an interactive mapping step between upload and parsing. Users see their file's actual column headers and map each one to a target field via dropdowns.
+
+```
+File Header          Target Field
++-----------+        +---------------+
+| Prenom    |   ->   | First Name  v |
++-----------+        +---------------+
++-----------+        +---------------+
+| Nom       |   ->   | Last Name   v |
++-----------+        +---------------+
+```
+
+Dropdowns are pre-filled from the Target's `csv_mapping` when headers match. Users can adjust any mapping before continuing to the preview step.
+
+## Required fields
+
+Required target fields are marked with `*` in the dropdown labels. If any required field is left unmapped, a warning banner appears listing the missing fields. This validation is client-side only -- it warns but does not block submission.
+
+## Duplicate detection
+
+If two file headers are mapped to the same target field, the affected rows are highlighted with an orange border and a warning message appears. This helps catch accidental duplicate mappings before parsing.
+
+## Mapping Templates
+
+Mappings can be saved as reusable templates. When starting a new import, users select a saved template from a dropdown to auto-fill all column mappings at once. Templates are stored per-target, so each import type has its own template library.
+
+### Managing templates
+
+- **Inline**: Check "Save as template" in the mapping form and give it a name
+- **CRUD**: Use the "Mapping Templates" link on the imports index page to create, edit, and delete templates
+
+When a template is loaded, the "Save as template" checkbox is hidden since the user is already working from an existing template.
+
+## Mapping Priority
+
+When parsing, mappings are resolved in priority order:
+
+1. **User mapping** -- from the mapping UI (`config["column_mapping"]`)
+2. **Code mapping** -- from the Target DSL (`csv_mapping`)
+3. **Auto-map** -- parameterize headers to match column names
+
+Non-file sources (JSON, API) skip the mapping step entirely.

data/docs/SOURCES.md

@@ -0,0 +1,94 @@
+# Sources
+
+DataPorter supports four source types. Each source reads data from a different format and feeds it through the same parsing pipeline.
+
+## CSV
+
+Upload a CSV file. Headers are extracted automatically and presented in the [column mapping](MAPPING.md) step. Configure header mappings with `csv_mapping` in your [Target](TARGETS.md) when file headers don't match your column names.
+
+Custom separator:
+
+```ruby
+import.config = { "separator" => ";" }
+```
+
+## XLSX
+
+Upload an Excel `.xlsx` file. Uses the same `csv_mapping` for header-to-column mapping as CSV. By default the first sheet is parsed; select a different sheet via config:
+
+```ruby
+import.config = { "sheet_index" => 1 }
+```
+
+Powered by [creek](https://github.com/pythonicrubyist/creek) for streaming, memory-efficient parsing.
+
+## JSON
+
+Upload a JSON file. Use `json_root` in your Target to specify the path to the records array. Raw JSON arrays are supported without `json_root`.
+
+```ruby
+json_root "data.users"
+```
+
+Given `{ "data": { "users": [...] } }`, records are extracted from `data.users`.
+
+## API
+
+Fetch records from an external API endpoint. No file upload is needed -- the engine calls the API directly.
+
+### Basic usage
+
+```ruby
+api_config do
+  endpoint "https://api.example.com/data"
+  headers({ "Authorization" => "Bearer token" })
+  response_root "results"
+end
+```
+
+| Option | Type | Description |
+|---|---|---|
+| `endpoint` | String or Proc | URL to fetch records from |
+| `headers` | Hash or Proc | HTTP headers sent with the request |
+| `response_root` | String | Key in the JSON response containing the records array (omit for top-level arrays) |
+
+### Dynamic endpoints and headers
+
+Both `endpoint` and `headers` accept lambdas for runtime values. The endpoint lambda receives the import's `config` hash:
+
+```ruby
+api_config do
+  endpoint ->(params) { "https://api.example.com/events?page=#{params[:page]}" }
+  headers -> { { "Authorization" => "Bearer #{ENV['API_TOKEN']}" } }
+  response_root "data"
+end
+```
+
+### Full example
+
+```ruby
+class EventTarget < DataPorter::Target
+  label "Events"
+  model_name "Event"
+  sources :api
+
+  api_config do
+    endpoint "https://api.example.com/events"
+    headers -> { { "Authorization" => "Bearer #{ENV['EVENTS_API_KEY']}" } }
+    response_root "events"
+  end
+
+  columns do
+    column :name,     type: :string, required: true
+    column :date,     type: :date
+    column :venue,    type: :string
+    column :capacity, type: :integer
+  end
+
+  def persist(record, context:)
+    Event.create!(record.attributes)
+  end
+end
+```
+
+When a user creates an import with source type **API**, the engine skips file upload entirely, calls the configured endpoint, parses the JSON response, and feeds the records through the same preview/validate/import pipeline as file-based sources.

data/docs/TARGETS.md

@@ -0,0 +1,176 @@
+# Targets
+
+Targets are plain Ruby classes in `app/importers/` that inherit from `DataPorter::Target`. Each target defines one import type: its columns, sources, mappings, and persistence logic.
+
+## Generator
+
+```bash
+bin/rails generate data_porter:target ModelName column:type[:required] ...
+```
+
+Examples:
+
+```bash
+bin/rails generate data_porter:target User email:string:required name:string age:integer
+bin/rails generate data_porter:target Product name:string price:decimal
+```
+
+Column format: `name:type[:required]`
+
+Supported types: `string`, `integer`, `decimal`, `boolean`, `date`.
+
+## Class-level DSL
+
+```ruby
+class OrderTarget < DataPorter::Target
+  label "Orders"
+  model_name "Order"
+  icon "fas fa-shopping-cart"
+  sources :csv, :json, :api, :xlsx
+
+  columns do
+    column :order_number, type: :string, required: true
+    column :total,        type: :decimal
+    column :placed_at,    type: :date
+    column :active,       type: :boolean
+    column :quantity,     type: :integer
+  end
+
+  csv_mapping do
+    map "Order #" => :order_number
+    map "Total ($)" => :total
+  end
+
+  json_root "data.orders"
+
+  api_config do
+    endpoint "https://api.example.com/orders"
+    headers({ "Authorization" => "Bearer token" })
+    response_root "data.orders"
+  end
+
+  deduplicate_by :order_number
+
+  dry_run_enabled
+end
+```
+
+### `label(value)`
+
+Human-readable name shown in the UI.
+
+### `model_name(value)`
+
+The ActiveRecord model name this target imports into (for display purposes).
+
+### `icon(value)`
+
+CSS icon class (e.g. FontAwesome) shown in the UI.
+
+### `sources(*types)`
+
+Accepted source types: `:csv`, `:json`, `:api`, `:xlsx`.
+
+### `columns { ... }`
+
+Defines the expected columns for this import. Each column accepts:
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `name` | Symbol | (required) | Column identifier |
+| `type` | Symbol | `:string` | One of `:string`, `:integer`, `:decimal`, `:boolean`, `:date` |
+| `required` | Boolean | `false` | Whether the column must have a value |
+| `label` | String | Humanized name | Display label in the preview |
+
+### `csv_mapping { ... }`
+
+Maps CSV/XLSX header names to column names when they don't match:
+
+```ruby
+csv_mapping do
+  map "First Name" => :first_name
+  map "E-mail" => :email
+end
+```
+
+### `json_root(path)`
+
+Dot-separated path to the array of records within a JSON document:
+
+```ruby
+json_root "data.users"
+```
+
+Given `{ "data": { "users": [...] } }`, records are extracted from `data.users`.
+
+### `api_config { ... }`
+
+See [Sources: API](SOURCES.md#api) for full documentation.
+
+### `deduplicate_by(*keys)`
+
+Skip records that share the same value(s) for the given column(s):
+
+```ruby
+deduplicate_by :email
+deduplicate_by :first_name, :last_name
+```
+
+### `dry_run_enabled`
+
+Enables dry run mode for this target. A "Dry Run" button appears in the preview step. Dry run executes the full import pipeline (transform, validate, persist) inside a rolled-back transaction, giving a validation report without modifying the database.
+
+## Instance Methods
+
+Override these in your target to customize behavior.
+
+### `transform(record)`
+
+Transform a record before validation. Must return the (modified) record.
+
+```ruby
+def transform(record)
+  record.attributes["email"] = record.attributes["email"]&.downcase
+  record
+end
+```
+
+### `validate(record)`
+
+Add custom validation errors to a record:
+
+```ruby
+def validate(record)
+  record.add_error("Email is invalid") unless record.attributes["email"]&.include?("@")
+end
+```
+
+### `persist(record, context:)`
+
+**Required.** Save the record to your database. Raises `NotImplementedError` if not overridden.
+
+```ruby
+def persist(record, context:)
+  User.create!(record.attributes)
+end
+```
+
+### `after_import(results, context:)`
+
+Called once after all records have been processed:
+
+```ruby
+def after_import(results, context:)
+  AdminMailer.import_complete(context.user, results).deliver_later
+end
+```
+
+### `on_error(record, error, context:)`
+
+Called when a record fails to import:
+
+```ruby
+def on_error(record, error, context:)
+  Sentry.capture_exception(error, extra: { record: record.attributes })
+end
+```

data/lib/data_porter/components/mapping/column_row.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    module Mapping
+      class ColumnRow < Base
+        def initialize(file_header:, target_fields:, selected: nil)
+          super()
+          @file_header = file_header
+          @target_fields = target_fields
+          @selected = selected
+        end
+
+        def view_template
+          div(class: "dp-mapping-row") do
+            render_header
+            render_arrow
+            render_select
+          end
+        end
+
+        private
+
+        def render_header
+          span(class: "dp-mapping-row__header") { @file_header }
+        end
+
+        def render_arrow
+          span(class: "dp-mapping-row__arrow") { "→" }
+        end
+
+        def render_select
+          select(
+            name: "column_mapping[#{@file_header}]",
+            class: "dp-select dp-mapping-row__select",
+            data_data_porter__mapping_target: "columnSelect",
+            data_action: "change->data-porter--mapping#onChange"
+          ) do
+            option(value: "") { "Skip this column" }
+            @target_fields.each { |label, value, required| render_field_option(label, value, required) }
+          end
+        end
+
+        def render_field_option(label, value, required)
+          attrs = { value: value, selected: value == @selected }
+          attrs[:data_required] = "true" if required
+          option(**attrs) { required ? "#{label} *" : label }
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components/mapping/form.rb

@@ -0,0 +1,127 @@
+# frozen_string_literal: true
+
+require "json"
+
+module DataPorter
+  module Components
+    module Mapping
+      class Form < Base
+        def initialize(import:, action_url:, **options)
+          super()
+          @import = import
+          @action_url = action_url
+          @csrf_token = options[:csrf_token]
+          @file_headers = options.fetch(:file_headers)
+          @target_columns = options.fetch(:target_columns)
+          @templates = options.fetch(:templates)
+          @default_mapping = options.fetch(:default_mapping)
+        end
+
+        def view_template
+          form(
+            action: @action_url,
+            method: "post",
+            class: "dp-mapping-form",
+            data_controller: "data-porter--mapping",
+            data_data_porter__mapping_required_columns_value: required_columns_json
+          ) { render_form_body }
+        end
+
+        private
+
+        def render_form_body
+          render_csrf_token
+          render_method_override
+          render_template_section
+          render_warning("required")
+          render_warning("duplicate")
+          render_column_rows
+          render_save_template
+          render_actions
+        end
+
+        def render_csrf_token
+          return unless @csrf_token
+
+          input(type: "hidden", name: "authenticity_token", value: @csrf_token)
+        end
+
+        def render_method_override
+          input(type: "hidden", name: "_method", value: "patch")
+        end
+
+        def render_template_section
+          return if @templates.empty?
+
+          div(class: "dp-field") do
+            label(class: "dp-label") { "Load Template" }
+            render TemplateSelect.new(templates: @templates)
+          end
+        end
+
+        def render_warning(type)
+          div(
+            class: "dp-mapping-#{type}-warning",
+            data_data_porter__mapping_target: "#{type}Warning",
+            style: "display: none;"
+          ) { "" }
+        end
+
+        def render_column_rows
+          div(class: "dp-mapping-rows") do
+            @file_headers.each { |header| render_row(header) }
+          end
+        end
+
+        def render_row(header)
+          selected = @default_mapping[header]
+          render ColumnRow.new(
+            file_header: header,
+            target_fields: @target_columns,
+            selected: selected
+          )
+        end
+
+        def render_save_template
+          div(
+            class: "dp-field",
+            style: "margin-top: 1.5rem;",
+            data_data_porter__mapping_target: "saveTemplate"
+          ) do
+            render_template_checkbox
+            render_template_name_input
+          end
+        end
+
+        def render_template_checkbox
+          label(style: "display: flex; align-items: center; gap: 0.5rem;") do
+            input(type: "checkbox", name: "save_template", value: "1")
+            span { "Save as template" }
+          end
+        end
+
+        def render_template_name_input
+          input(
+            type: "text", name: "template_name",
+            placeholder: "Template name",
+            class: "dp-select",
+            style: "margin-top: 0.5rem;"
+          )
+        end
+
+        def render_actions
+          div(class: "dp-actions") do
+            button(type: "submit", class: "dp-btn dp-btn--primary") { "Continue" }
+          end
+        end
+
+        def required_columns_json
+          @target_columns
+            .select { |_, _, required| required }
+            .map { |label, name, _| { label: label, name: name } }
+            .to_json
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components/mapping/template_select.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    module Mapping
+      class TemplateSelect < Base
+        def initialize(templates:, selected_id: nil)
+          super()
+          @templates = templates
+          @selected_id = selected_id
+        end
+
+        def view_template
+          select(
+            class: "dp-select dp-mapping-template",
+            data_action: "change->data-porter--mapping#loadTemplate"
+          ) do
+            option(value: "") { "Select a template..." }
+            @templates.each { |t| render_option(t) }
+          end
+        end
+
+        private
+
+        def render_option(template)
+          option(
+            value: template.id.to_s,
+            selected: template.id == @selected_id,
+            data_mapping: template.mapping.to_json
+          ) { template.name }
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components/preview/results_summary.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    module Preview
+      class ResultsSummary < Base
+        def initialize(report:)
+          super()
+          @report = report
+        end
+
+        def view_template
+          div(class: "dp-results") do
+            p { "Created: #{@report.imported_count}" }
+            p { "Errors: #{@report.errored_count}" }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components/preview/summary_cards.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    module Preview
+      class SummaryCards < Base
+        def initialize(report:)
+          super()
+          @report = report
+        end
+
+        def view_template
+          div(class: "dp-summary-cards") do
+            card("dp-card--complete", @report.complete_count, "Ready")
+            card("dp-card--partial", @report.partial_count, "Incomplete")
+            card("dp-card--missing", @report.missing_count, "Missing")
+            card("dp-card--duplicate", @report.duplicate_count, "Duplicates")
+          end
+        end
+
+        private
+
+        def card(css_class, count, label)
+          div(class: "dp-card #{css_class}") do
+            strong { count.to_s }
+            plain " #{label}"
+          end
+        end
+      end
+    end
+  end
+end