data_porter 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e95ffce9eec8e4473e77c4da1876da232dd95ade124bc204b867c7862d14938f
-  data.tar.gz: d4ef943174d8374bd272331b94014723d37bf2c444c0edc2a5faad132ba9859f
+  metadata.gz: cf5cdd3a072250ed1b6f8066766f4bb94c0447c80f993de6031f562ef29c969f
+  data.tar.gz: c098c9c856c42c7f0cbdb2f0e4efef8fa298f21bac79ecc8abeb8d0c9435160a
 SHA512:
-  metadata.gz: a3d9ca57d376046a35c8024ab1f1ed15416544737ff44d34b94d6ed1dd36c9c50693a93b80357bff90179e26c1467dbd01f3be872bcf241b8f3fd6eaeabaa49c
-  data.tar.gz: fe570eec3633168b17762849e2387aa7cd56a01bbf62668e7e823b84f333b64d980a7aa55e95732d7ee0c2b7c9f4e36549db10e3aa36b59720de2c7348c45ceb
+  metadata.gz: '08d4f1bc3867112f83a6637129d6e34373e5f5b29913e116cc822fdd150447eb047cac185bb2be5c05180340672ded8c712b8f2e0ec14ca28f71c3c653381898'
+  data.tar.gz: a8808881059e7a84fa37042a52999462c27fca0da39bfdc3828c11e4eab88d095617394443baf6974c49e6f9bb0404dd861b561d6f5f6e3c40aafe5018d0633c

data/CHANGELOG.md

@@ -5,6 +5,55 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-02-07
+
+### Added
+
+- **Standalone engine layout** -- Self-contained HTML layout with importmap, loading Stimulus and Turbo from CDN. The engine no longer depends on the host app's layout or asset pipeline
+- **Turbo Drive** -- All navigation within the engine is now handled by Turbo Drive for instant page transitions
+- **Required field indication** -- Mapping form marks required target fields with `*` and shows a warning listing unmapped required fields
+- **Duplicate mapping detection** -- Visual warning (orange border + message) when two file headers are mapped to the same target field
+- **File validation on create** -- Controller-level validation rejects CSV/JSON/XLSX imports without a file attached, with error message displayed on the form
+- **Hide save-as-template** -- The save-as-template checkbox is hidden when a mapping template is loaded
+- **Import details card** -- Show page displays target, source type, file name, date, and record count
+
+### Changed
+
+- Mapping templates form rewritten with `<select>` elements for target fields instead of plain text inputs
+- Templates index buttons styled as proper `dp-btn--secondary` / `dp-btn--danger`
+- "Back to imports" moved to header as a button across all pages
+- Progress controller uses `Turbo.visit` instead of `window.location.reload`
+- ActionCable loaded via dynamic import with polling fallback (avoids CDN MIME type issues)
+- Controllers return `422 Unprocessable Entity` on form validation errors for Turbo compatibility
+- Rubocop limits relaxed: ClassLength 150, MethodLength 15
+- 280 RSpec examples (up from 265), 0 failures
+
+## [0.3.0] - 2026-02-07
+
+### Added
+
+- **Interactive column mapping** -- File-based imports (CSV/XLSX) now pause on a mapping step where users match file headers to target fields via dropdowns
+- **Header extraction** -- New `ExtractHeadersJob` reads the first row of a file without parsing all data, with `extracting_headers` and `mapping` statuses
+- **Dynamic mapping priority** -- User mapping (from UI) > code mapping (from Target DSL) > auto-map (parameterized headers)
+- **`#headers` method** on `Sources::Csv` and `Sources::Xlsx` for lightweight first-row extraction
+- **`#file_based?` helper** on `DataImport` to distinguish file sources from structured sources
+- **MappingTemplate model** -- Persist reusable column mappings per target (`data_porter_mapping_templates` table)
+- **MappingTemplatesController** -- Full CRUD for managing saved mapping templates
+- **Mapping Phlex components** -- `Mapping::Form`, `Mapping::ColumnRow`, `Mapping::TemplateSelect` for the mapping UI
+- **Stimulus mapping controller** -- Client-side template loading with zero network requests (reads `data-mapping` JSON attributes)
+- **Save-as-template** -- Checkbox in the mapping form to save the current mapping for future imports
+- **Badge styles** for `extracting_headers` and `mapping` statuses
+- **Install generator** now creates `data_porter_mapping_templates` migration
+
+### Changed
+
+- CSS split from monolithic `application.css` into 10 domain-specific stylesheets (base, layout, table, badges, cards, preview, progress, alerts, modal, mapping)
+- Phlex components reorganized into subdirectories: `Shared::`, `Preview::`, `Progress::`, `Mapping::`
+- File-based imports route through `ExtractHeadersJob` instead of `ParseJob` on create
+- `Orchestrator` gains `extract_headers!` method and `build_source` / `store_headers` helpers
+- `Sources::Base#apply_csv_mapping` now checks three mapping sources in priority order
+- 265 RSpec examples (up from 225), 0 failures
+
 ## [0.2.0] - 2026-02-07
 
 ### Added

data/README.md

@@ -1,48 +1,54 @@
 # DataPorter
 
-A mountable Rails engine for 3-step data import workflows: **Upload**, **Preview**, **Import**.
+> [!WARNING]
+> This gem is under active development and not yet production-ready. APIs and features may change without notice.
 
-Supports CSV, JSON, XLSX, and API sources with a declarative DSL for defining import targets. Business-agnostic by design -- all domain logic lives in your host app.
-
-![Import list with status badges](docs/screenshots/index-with-previewing.jpg)
+A mountable Rails engine for data import workflows: **Upload**, **Map**, **Preview**, **Import**.
 
-![New import modal with dropzone](docs/screenshots/modal-new-import.jpg)
+Supports CSV, JSON, XLSX, and API sources with a declarative DSL for defining import targets. Business-agnostic by design -- all domain logic lives in your host app.
 
-![Preview with summary cards and data table](docs/screenshots/preview.jpg)
+<table>
+  <tr>
+    <td><img src="docs/screenshots/index-with-previewing.jpg" width="400" alt="Import list with status badges" /></td>
+    <td><img src="docs/screenshots/modal-new-import.jpg" width="400" alt="New import modal with dropzone" /></td>
+  </tr>
+  <tr>
+    <td><img src="docs/screenshots/mapping.jpg" width="400" alt="Interactive column mapping with templates" /></td>
+    <td><img src="docs/screenshots/preview.jpg" width="400" alt="Preview with summary cards and data table" /></td>
+  </tr>
+</table>
+
+## Features
+
+- **4 source types** -- CSV, XLSX, JSON, and API with a unified parsing pipeline
+- **Interactive column mapping** -- Drag-free UI to match file headers to target fields ([docs](docs/MAPPING.md))
+- **Mapping templates** -- Save and reuse column mappings across imports ([docs](docs/MAPPING.md#mapping-templates))
+- **Real-time progress** -- ActionCable updates with polling fallback
+- **Dry run mode** -- Validate against the database without persisting
+- **Standalone UI** -- Self-contained layout with Turbo Drive and Stimulus, no host app dependencies
+- **Declarative Target DSL** -- One class per import type, zero boilerplate ([docs](docs/TARGETS.md))
 
 ## Requirements
 
 - Ruby >= 3.2
 - Rails >= 7.0
-- ActionCable (for real-time progress updates)
 - ActiveStorage (for file uploads)
+- ActionCable (optional, for real-time progress)
 
 ## Installation
 
-Add the gem to your Gemfile:
-
 ```bash
 bundle add data_porter
-```
-
-Run the install generator:
-
-```bash
 bin/rails generate data_porter:install
-```
-
-This will:
-- Create the migration for `data_porter_imports`
-- Add an initializer at `config/initializers/data_porter.rb`
-- Create the `app/importers/` directory
-- Mount the engine at `/imports`
-
-Run the migration:
-
-```bash
 bin/rails db:migrate
 ```
 
+The generator creates:
+- Migrations for `data_porter_imports` and `data_porter_mapping_templates`
+- An initializer at `config/initializers/data_porter.rb`
+- The `app/importers/` directory
+- Engine mount at `/imports`
+
 ## Quick Start
 
 Generate a target:
@@ -51,11 +57,9 @@ Generate a target:
 bin/rails generate data_porter:target Product name:string:required price:integer sku:string
 ```
 
-Implement the `persist` method in `app/importers/product_target.rb`:
+Implement `persist` in `app/importers/product_target.rb`:
 
 ```ruby
-# frozen_string_literal: true
-
 class ProductTarget < DataPorter::Target
   label "Product"
   model_name "Product"
@@ -63,7 +67,7 @@ class ProductTarget < DataPorter::Target
   sources :csv
 
   columns do
-    column :name,  type: :string,  required: true
+    column :name,  type: :string, required: true
     column :price, type: :integer
     column :sku,   type: :string
   end
@@ -76,377 +80,50 @@ end
 
 Visit `/imports` and start importing.
 
-## Configuration
-
-All options are set in `config/initializers/data_porter.rb`:
-
-```ruby
-DataPorter.configure do |config|
-  # Parent controller for the engine's controllers to inherit from.
-  # Controls authentication, layouts, and helpers.
-  config.parent_controller = "ApplicationController"
-
-  # ActiveJob queue name for import jobs.
-  config.queue_name = :imports
-
-  # ActiveStorage service for uploaded files.
-  config.storage_service = :local
-
-  # ActionCable channel prefix.
-  config.cable_channel_prefix = "data_porter"
-
-  # Context builder: inject business data into targets.
-  # Receives the current controller instance.
-  config.context_builder = ->(controller) {
-    OpenStruct.new(user: controller.current_user)
-  }
-
-  # Maximum number of records displayed in preview.
-  config.preview_limit = 500
-
-  # Enabled source types.
-  config.enabled_sources = %i[csv json api xlsx]
-end
-```
-
-| Option | Default | Description |
-|---|---|---|
-| `parent_controller` | `"ApplicationController"` | Controller class the engine inherits from |
-| `queue_name` | `:imports` | ActiveJob queue for import jobs |
-| `storage_service` | `:local` | ActiveStorage service name |
-| `cable_channel_prefix` | `"data_porter"` | ActionCable stream prefix |
-| `context_builder` | `nil` | Lambda receiving the controller, returns context passed to target methods |
-| `preview_limit` | `500` | Max records shown in the preview step |
-| `enabled_sources` | `%i[csv json api xlsx]` | Source types available in the UI |
-
-## Defining Targets
-
-Targets are plain Ruby classes in `app/importers/` that inherit from `DataPorter::Target`.
-
-### Class-level DSL
-
-```ruby
-class OrderTarget < DataPorter::Target
-  label "Orders"
-  model_name "Order"
-  icon "fas fa-shopping-cart"
-  sources :csv, :json, :api, :xlsx
-
-  columns do
-    column :order_number, type: :string, required: true
-    column :total,        type: :decimal
-    column :placed_at,    type: :date
-    column :active,       type: :boolean
-    column :quantity,     type: :integer
-  end
-
-  csv_mapping do
-    map "Order #" => :order_number
-    map "Total ($)" => :total
-  end
-
-  json_root "data.orders"
-
-  api_config do
-    endpoint "https://api.example.com/orders"
-    headers({ "Authorization" => "Bearer token" })
-    response_root "data.orders"
-  end
-
-  deduplicate_by :order_number
-
-  dry_run_enabled
-
-  # ...
-end
-```
-
-#### `label(value)`
-
-Human-readable name shown in the UI.
-
-#### `model_name(value)`
-
-The ActiveRecord model name this target imports into (for display purposes).
-
-#### `icon(value)`
-
-CSS icon class (e.g. FontAwesome) shown in the UI.
-
-#### `sources(*types)`
-
-Accepted source types: `:csv`, `:json`, `:api`, `:xlsx`.
-
-#### `columns { ... }`
-
-Defines the expected columns for this import. Each column accepts:
-
-| Parameter | Type | Default | Description |
-|---|---|---|---|
-| `name` | Symbol | (required) | Column identifier |
-| `type` | Symbol | `:string` | One of `:string`, `:integer`, `:decimal`, `:boolean`, `:date` |
-| `required` | Boolean | `false` | Whether the column must have a value |
-| `label` | String | Humanized name | Display label in the preview |
-
-#### `csv_mapping { ... }`
-
-Maps CSV header names to column names when they don't match:
-
-```ruby
-csv_mapping do
-  map "First Name" => :first_name
-  map "E-mail" => :email
-end
-```
-
-#### `json_root(path)`
-
-Dot-separated path to the array of records within a JSON document:
-
-```ruby
-json_root "data.users"
-```
-
-Given `{ "data": { "users": [...] } }`, records are extracted from `data.users`.
-
-#### `api_config { ... }`
-
-Configures the API source:
-
-```ruby
-api_config do
-  endpoint "https://api.example.com/records"
-  headers({ "Authorization" => "Bearer token", "Accept" => "application/json" })
-  response_root "data.items"
-end
-```
-
-#### `deduplicate_by(*keys)`
-
-Skip records that share the same value(s) for the given column(s):
-
-```ruby
-deduplicate_by :email
-deduplicate_by :first_name, :last_name
-```
-
-#### `dry_run_enabled`
-
-Enables dry run mode for this target (see [Dry Run](#dry-run)).
-
-### Instance Methods
-
-Override these in your target to customize behavior:
-
-#### `transform(record)`
-
-Transform a record before validation. Must return the (modified) record.
-
-```ruby
-def transform(record)
-  record.attributes["email"] = record.attributes["email"]&.downcase
-  record
-end
-```
-
-#### `validate(record)`
-
-Add custom validation errors to a record:
-
-```ruby
-def validate(record)
-  record.add_error("Email is invalid") unless record.attributes["email"]&.include?("@")
-end
-```
-
-#### `persist(record, context:)`
-
-**Required.** Save the record to your database. Raises `NotImplementedError` if not overridden.
-
-```ruby
-def persist(record, context:)
-  User.create!(record.attributes)
-end
-```
-
-#### `after_import(results, context:)`
-
-Called once after all records have been processed:
-
-```ruby
-def after_import(results, context:)
-  AdminMailer.import_complete(context.user, results).deliver_later
-end
-```
-
-#### `on_error(record, error, context:)`
-
-Called when a record fails to import:
-
-```ruby
-def on_error(record, error, context:)
-  Sentry.capture_exception(error, extra: { record: record.attributes })
-end
-```
-
-## Source Types
-
-### CSV
-
-Upload a CSV file. Configure header mappings with `csv_mapping` when headers don't match your column names.
-
-### XLSX
-
-Upload an Excel `.xlsx` file. Uses the same `csv_mapping` for header-to-column mapping. By default the first sheet is parsed; select a different sheet via config:
-
-```ruby
-import.config = { "sheet_index" => 1 }
-```
-
-Powered by [creek](https://github.com/pythonicrubyist/creek) for streaming, memory-efficient parsing.
-
-### JSON
-
-Upload a JSON file. Use `json_root` to specify the path to the records array. Raw JSON arrays are supported without `json_root`.
-
-### API
-
-Fetch records from an external API endpoint. No file upload is needed -- the engine calls the API directly.
-
-#### Basic usage
-
-```ruby
-api_config do
-  endpoint "https://api.example.com/data"
-  headers({ "Authorization" => "Bearer token" })
-  response_root "results"
-end
-```
-
-| Option | Type | Description |
-|---|---|---|
-| `endpoint` | String or Proc | URL to fetch records from |
-| `headers` | Hash or Proc | HTTP headers sent with the request |
-| `response_root` | String | Key in the JSON response containing the records array (omit for top-level arrays) |
-
-#### Dynamic endpoints and headers
-
-Both `endpoint` and `headers` accept lambdas for runtime values. The endpoint lambda receives the import's `config` hash (populated from the form):
-
-```ruby
-api_config do
-  endpoint ->(params) { "https://api.example.com/events?page=#{params[:page]}" }
-  headers -> { { "Authorization" => "Bearer #{ENV['API_TOKEN']}" } }
-  response_root "data"
-end
-```
-
-#### Example: importing from a paginated API
-
-```ruby
-class EventTarget < DataPorter::Target
-  label "Events"
-  model_name "Event"
-  sources :api
-
-  api_config do
-    endpoint "https://api.example.com/events"
-    headers -> { { "Authorization" => "Bearer #{ENV['EVENTS_API_KEY']}", "Accept" => "application/json" } }
-    response_root "events"
-  end
-
-  columns do
-    column :name,       type: :string, required: true
-    column :date,       type: :date
-    column :venue,      type: :string
-    column :capacity,   type: :integer
-  end
-
-  def persist(record, context:)
-    Event.create!(record.attributes)
-  end
-end
-```
-
-When a user creates an import with source type **API**, the engine skips file upload entirely, calls the configured endpoint, parses the JSON response, and feeds the records through the same preview/validate/import pipeline as CSV and JSON sources.
-
 ## Import Workflow
 
-Each import progresses through these statuses:
-
 ```
+File-based (CSV/XLSX):
+pending -> extracting_headers -> mapping -> parsing -> previewing -> importing -> completed
+
+Non-file (JSON/API):
 pending -> parsing -> previewing -> importing -> completed
-                                             \-> failed
-         pending -> parsing -> dry_running -> previewing
 ```
 
 | Status | Description |
 |---|---|
-| `pending` | Import created, waiting for file/source |
-| `parsing` | Source is being read and records extracted |
-| `previewing` | Records parsed and ready for review |
-| `importing` | Records are being persisted |
+| `pending` | Waiting for processing |
+| `extracting_headers` | Reading file headers for column mapping |
+| `mapping` | Waiting for user to map columns |
+| `parsing` | Records being extracted |
+| `previewing` | Records ready for review |
+| `importing` | Records being persisted |
 | `completed` | All records processed |
-| `failed` | Import encountered a fatal error |
+| `failed` | Fatal error encountered |
 | `dry_running` | Dry run validation in progress |
 
-### Routes
+## Documentation
 
-The engine provides these routes (mounted at your chosen path):
+| Topic | Description |
+|---|---|
+| [Configuration](docs/CONFIGURATION.md) | All options, authentication, context builder, real-time updates |
+| [Targets](docs/TARGETS.md) | DSL reference, columns, hooks, generator |
+| [Sources](docs/SOURCES.md) | CSV, JSON, XLSX, API setup and examples |
+| [Column Mapping](docs/MAPPING.md) | Interactive mapping, templates, priority order |
+
+## Routes
 
 | Method | Path | Action |
 |---|---|---|
 | GET | `/imports` | List imports |
-| GET | `/imports/new` | New import form |
 | POST | `/imports` | Create import |
 | GET | `/imports/:id` | Show import |
-| POST | `/imports/:id/parse` | Parse uploaded source |
-| POST | `/imports/:id/confirm` | Confirm and run import |
+| PATCH | `/imports/:id/update_mapping` | Save column mapping |
+| POST | `/imports/:id/parse` | Parse source |
+| POST | `/imports/:id/confirm` | Run import |
 | POST | `/imports/:id/cancel` | Cancel import |
-| POST | `/imports/:id/dry_run` | Run dry validation |
-
-## Dry Run
-
-When `dry_run_enabled` is declared on a target, a "Dry Run" button appears in the preview step. Dry run executes the full import pipeline (transform, validate, persist) inside a rolled-back transaction, giving you a validation report without modifying the database.
-
-## Real-time Updates
-
-DataPorter broadcasts import progress via ActionCable. The channel streams on:
-
-```
-#{cable_channel_prefix}/imports/#{import_id}
-```
-
-The default prefix is `data_porter`, so a typical stream name is `data_porter/imports/42`.
-
-The engine ships with a Stimulus controller that automatically subscribes to the channel and updates the UI during parsing and importing.
-
-## Generators
-
-### `data_porter:install`
-
-```bash
-bin/rails generate data_porter:install
-```
-
-Sets up the migration, initializer, `app/importers/` directory, and mounts the engine.
-
-### `data_porter:target`
-
-```bash
-bin/rails generate data_porter:target ModelName column:type[:required] ...
-```
-
-Examples:
-
-```bash
-bin/rails generate data_porter:target User email:string:required name:string age:integer
-bin/rails generate data_porter:target Product name:string price:decimal
-```
-
-Column format: `name:type[:required]`
-
-Supported types: `string`, `integer`, `decimal`, `boolean`, `date`.
+| POST | `/imports/:id/dry_run` | Dry run validation |
+| | `/mapping_templates` | Full CRUD for templates |
 
 ## Development
 
@@ -454,18 +131,8 @@ Supported types: `string`, `integer`, `decimal`, `boolean`, `date`.
 git clone https://github.com/SerylLns/data_porter.git
 cd data_porter
 bin/setup
-```
-
-Run the test suite:
-
-```bash
-bundle exec rspec
-```
-
-Run the linter:
-
-```bash
-bundle exec rubocop
+bundle exec rspec     # 280 specs
+bundle exec rubocop   # 0 offenses
 ```
 
 ## License

data/ROADMAP.md

@@ -1,20 +1,33 @@
 # Roadmap
 
-## v2.0 - Planned
+## Completed
 
-### High Priority
+### v0.2.0 -- XLSX Source
+- ~~Parse `.xlsx` files natively via `creek` gem~~
+- ~~Sheet selector via `config["sheet_index"]`~~
+- ~~Same parsing pipeline as CSV~~
+
+### v0.3.0 -- Interactive Column Mapping & Templates
+- ~~Mapping UI: each CSV/XLSX column header gets a dropdown to select the target field~~
+- ~~Save mapping as a reusable template (name + column-to-field pairs)~~
+- ~~Template selector that pre-fills all dropdowns at once~~
+- ~~Stored per-target so each import type has its own template library~~
+- ~~Header extraction step before parsing for file-based sources~~
+- ~~Dynamic mapping priority: user mapping > code mapping > auto-map~~
+
+### v0.4.0 -- Standalone Engine UX
+- ~~Self-contained layout with Stimulus + Turbo Drive via CDN importmap~~
+- ~~Required field indication and duplicate mapping detection~~
+- ~~File validation on create for file-based sources~~
+- ~~Turbo Drive for instant page navigation~~
+- ~~Import details card on show page~~
+- ~~Improved template management UI~~
 
-#### XLSX Source
-- Parse `.xlsx` files natively (via `creek` or `roo` gem)
-- Sheet selector when the file contains multiple sheets
-- Same parsing pipeline as CSV (prerequisite for column mapping)
+---
 
-#### Interactive Column Mapping & Templates
-- Mapping UI on the preview step: each CSV/XLSX column header gets a dropdown to select the target field
-- Auto-suggest based on column name similarity
-- Save mapping as a reusable template (name + column-to-field pairs)
-- Template selector that pre-fills all dropdowns at once
-- Stored per-target so each import type has its own template library
+## Planned
+
+### High Priority
 
 #### Export (reverse workflow)
 - `ExportTarget` DSL mirroring the import Target
@@ -43,6 +56,11 @@
 column :email, type: :email, transform: ->(v) { v.downcase.strip }
 ```
 
+#### Auto-suggest Mapping
+- Fuzzy matching between file headers and target columns
+- Suggest mappings based on Levenshtein distance or string similarity
+- Pre-fill dropdowns with best guesses, user confirms
+
 #### Diff Mode
 - Compare incoming records with existing database data
 - Show what will be created, updated, or left unchanged