data_porter 0.9.0 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 20e2b579cf4078a611095abd155090d7754e1e33e044a022136c846edb793bcb
-  data.tar.gz: 58e1bcad055c198aab6ccb71ddbbb5b219d5b82a7a94a34fb8c064f109d37878
+  metadata.gz: 3f4f148347707854fe641913e1e21616ca6ed93b8e6c5ddcd0afb2421ddba7dc
+  data.tar.gz: e1ac18c356d7edcca901c54f1349d003e5a9b31d9feca5aed1ac51717402bbeb
 SHA512:
-  metadata.gz: 9929013331242330b53fe9f5e4b4940600a339c7e2d723551e864a969406cd2c516e70a10c95eff25c820b7f7e5904b63da4156b473d140e261d3a65ed9811eb
-  data.tar.gz: c9be1bbff65d0da5b36b5a837f590514ed29eb911a4ba527baa3e2f228f2d27252a4f4ff9bff3af579ac6674b6be19b32f0f4bf23aab992d42291877ea019e56
+  metadata.gz: 4e96e22590744496c31aa51060cb1c043365eabbcefa9d17f97edc8eba18ae045fd598a6483aa57fad35b3524b120a2ee358c3103a36fbc44f595ff5eda23bc8
+  data.tar.gz: 5dbc857197bafd89108528aa95ee12617c5ee7e09b9784bbeb924442d11e07300f469a53d72ec44c0c39099836d2b24ac9a7d616422ddbc7d68fcbcbc06606f4

data/CHANGELOG.md

@@ -5,6 +5,48 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.1] - 2026-02-07
+
+### Added
+
+- **CSV delimiter auto-detection** -- Automatically detect `,` `;` `\t` separators via frequency analysis on the first line; explicit `col_sep` config still takes precedence
+- **CSV encoding auto-detection** -- Detect and transcode Latin-1 / ISO-8859-1 content to UTF-8; strip UTF-8 BOM when present
+
+### Fixed
+
+- **`param.collection` accepts arrays** -- `Registry.serialize_param` now duck-types with `respond_to?(:call)` so both lambdas and plain arrays work
+- **`dp-input` styling** -- Text inputs now share the same CSS rules as `dp-select` and `dp-file-input`
+- **Migration template nullable user** -- Removed `null: false` from polymorphic `user` reference so the engine works without authentication
+- **Skipped records visible in results** -- Added "Skipped" stat card for missing + partial records; title reflects errors; export rejects button includes all rejected rows
+- **Hidden param label removed** -- `type: :hidden` params no longer render a label or wrapper div
+
+### Changed
+
+- 402 RSpec examples (up from 391), 0 failures
+
+## [1.0.0] - 2026-02-07
+
+### Added
+
+- **Max records guard** -- `config.max_records` (default: 10,000) rejects files exceeding the limit before parsing
+- **Transaction mode** -- `config.transaction_mode` (`:per_record` or `:all`); `:all` wraps entire import in a single transaction that rolls back on any failure
+- **Fallback headers** -- Auto-generate `col_1, col_2...` when CSV/XLSX header row is empty
+- **Reject rows CSV export** -- Download CSV of failed/errored records with original data + error messages after import; available when `errored_count > 0`
+- **E2E specs** -- 6 end-to-end integration tests covering all source types (CSV, XLSX, JSON, API), import params, and reject rows export
+
+### Fixed
+
+- **Import params whitelist** -- `merge_import_params` now permits only param names declared in the Target DSL instead of using `permit!`
+- **Column mapping whitelist** -- `permitted_column_mapping` filters mapping values to valid target column names; invalid values replaced with `""`
+- **File size validation** -- Uploads exceeding `config.max_file_size` (default: 10 MB) are rejected before save
+- **MIME type validation** -- Uploaded files must match allowed content types per source (CSV: `text/csv`, `text/plain`; JSON: `application/json`, `text/plain`; XLSX: OpenXML spreadsheet)
+- **XSS in template form** -- Replaced `innerHTML` with safe DOM methods in `template_form_controller.js`
+
+### Changed
+
+- Validation chain refactored to `all_validations_pass?` using `.all?` to collect all errors at once instead of short-circuiting
+- 391 RSpec examples (up from 354), 0 failures
+
 ## [0.9.0] - 2026-02-07
 
 ### Added

data/README.md

@@ -30,6 +30,9 @@ Supports CSV, JSON, XLSX, and API sources with a declarative DSL for defining im
 - **Import params** -- Declare extra form fields (select, text, number, hidden) per target for scoped imports ([docs](docs/TARGETS.md#params--))
 - **Per-target source filtering** -- Each target declares its allowed sources, the UI filters accordingly
 - **Import deletion & auto-purge** -- Delete imports from the UI, or schedule `rake data_porter:purge` for automatic cleanup
+- **Reject rows export** -- Download a CSV of failed/errored records with error messages after import
+- **Security validations** -- File size limit, MIME type check, strong parameter whitelisting
+- **Safety guards** -- Max records limit (`config.max_records`), configurable transaction mode (`:per_record` or `:all`)
 - **Declarative Target DSL** -- One class per import type, zero boilerplate ([docs](docs/TARGETS.md))
 
 ## Requirements
@@ -129,6 +132,7 @@ pending -> parsing -> previewing -> importing -> completed
 | POST | `/imports/:id/confirm` | Run import |
 | POST | `/imports/:id/cancel` | Cancel import |
 | POST | `/imports/:id/dry_run` | Dry run validation |
+| GET | `/imports/:id/export_rejects` | Download rejects CSV |
 | | `/mapping_templates` | Full CRUD for templates |
 
 ## Development
@@ -137,7 +141,7 @@ pending -> parsing -> previewing -> importing -> completed
 git clone https://github.com/SerylLns/data_porter.git
 cd data_porter
 bin/setup
-bundle exec rspec     # 354 specs
+bundle exec rspec     # 391 specs
 bundle exec rubocop   # 0 offenses
 ```
 

data/app/assets/javascripts/data_porter/import_form_controller.js

@@ -32,6 +32,7 @@ export default class extends Controller {
   }
 
   buildParamField(p) {
+    if (p.type === "hidden") return this.buildInput(p)
     var div = document.createElement("div")
     div.className = "dp-field"
     div.appendChild(this.buildLabel(p))

data/app/assets/javascripts/data_porter/template_form_controller.js

@@ -18,7 +18,8 @@ export default class extends Controller {
     const pair = document.createElement("div")
     pair.className = "dp-mapping-pair"
     pair.style.cssText = "display: flex; gap: 0.5rem; margin-bottom: 0.5rem;"
-    pair.innerHTML = this.pairHTML(columns)
+    pair.appendChild(this.buildKeyInput())
+    pair.appendChild(this.buildValueSelect(columns))
     container.appendChild(pair)
   }
 
@@ -34,13 +35,35 @@ export default class extends Controller {
     })
   }
 
-  pairHTML(columns) {
-    const options = columns.map(([label, name]) =>
-      `<option value="${name}">${label}</option>`
-    ).join("")
+  buildKeyInput() {
+    const input = document.createElement("input")
+    input.type = "text"
+    input.name = "mapping_template[mapping_keys][]"
+    input.placeholder = "File header"
+    input.className = "dp-select"
+    input.style.flex = "1"
+    return input
+  }
+
+  buildValueSelect(columns) {
+    const select = document.createElement("select")
+    select.name = "mapping_template[mapping_values][]"
+    select.className = "dp-select"
+    select.style.flex = "1"
+    select.dataset.dataPorterTemplateFormTarget = "fieldSelect"
+
+    const blank = document.createElement("option")
+    blank.value = ""
+    blank.textContent = "Select a field..."
+    select.appendChild(blank)
+
+    columns.forEach(([label, name]) => {
+      const opt = document.createElement("option")
+      opt.value = name
+      opt.textContent = label
+      select.appendChild(opt)
+    })
 
-    return `<input type="text" name="mapping_template[mapping_keys][]" placeholder="File header" class="dp-select" style="flex: 1;" />` +
-      `<select name="mapping_template[mapping_values][]" class="dp-select" style="flex: 1;" data-data-porter--template-form-target="fieldSelect">` +
-      `<option value="">Select a field...</option>${options}</select>`
+    return select
   }
 }

data/app/assets/stylesheets/data_porter/alerts.css

@@ -32,7 +32,7 @@
   display: grid;
   grid-template-columns: repeat(auto-fit, minmax(120px, 1fr));
   gap: 1rem;
-  max-width: 400px;
+  max-width: 500px;
   margin: 0 auto;
 }
 
@@ -60,6 +60,7 @@
 
 .dp-results__stat--success strong { color: var(--dp-success); }
 .dp-results__stat--error strong { color: var(--dp-danger); }
+.dp-results__stat--warning strong { color: var(--dp-warning); }
 
 .dp-results__duration {
   margin-top: 1rem;

data/app/assets/stylesheets/data_porter/layout.css

@@ -29,7 +29,7 @@
   color: var(--dp-gray-700);
 }
 
-.dp-select, .dp-file-input {
+.dp-select, .dp-input, .dp-file-input {
   display: block;
   width: 100%;
   padding: 0.625rem 0.875rem;
@@ -50,7 +50,7 @@
   padding-right: 2.5rem;
 }
 
-.dp-select:focus, .dp-file-input:focus {
+.dp-select:focus, .dp-input:focus, .dp-file-input:focus {
   outline: none;
   border-color: var(--dp-primary);
   box-shadow: 0 0 0 3px rgba(79, 70, 229, 0.15);

data/app/controllers/data_porter/concerns/import_validation.rb

@@ -5,6 +5,12 @@ module DataPorter
     module ImportValidation
       extend ActiveSupport::Concern
 
+      ALLOWED_CONTENT_TYPES = {
+        "csv" => %w[text/csv text/plain],
+        "json" => %w[application/json text/plain],
+        "xlsx" => %w[application/vnd.openxmlformats-officedocument.spreadsheetml.sheet]
+      }.freeze
+
       private
 
       def valid_source_for_target?
@@ -42,6 +48,29 @@ module DataPorter
       def import_param_values
         (@import.config || {}).fetch("import_params", {})
       end
+
+      def valid_file_size?
+        return true unless @import.file.attached?
+
+        max = DataPorter.configuration.max_file_size
+        return true if @import.file.blob.byte_size <= max
+
+        @import.errors.add(:file, "is too large (max #{max / 1.megabyte} MB)")
+        false
+      end
+
+      def valid_file_content_type?
+        return true unless @import.file.attached?
+
+        allowed = ALLOWED_CONTENT_TYPES[@import.source_type]
+        return true unless allowed
+
+        content_type = @import.file.blob.content_type
+        return true if allowed.include?(content_type)
+
+        @import.errors.add(:file, "has an invalid content type (#{content_type})")
+        false
+      end
     end
   end
 end

data/app/controllers/data_porter/concerns/mapping_management.rb

@@ -23,8 +23,7 @@ module DataPorter
       end
 
       def save_column_mapping
-        mapping = params.require(:column_mapping).permit!.to_h
-        merged = (@import.config || {}).merge("column_mapping" => mapping)
+        merged = (@import.config || {}).merge("column_mapping" => permitted_column_mapping)
         @import.update!(config: merged, status: :pending)
       end
 
@@ -32,11 +31,21 @@ module DataPorter
         return unless params[:save_template] == "1"
         return unless defined?(DataPorter::MappingTemplate)
 
-        mapping = params.require(:column_mapping).permit!.to_h
         DataPorter::MappingTemplate.find_or_initialize_by(
           target_key: @import.target_key,
           name: params[:template_name].presence || "Default"
-        ).update!(mapping: mapping)
+        ).update!(mapping: permitted_column_mapping)
+      end
+
+      def permitted_column_mapping
+        raw = params.require(:column_mapping).permit!.to_h
+        valid_names = valid_column_names
+        raw.transform_values { |v| valid_names.include?(v) ? v : "" }
+      end
+
+      def valid_column_names
+        columns = @import.target_class._columns || []
+        columns.to_set { |c| c.name.to_s }
       end
     end
   end

data/app/controllers/data_porter/imports_controller.rb

@@ -8,7 +8,7 @@ module DataPorter
 
     layout "data_porter/application"
 
-    before_action :set_import, only: %i[show parse confirm cancel dry_run update_mapping status destroy]
+    before_action :set_import, only: %i[show parse confirm cancel dry_run update_mapping status export_rejects destroy]
     before_action :load_targets, only: %i[index new create]
 
     def index
@@ -22,7 +22,7 @@ module DataPorter
     def create
       build_import
 
-      if valid_source_for_target? && valid_file_presence? && valid_import_params? && @import.save
+      if all_validations_pass? && @import.save
         enqueue_after_create
         redirect_to import_path(@import)
       else
@@ -73,6 +73,12 @@ module DataPorter
       render json: { status: @import.status, progress: progress }
     end
 
+    def export_rejects
+      columns = @import.target_class._columns || []
+      csv = RejectsCsvBuilder.new(columns, @import.records).generate
+      send_data csv, filename: "rejects_import_#{@import.id}.csv", type: "text/csv"
+    end
+
     def destroy
       @import.file.purge if @import.file.attached?
       @import.destroy!
@@ -95,6 +101,16 @@ module DataPorter
       @import.status = :pending
     end
 
+    def all_validations_pass?
+      [
+        valid_source_for_target?,
+        valid_file_presence?,
+        valid_file_size?,
+        valid_file_content_type?,
+        valid_import_params?
+      ].all?
+    end
+
     def import_params
       permitted = params.require(:data_import).permit(:target_key, :source_type, :file, config: {})
       merge_import_params(permitted)
@@ -104,11 +120,19 @@ module DataPorter
       nested = params.dig(:data_import, :config, :import_params)
       return permitted unless nested
 
-      config = permitted[:config] || {}
-      config["import_params"] = nested.permit!.to_h
+      config = permitted[:config]&.to_unsafe_h || {}
+      config["import_params"] = nested.permit(*allowed_param_keys).to_h
       permitted.merge(config: config)
     end
 
+    def allowed_param_keys
+      target_key = params.dig(:data_import, :target_key)
+      return [] unless target_key
+
+      target = DataPorter::Registry.find(target_key)
+      (target._params || []).map { |p| p.name.to_s }
+    end
+
     def enqueue_after_create
       if @import.file_based?
         DataPorter::ExtractHeadersJob.perform_later(@import.id)

data/app/views/data_porter/imports/show.html.erb

@@ -86,6 +86,10 @@
     <% end %>
     <div class="dp-actions">
       <%= link_to "Back to imports", imports_path, class: "dp-btn dp-btn--primary" %>
+      <% rejected = @import.report.errored_count.to_i + @import.report.missing_count.to_i + @import.report.partial_count.to_i %>
+      <% if rejected.positive? %>
+        <%= link_to "Download rejects CSV", export_rejects_import_path(@import), class: "dp-btn dp-btn--secondary" %>
+      <% end %>
       <%= button_to "Delete", import_path(@import),
             method: :delete, class: "dp-btn dp-btn--danger",
             data: { turbo_confirm: "Delete this import?" } %>

data/config/routes.rb

@@ -9,6 +9,7 @@ DataPorter::Engine.routes.draw do
       post :dry_run
       patch :update_mapping
       get :status
+      get :export_rejects
     end
   end
 

data/docs/CONFIGURATION.md

@@ -18,9 +18,9 @@ DataPorter.configure do |config|
   config.cable_channel_prefix = "data_porter"
 
   # Context builder: inject business data into targets.
-  # Receives the current controller instance.
-  config.context_builder = ->(controller) {
-    OpenStruct.new(user: controller.current_user)
+  # Receives the DataImport record.
+  config.context_builder = ->(data_import) {
+    { user: data_import.user }
   }
 
   # Maximum number of records displayed in preview.
@@ -32,6 +32,18 @@ DataPorter.configure do |config|
   # Auto-purge completed/failed imports older than this duration.
   # Set to nil to disable. Run `rake data_porter:purge` manually or via cron.
   config.purge_after = 60.days
+
+  # Maximum file size for uploads (default: 10 MB).
+  config.max_file_size = 10.megabytes
+
+  # Maximum number of records per import (default: 10,000).
+  # Set to nil to disable.
+  config.max_records = 10_000
+
+  # Transaction mode for imports.
+  # :per_record -- each record persisted independently (default)
+  # :all -- single transaction, rolls back entirely on any failure
+  config.transaction_mode = :per_record
 end
 ```
 
@@ -43,10 +55,13 @@ end
 | `queue_name` | `:imports` | ActiveJob queue for import jobs |
 | `storage_service` | `:local` | ActiveStorage service name |
 | `cable_channel_prefix` | `"data_porter"` | ActionCable stream prefix |
-| `context_builder` | `nil` | Lambda receiving the controller, returns context passed to target methods |
+| `context_builder` | `nil` | Lambda receiving the DataImport record, returns context passed to target methods |
 | `preview_limit` | `500` | Max records shown in the preview step |
 | `enabled_sources` | `%i[csv json api xlsx]` | Source types available in the UI |
 | `purge_after` | `60.days` | Auto-purge completed/failed imports older than this duration |
+| `max_file_size` | `10.megabytes` | Maximum file size for uploads |
+| `max_records` | `10_000` | Maximum number of records per import (nil to disable) |
+| `transaction_mode` | `:per_record` | `:per_record` or `:all` (single transaction rollback) |
 
 ## Authentication
 
@@ -60,14 +75,11 @@ All engine routes will require the same authentication as your base controller.
 
 ## Context builder
 
-The `context_builder` lambda lets you inject business data (current user, tenant, permissions) into target methods (`persist`, `after_import`, `on_error`):
+The `context_builder` lambda lets you inject business data (current user, tenant, permissions) into target methods (`persist`, `after_import`, `on_error`). It receives the `DataImport` record:
 
 ```ruby
-config.context_builder = ->(controller) {
-  OpenStruct.new(
-    user: controller.current_user,
-    organization: controller.current_organization
-  )
+config.context_builder = ->(data_import) {
+  { user: data_import.user, import_id: data_import.id }
 }
 ```
 

data/docs/ROADMAP.md

@@ -1,6 +1,6 @@
 # Roadmap
 
-## v1.0 — Production-ready
+## v1.0 — Production-ready DONE
 
 The goal is a gem that handles real-world imports reliably at scale.
 
@@ -15,14 +15,185 @@ Implemented in v0.9.0. Targets declare `params` with a DSL (`:select`, `:text`,
 `:number`, `:hidden`). Values stored in `config["import_params"]`, accessible
 via `import_params` in all target instance methods. See [Targets docs](TARGETS.md#params--).
 
+### ~~3. Security audit~~ DONE
+
+- Replaced `permit!` on import params and column mapping with whitelists
+- File size validation (`config.max_file_size`, default 10 MB)
+- MIME type validation per source type
+- XSS fix in template form controller (safe DOM methods)
+
+### ~~4. Safety guards~~ DONE
+
+- Max records guard (`config.max_records`, default 10,000)
+- Transaction mode (`config.transaction_mode`: `:per_record` or `:all`)
+- Fallback headers (auto-generate `col_1, col_2...` for empty header rows)
+
+### ~~5. Reject rows export~~ DONE
+
+Download CSV of failed/errored records with original data + error messages.
+Zero-dependency streaming via `send_data`.
+
+### ~~6. E2E integration tests~~ DONE
+
+6 end-to-end specs covering all source types (CSV, XLSX, JSON, API),
+import params flow, and reject rows CSV export. 395 specs total.
+
+---
+
+## v1.1 — Deploy-ready
+
+Priority: features required to deploy DataPorter to real users.
+
+### Scoped imports
+
+Wire up `config.scope` so each user only sees their own imports. The
+configuration hook already exists but isn't connected to the controller query:
+
+```ruby
+DataPorter.configure do |config|
+  config.scope = ->(user) { { user_type: user.class.name, user_id: user.id } }
+end
+```
+
+The controller `index` and `show` actions apply the scope automatically.
+Combined with `parent_controller` inheriting from an authenticated controller,
+this enables full multi-tenant isolation — suitable for both B2B (tenant per
+organization) and B2C (user-level) scenarios.
+
+### Preview ↔ Mapping navigation
+
+Allow users to go back from preview to the mapping step and adjust column
+mapping without restarting the import. Currently the flow is one-way:
+mapping → parse → preview. Adding a "Back to mapping" button on the preview
+page would let users correct mapping mistakes after seeing the parsed data.
+
+### ~~CSV auto-detect: delimiter & encoding~~ DONE
+
+Implemented in v1.0.1. Auto-detect CSV delimiter (`,` `;` `\t`) via frequency
+analysis on the first line. Auto-detect file encoding: strip UTF-8 BOM, validate
+UTF-8, fallback to ISO-8859-1 transcoding. Explicit `col_sep` config takes
+precedence.
+
+### Column mapping for JSON and API sources
+
+The interactive column mapping step currently only works for file-based sources
+(CSV, XLSX). JSON and API sources have stable, predictable keys that rarely need
+remapping, but supporting mapping for all sources would provide a consistent UX.
+
+---
+
+## v1.2 — Smart imports
+
+### Dry-run performance estimate
+
+After a dry run, display an estimated import time based on average record
+processing speed: "Estimated import time: ~2m30s". Helps users decide whether
+to launch the import now or schedule it for off-peak hours.
+
+### Permissions / RBAC
+
+Role-based access control for import operations. Allow host apps to restrict
+who can create imports, confirm imports, or access specific targets. Integrate
+with existing authorization frameworks (Pundit, CanCanCan) via a configurable
+policy hook.
+
+### Column transformers
+
+Built-in transformation pipeline applied per-column before the target's
+`transform` method. Declarative DSL in the target:
+
+```ruby
+columns do
+  column :email, type: :email, transform: [:strip, :downcase]
+  column :phone, type: :string, transform: [:strip, :normalize_phone]
+  column :born_on, type: :date, transform: [:parse_date]
+end
+```
+
+Ships with common transformers (`strip`, `downcase`, `titleize`,
+`normalize_phone`, `parse_date`). Custom transformers via a registry.
+
+### Auto-map heuristics
+
+Smart column mapping suggestions using tokenized header matching and synonym
+dictionaries. When a CSV has "E-mail Address", auto-suggest mapping to `:email`.
+Built-in synonyms for common patterns (phone → phone_number,
+first name → first_name). Configurable synonym lists per target.
+
 ---
 
-## v2+ (future)
+## v2.0 — Scale & Automation
+
+### Bulk import
+
+High-volume import support using `insert_all` / `upsert_all` for batch
+persistence. Bypass per-record `persist` calls when the target opts in,
+enabling 10-100x throughput for simple create/upsert scenarios. Configurable
+batch size, with fallback to per-record mode on conflict.
+
+### Update & diff mode
+
+Support update (upsert) imports alongside create-only. Given a
+`deduplicate_by` key, detect existing records and show a diff preview:
+new records, changed fields (highlighted), unchanged rows. User confirms
+which changes to apply. Enables recurring data sync workflows.
+
+### Resume / retry on failure
+
+If an import fails mid-way (timeout, crash, transient error), resume from
+the last successful record instead of restarting from scratch. Track a
+checkpoint index in the report. Critical for large imports (5k+ records)
+where re-processing everything is not acceptable.
+
+### API pagination
+
+Support paginated API sources. The current API source does a single GET,
+which works for small datasets but not for APIs returning thousands of
+records across multiple pages. Support offset, cursor, and link-header
+pagination strategies via `api_config`:
+
+```ruby
+api_config do
+  endpoint "https://api.example.com/contacts"
+  pagination :cursor, param: "after", root: "data", next_key: "meta.next_cursor"
+end
+```
+
+### Scheduled imports
+
+Recurring imports from API or remote sources on a cron schedule. A target
+declares a schedule, and DataPorter automatically fetches and imports at
+the configured interval. Built on ActiveJob with configurable queue.
+
+---
+
+## v3.0 — Platform
+
+### Webhooks
+
+HTTP callbacks on import lifecycle events (started, completed, failed).
+Configurable per-target with URL, headers, and payload template. Enables
+integration with Slack notifications, CI pipelines, or external dashboards.
+
+### External connectors
+
+Source plugins beyond local files and HTTP APIs:
+
+- **Google Sheets** — OAuth2 + Sheets API, treat a spreadsheet as a source
+- **SFTP** — Poll a remote directory for new files
+- **AWS S3** — Watch a bucket/prefix for uploads
+- **Remote HTTP polling** — Periodically fetch from a paginated API
+
+Each connector implements the `Sources::Base` interface. Installed as
+optional companion gems (`data_porter-google_sheets`, `data_porter-s3`).
+
+### i18n
+
+Full internationalization of all UI strings, error messages, and status
+labels. Ship with English and French translations. Host apps can override
+or add languages via standard Rails I18n.
+
+### Dashboard & analytics
 
-- Scoped imports (filter index by user/tenant)
-- Webhooks / callbacks on import completion
-- Batch persist (`insert_all` support)
-- Resume / partial retry
-- Scheduled imports (recurring API source)
-- i18n
-- Dashboard stats
+Import statistics dashboard: success rates, average duration, records per
+import, most-used targets, failure trends. Mountable as an admin-only route.

data/docs/TARGETS.md

@@ -152,9 +152,14 @@ Each param accepts:
 | `required` | Boolean | `false` | Validated on import creation, shown with `*` in the form |
 | `label` | String | Humanized name | Display label in the form |
 | `default` | String | `nil` | Pre-filled value in the form |
-| `collection` | Lambda | `nil` | For `:select` type -- returns `[[label, value], ...]` |
+| `collection` | Lambda or Array | `nil` | For `:select` type -- `[[label, value], ...]` |
 
-Collection lambdas are evaluated when the form loads, not at boot time. This ensures fresh data (e.g., newly created hotels appear immediately).
+Collection accepts both a lambda and a plain array. Use a lambda for dynamic data (evaluated when the form loads, not at boot time):
+
+```ruby
+param :hotel_id, type: :select, collection: -> { Hotel.pluck(:name, :id) }
+param :status,   type: :select, collection: [%w[Active active], %w[Archived archived]]
+```
 
 ## Instance Methods
 
@@ -225,3 +230,103 @@ def on_error(record, error, context:)
   Sentry.capture_exception(error, extra: { record: record.attributes })
 end
 ```
+
+## Full example
+
+A complete target using most DSL features: multiple sources, import params, JSON root, API config, transform, custom validation, and lifecycle hooks.
+
+```ruby
+# frozen_string_literal: true
+
+class ContactTarget < DataPorter::Target
+  label "Contacts"
+  model_name "Contact"
+  icon "fas fa-address-book"
+  sources :csv, :xlsx, :json, :api
+  dry_run_enabled
+
+  columns do
+    column :name, type: :string, required: true
+    column :email, type: :email
+    column :phone_number, type: :string
+    column :address, type: :string
+    column :room, type: :string
+  end
+
+  params do
+    param :default_room, type: :text, label: "Default room"
+    param :import_source, type: :select, label: "Import source",
+          collection: [%w[Manual manual], %w[Migration migration], ["Directory sync", "sync"]]
+  end
+
+  json_root "contacts"
+
+  api_config do
+    endpoint "http://localhost:3001/contacts"
+    headers({ "Authorization" => "Bearer token" })
+    response_root :contacts
+  end
+
+  def transform(record)
+    apply_default_room(record)
+    normalize_phone(record)
+    record
+  end
+
+  def validate(record)
+    validate_email_format(record)
+  end
+
+  def persist(record, context:)
+    Contact.create!(record.attributes)
+  end
+
+  def after_import(results, context:)
+    Rails.logger.info("[DataPorter] Contacts: #{results[:created]} created, #{results[:errored]} errors")
+  end
+
+  def on_error(record, error, context:)
+    Rails.logger.warn("[DataPorter] Line #{record.line_number}: #{error.message}")
+  end
+
+  private
+
+  def apply_default_room(record)
+    return if record.data["room"].present?
+    return unless import_params["default_room"].present?
+
+    record.data["room"] = import_params["default_room"]
+  end
+
+  def normalize_phone(record)
+    phone = record.data["phone_number"]
+    return if phone.blank?
+
+    record.data["phone_number"] = phone.gsub(/\s/, "")
+  end
+
+  def validate_email_format(record)
+    email = record.data["email"]
+    return if email.blank?
+    return if email.match?(/\A[^@\s]+@[^@\s]+\z/)
+
+    record.add_error("Invalid email format: #{email}")
+  end
+end
+```
+
+This target exercises:
+
+| Feature | DSL / Hook | Effect |
+|---|---|---|
+| 4 sources | `sources :csv, :xlsx, :json, :api` | All source types available |
+| Typed columns | `type: :string, :email` | Built-in validation |
+| Required field | `required: true` on `name` | Rows without name get "missing" status |
+| Dry run | `dry_run_enabled` | Dry run button in preview |
+| Import params | `param :default_room`, `param :import_source` | Dynamic form fields |
+| JSON root | `json_root "contacts"` | Extracts from `{"contacts": [...]}` |
+| API config | `endpoint`, `headers`, `response_root` | Authenticated API fetch |
+| Transform | `apply_default_room`, `normalize_phone` | Fill blanks, strip whitespace |
+| Validate | `validate_email_format` | Custom error on invalid email |
+| After import | Logs summary | Post-import hook |
+| On error | Logs failed line | Per-record error hook |

data/lib/data_porter/components/preview/results_summary.rb

@@ -33,6 +33,7 @@ module DataPorter
           div(class: "dp-results__cards") do
             stat("dp-results__stat--success", @report.imported_count, "Imported")
             stat("dp-results__stat--error", @report.errored_count, "Errors")
+            stat("dp-results__stat--warning", skipped_count, "Skipped") if skipped_count.positive?
           end
         end
 
@@ -52,7 +53,11 @@ module DataPorter
         end
 
         def success?
-          @report.errored_count.zero?
+          @report.errored_count.zero? && skipped_count.zero?
+        end
+
+        def skipped_count
+          @report.missing_count.to_i + @report.partial_count.to_i
         end
       end
     end

data/lib/data_porter/configuration.rb

@@ -10,7 +10,10 @@ module DataPorter
                   :preview_limit,
                   :enabled_sources,
                   :scope,
-                  :purge_after
+                  :purge_after,
+                  :max_file_size,
+                  :max_records,
+                  :transaction_mode
 
     def initialize
       @parent_controller = "ApplicationController"
@@ -22,6 +25,9 @@ module DataPorter
       @enabled_sources = %i[csv json api xlsx]
       @scope = nil
       @purge_after = 60.days
+      @max_file_size = 10.megabytes
+      @max_records = 10_000
+      @transaction_mode = :per_record
     end
   end
 end

data/lib/data_porter/orchestrator/importer.rb

@@ -6,6 +6,14 @@ module DataPorter
       private
 
       def import_records
+        if DataPorter.configuration.transaction_mode == :all
+          import_all_or_nothing
+        else
+          import_per_record
+        end
+      end
+
+      def import_per_record
         importable = @data_import.importable_records
         context = build_context
         results = { created: 0, errored: 0 }
@@ -16,6 +24,25 @@ module DataPorter
           broadcast_progress(index + 1, total)
         end
 
+        finalize_import(results)
+      end
+
+      def import_all_or_nothing
+        importable = @data_import.importable_records
+        context = build_context
+        total = importable.size
+
+        ActiveRecord::Base.transaction do
+          importable.each_with_index do |record, index|
+            @target.persist(record, context: context)
+            broadcast_progress(index + 1, total)
+          end
+        end
+
+        finalize_import(created: total, errored: 0)
+      end
+
+      def finalize_import(results)
         @data_import.update!(status: :completed)
         @broadcaster.success
         results

data/lib/data_porter/orchestrator/record_builder.rb

@@ -8,6 +8,7 @@ module DataPorter
       def build_records
         source = build_source
         raw_rows = source.fetch
+        enforce_max_records!(raw_rows.size)
         columns = @target.class._columns || []
         validator = RecordValidator.new(columns)
 
@@ -16,6 +17,14 @@ module DataPorter
         end
       end
 
+      def enforce_max_records!(count)
+        max = DataPorter.configuration.max_records
+        return unless max
+        return if count <= max
+
+        raise Error, "File contains #{count} records, exceeds maximum of #{max}"
+      end
+
       def build_record(row, index, columns, validator)
         record = StoreModels::ImportRecord.new(
           line_number: index + 1,

data/lib/data_porter/registry.rb

@@ -37,6 +37,12 @@ module DataPorter
 
       private
 
+      def resolve_collection(collection)
+        return unless collection
+
+        collection.respond_to?(:call) ? collection.call : collection
+      end
+
       def serialize_params(params)
         return [] unless params
 
@@ -50,7 +56,7 @@ module DataPorter
           required: param.required,
           label: param.label,
           default: param.default,
-          collection: param.collection&.call
+          collection: resolve_collection(param.collection)
         }.compact
       end
     end

data/lib/data_porter/rejects_csv_builder.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+require "csv"
+
+module DataPorter
+  class RejectsCsvBuilder
+    def initialize(columns, records)
+      @columns = columns
+      @records = records
+    end
+
+    def generate
+      CSV.generate do |csv|
+        csv << header_row
+        rejected_records.each { |r| csv << record_row(r) }
+      end
+    end
+
+    private
+
+    def header_row
+      ["line"] + @columns.map { |c| c.name.to_s } + ["errors"]
+    end
+
+    def rejected_records
+      @records.reject(&:complete?)
+    end
+
+    def record_row(record)
+      values = @columns.map { |c| record.data[c.name.to_s] }
+      errors = record.errors_list.map(&:message).join("; ")
+      [record.line_number] + values + [errors]
+    end
+  end
+end

data/lib/data_porter/sources/base.rb

@@ -45,6 +45,12 @@ module DataPorter
       def auto_map(row)
         row.to_h.transform_keys { |k| k.parameterize(separator: "_").to_sym }
       end
+
+      def fallback_headers(raw_headers)
+        return raw_headers if raw_headers.any?(&:present?)
+
+        raw_headers.each_with_index.map { |_, i| "col_#{i + 1}" }
+      end
     end
   end
 end

data/lib/data_porter/sources/csv.rb

@@ -5,6 +5,8 @@ require "csv"
 module DataPorter
   module Sources
     class Csv < Base
+      SEPARATORS = [",", ";", "\t"].freeze
+
       def initialize(data_import, content: nil)
         super(data_import)
         @content = content
@@ -12,7 +14,8 @@ module DataPorter
 
       def headers
         first_line = csv_content.lines.first
-        ::CSV.parse_line(first_line, **extra_options).map(&:to_s)
+        raw = ::CSV.parse_line(first_line, **extra_options).map(&:to_s)
+        fallback_headers(raw)
       end
 
       def fetch
@@ -26,11 +29,28 @@ module DataPorter
       private
 
       def csv_content
-        @content || download_file
+        @csv_content ||= ensure_utf8(@content || download_file)
       end
 
       def download_file
-        @data_import.file.download.force_encoding("UTF-8")
+        @data_import.file.download
+      end
+
+      def ensure_utf8(raw)
+        raw = strip_bom(raw)
+        return raw if raw.encoding == Encoding::UTF_8 && raw.valid_encoding?
+
+        raw.force_encoding("UTF-8")
+        return raw if raw.valid_encoding?
+
+        raw.encode("UTF-8", "ISO-8859-1")
+      end
+
+      def strip_bom(raw)
+        bytes = raw.b
+        return raw unless bytes.start_with?("\xEF\xBB\xBF".b)
+
+        bytes[3..].force_encoding("UTF-8")
       end
 
       def csv_options
@@ -39,9 +59,16 @@ module DataPorter
 
       def extra_options
         config = @data_import.config
-        return {} unless config.is_a?(Hash)
+        return { col_sep: detect_separator } unless config.is_a?(Hash)
+
+        opts = config.symbolize_keys.slice(:col_sep, :encoding)
+        opts[:col_sep] ||= detect_separator
+        opts
+      end
 
-        config.symbolize_keys.slice(:col_sep, :encoding)
+      def detect_separator
+        first_line = csv_content.lines.first.to_s
+        SEPARATORS.max_by { |sep| first_line.count(sep) }
       end
     end
   end

data/lib/data_porter/sources/xlsx.rb

@@ -14,7 +14,8 @@ module DataPorter
       def headers
         sheet = target_sheet
         first_row = sheet.simple_rows.first
-        first_row&.values&.map(&:to_s) || []
+        raw = first_row&.values&.map(&:to_s) || []
+        fallback_headers(raw)
       ensure
         cleanup
       end

data/lib/data_porter/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DataPorter
-  VERSION = "0.9.0"
+  VERSION = "1.0.1"
 end

data/lib/data_porter.rb

@@ -18,6 +18,7 @@ require_relative "data_porter/sources"
 require_relative "data_porter/record_validator"
 require_relative "data_porter/broadcaster"
 require_relative "data_porter/orchestrator"
+require_relative "data_porter/rejects_csv_builder"
 require_relative "data_porter/components"
 require_relative "data_porter/engine"
 

data/lib/generators/data_porter/install/templates/create_data_porter_imports.rb.erb

@@ -10,7 +10,7 @@ class CreateDataPorterImports < ActiveRecord::Migration[<%= ActiveRecord::Migrat
       t.jsonb   :report,      null: false, default: {}
       t.jsonb   :config,      null: false, default: {}
 
-      t.references :user, polymorphic: true, null: false
+      t.references :user, polymorphic: true
 
       t.timestamps
     end

data/lib/generators/data_porter/install/templates/initializer.rb

@@ -15,11 +15,9 @@ DataPorter.configure do |config|
   # config.cable_channel_prefix = "data_porter"
 
   # Context builder: inject business data into targets.
-  # Receives the current controller instance.
-  # config.context_builder = ->(controller) {
-  #   OpenStruct.new(
-  #     user: controller.current_user
-  #   )
+  # Receives the DataImport record.
+  # config.context_builder = ->(data_import) {
+  #   { user: data_import.user }
   # }
 
   # Maximum number of records displayed in preview.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_porter
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 1.0.1
 platform: ruby
 authors:
 - Seryl Lounis
@@ -174,6 +174,7 @@ files:
 - lib/data_porter/orchestrator/record_builder.rb
 - lib/data_porter/record_validator.rb
 - lib/data_porter/registry.rb
+- lib/data_porter/rejects_csv_builder.rb
 - lib/data_porter/sources.rb
 - lib/data_porter/sources/api.rb
 - lib/data_porter/sources/base.rb
@@ -198,8 +199,11 @@ homepage: https://github.com/SerylLns/data_porter
 licenses:
 - MIT
 metadata:
+  homepage_uri: https://github.com/SerylLns/data_porter
   source_code_uri: https://github.com/SerylLns/data_porter
   changelog_uri: https://github.com/SerylLns/data_porter/blob/main/CHANGELOG.md
+  documentation_uri: https://github.com/SerylLns/data_porter#readme
+  bug_tracker_uri: https://github.com/SerylLns/data_porter/issues
   rubygems_mcp_server_uri: https://rubygems.org/gems/data_porter
   rubygems_mfa_required: 'true'
 rdoc_options: []