data_porter 0.1.0

data/docs/blog/004-store-model-jsonb.md

@@ -0,0 +1,237 @@
+---
+title: "Building DataPorter #4 — Modeling import data with StoreModel & JSONB"
+series: "Building DataPorter - A Data Import Engine for Rails"
+part: 4
+tags: [ruby, rails, rails-engine, gem-development, store-model, jsonb, data-modeling]
+published: false
+---
+
+# Modeling import data with StoreModel & JSONB
+
+> Storing structured import records, errors, and reports inside a single JSONB column -- no extra tables, no schema sprawl.
+
+## Context
+
+This is part 4 of the series where we build **DataPorter**, a mountable Rails engine for data import workflows. In [part 3](#), we built the configuration DSL that lets host apps customize the gem through a clean `configure` block.
+
+Now we shift from *how the gem behaves* to *what it operates on*: the data models for parsed records, validation errors, and summary reports. We'll model all three using the StoreModel gem and PostgreSQL JSONB columns.
+
+## The problem
+
+A typical import engine ends up with a lot of tables: imports, import rows, import errors, reports. Each needs a migration, foreign keys, indexes, and cleanup logic. For a gem that drops into any Rails app, that's a heavy footprint.
+
+But these records are ephemeral. They exist during the import workflow, get consulted in the results view, and nobody queries them independently. You never ask "give me all errors across all imports." They're always accessed through their parent.
+
+If the data is always read and written as a group, it doesn't need its own table. It needs a structured column.
+
+## What we're building
+
+A single `DataImport` record will carry its entire import payload in JSONB columns:
+
+```ruby
+# Anywhere in the engine
+import = DataPorter::DataImport.find(42)
+
+import.report.records_count      # => 150
+import.report.errored_count      # => 3
+import.report.error_reports.each { |e| puts e.message }
+
+import.records.first.line_number  # => 1
+import.records.first.status       # => "complete"
+import.records.first.data         # => { "name" => "Alice", "email" => "alice@example.com" }
+```
+
+No joins, no N+1 queries. Records and reports come back as typed Ruby objects with real attributes and methods -- not raw hashes.
+
+## Implementation
+
+### Step 1 -- The Error model
+
+Every import record can accumulate validation errors. We need a small object to represent each one. StoreModel lets us define it like an ActiveModel attribute model, but serialized into JSON.
+
+```ruby
+# lib/data_porter/store_models/error.rb
+module DataPorter
+  module StoreModels
+    class Error
+      include StoreModel::Model
+
+      attribute :message, :string
+    end
+  end
+end
+```
+
+`include StoreModel::Model` gives us ActiveModel-compatible attributes that serialize to and from JSON. Why not a plain hash? Because `error.message` is a method call with autocompletion, not `error["message"]` where you guess at indifferent access. If we need `:code` or `:severity` later, we add an attribute and existing data deserializes cleanly -- new fields default to nil.
+
+### Step 2 -- The ImportRecord model
+
+Each row from the source file becomes an `ImportRecord`. This is the workhorse of the import: it holds the parsed data, tracks validation status, and collects errors and warnings.
+
+```ruby
+# lib/data_porter/store_models/import_record.rb
+module DataPorter
+  module StoreModels
+    class ImportRecord
+      include StoreModel::Model
+
+      attribute :line_number, :integer
+      attribute :status, :string, default: "pending"
+      attribute :data, default: -> { {} }
+      attribute :errors_list, Error.to_array_type, default: -> { [] }
+      attribute :warnings, Error.to_array_type, default: -> { [] }
+      attribute :target_id, :integer
+      attribute :dry_run_passed, :boolean, default: false
+    end
+  end
+end
+```
+
+The `data` attribute stores whatever hash the source parser produces -- no explicit type, because each import target defines different columns. The lambda defaults (`-> { {} }`) are critical; without them, every record shares the same mutable object. `Error.to_array_type` makes `errors_list` a typed array: each JSON entry deserializes into an `Error` instance, not a raw hash.
+
+The model also carries behavior. Status determination runs after validation:
+
+```ruby
+# lib/data_porter/store_models/import_record.rb
+def determine_status!
+  self.status = if required_error?
+                  "missing"
+                elsif errors_list.any?
+                  "partial"
+                else
+                  "complete"
+                end
+end
+```
+
+Three statuses: "missing" (a required field is absent -- the record cannot be imported), "partial" (optional field errors exist -- the record can be imported with warnings), and "complete" (clean row, ready to go). The Orchestrator will call `determine_status!` after validation and use `importable?` to decide which records to persist.
+
+### Step 3 -- The Report model
+
+After parsing and validating, we need a summary. The Report model aggregates counts and collects top-level errors (like "file has no header row" or "unexpected encoding").
+
+```ruby
+# lib/data_porter/store_models/report.rb
+module DataPorter
+  module StoreModels
+    class Report
+      include StoreModel::Model
+
+      attribute :records_count, :integer, default: 0
+      attribute :complete_count, :integer, default: 0
+      attribute :partial_count, :integer, default: 0
+      attribute :missing_count, :integer, default: 0
+      attribute :duplicate_count, :integer, default: 0
+      attribute :imported_count, :integer, default: 0
+      attribute :errored_count, :integer, default: 0
+      attribute :error_reports, Error.to_array_type, default: []
+    end
+  end
+end
+```
+
+Every counter defaults to zero; the Orchestrator increments them during processing. `error_reports` reuses `Error.to_array_type` for import-level errors that don't belong to a specific row -- same typed-array pattern as `ImportRecord#errors_list`, so the UI can render both with the same component.
+
+### Step 4 -- TypeValidator: validating before the database
+
+StoreModel handles serialization. But we also need to validate *values* before they reach the model -- does "abc" pass as an integer? Is "not-a-date" a valid date? The TypeValidator module handles this at the column level, before the data ever touches ActiveRecord.
+
+```ruby
+# lib/data_porter/type_validator.rb
+module DataPorter
+  module TypeValidator
+    VALIDATORS = {
+      string:   ->(_value, _opts) { true },
+      integer:  ->(value, _opts) { Integer(value, exception: false) },
+      decimal:  ->(value, _opts) { Float(value, exception: false) },
+      date:     ->(value, opts) { parse_date(value, opts) },
+      email:    ->(value, _opts) { value.match?(/\A[^@\s]+@[^@\s]+\z/) },
+      phone:    ->(value, _opts) { value.match?(/\A[+\d][\d\s\-().]{6,}\z/) },
+      url:      ->(value, _opts) { valid_url?(value) },
+      boolean:  ->(value, _opts) { %w[true false 1 0].include?(value.to_s.downcase) }
+    }.freeze
+  end
+end
+```
+
+Each type maps to a lambda that returns truthy or falsy. The public API is one method: `TypeValidator.valid?("42", :integer)`. Integers use `Integer()` with `exception: false` to avoid rescue-driven control flow. Dates support an optional `:format` option for regional formatting like `"%d/%m/%Y"`.
+
+The key design choice: validation happens *before* data enters the StoreModel. During parsing, the source reads a row, column definitions declare expected types, and the validator checks each value. Errors get added to the ImportRecord via `add_error`. By the time `determine_status!` runs, all column-level issues are captured.
+
+This is deliberately separate from database-level validation (uniqueness, foreign keys), which runs later during the actual import. Keeping the two layers apart lets us show users a preview with type errors highlighted before any write attempt -- the foundation for the dry-run feature in part 14.
+
+## Decisions & tradeoffs
+
+| Decision | We chose | Over | Because |
+|----------|----------|------|---------|
+| Row storage | JSONB column (array of StoreModel) | Separate `import_rows` table | Records are always accessed through their parent; no independent queries needed. One fewer migration for host apps to manage |
+| Structured JSON | StoreModel gem | Hand-rolled `serialize` / raw hashes | StoreModel gives us ActiveModel attributes, typed arrays, defaults, and validations. Writing our own serializer would duplicate all of that |
+| Validation layer | Column-level TypeValidator + later DB-level | Database-only validation | Enables preview and dry-run without touching the database. Users see type errors immediately, before any write attempt |
+| Error representation | StoreModel class with `:message` | Plain strings in an array | Extensible -- we can add `:code`, `:severity`, `:column` later without changing the array structure or breaking existing serialized data |
+| Status logic | Method on ImportRecord (`determine_status!`) | External service or state machine gem | Status depends only on the record's own errors. No transitions or events needed. A method is the simplest thing that works |
+
+## Testing it
+
+ImportRecord specs verify status determination:
+
+```ruby
+# spec/data_porter/store_models/import_record_spec.rb
+RSpec.describe DataPorter::StoreModels::ImportRecord do
+  subject(:record) { described_class.new(line_number: 1, data: { name: "Alice" }) }
+
+  describe "#determine_status!" do
+    it "sets missing when required field error exists" do
+      record.add_error("Name is required")
+      record.determine_status!
+      expect(record.status).to eq("missing")
+    end
+
+    it "sets partial when non-required error exists" do
+      record.add_error("Email: invalid email")
+      record.determine_status!
+      expect(record.status).to eq("partial")
+    end
+
+    it "sets complete when no errors" do
+      record.determine_status!
+      expect(record.status).to eq("complete")
+    end
+  end
+end
+```
+
+TypeValidator specs cover each type, including edge cases like custom date formats:
+
+```ruby
+# spec/data_porter/type_validator_spec.rb
+RSpec.describe DataPorter::TypeValidator do
+  it "accepts valid integers" do
+    expect(described_class.valid?("42", :integer)).to be true
+  end
+
+  it "rejects non-integers" do
+    expect(described_class.valid?("abc", :integer)).to be false
+  end
+
+  it "accepts dates with custom format" do
+    expect(described_class.valid?("15/01/2024", :date, format: "%d/%m/%Y")).to be true
+  end
+end
+```
+
+No database setup needed for any of these. StoreModel objects instantiate in memory like plain Ruby objects, which makes the specs fast and isolated.
+
+## Recap
+
+- **JSONB columns** let us store structured import data (records, errors, reports) without extra tables. The data is always accessed through the parent import, so a separate table would add complexity with no query benefit.
+- **StoreModel** turns those JSONB columns into proper Ruby objects with typed attributes, defaults, and methods. `Error.to_array_type` gives us typed arrays that serialize and deserialize automatically.
+- **ImportRecord** is the core unit of work: it holds a parsed row, collects errors and warnings, and determines its own status based on the errors it carries.
+- **TypeValidator** handles column-level validation before the database, enabling the preview and dry-run features. Each type is a lambda in a hash -- easy to extend, easy to test.
+
+## Next up
+
+We have configuration (part 3) and data models (this part). In part 5, we'll bring them together by designing the **Target DSL** -- the class-level interface that lets each import type declare its label, model, columns, and CSV mapping in a single file. One file per import type, zero boilerplate. If you've ever wanted `class_attribute` to do more heavy lifting, that's the one.
+
+---
+
+*This is part 4 of the series "Building DataPorter - A Data Import Engine for Rails". [Previous: Configuration DSL](#) | [Next: Designing a Target DSL](#)*

data/docs/blog/005-target-dsl.md

@@ -0,0 +1,284 @@
+---
+title: "Building DataPorter #5 — Designing a Target DSL"
+series: "Building DataPorter - A Data Import Engine for Rails"
+part: 5
+tags: [ruby, rails, rails-engine, gem-development, dsl, metaprogramming, registry-pattern]
+published: false
+---
+
+# Designing a Target DSL
+
+> How to make each import type a single, self-describing Ruby class -- one file, zero boilerplate.
+
+## Context
+
+This is part 5 of the series where we build **DataPorter**, a mountable Rails engine for data import workflows. In [part 4](#), we modeled import records, errors, and reports using StoreModel and JSONB columns -- the data structures the engine operates on.
+
+Now we need the layer that *describes* an import: what model does it target, what columns does it expect, how do CSV headers map to those columns? This is the Target DSL and the Registry that makes targets discoverable.
+
+## The problem
+
+Every import type in a Rails app needs the same boilerplate: column definitions, header mapping, validation rules, persistence logic. Without a convention, each import ends up in a different controller action or service object, with slightly different patterns. Adding a new import type means copying an existing one and changing field names.
+
+We want a developer to open a single file, declare what their import looks like, and have the engine handle everything else. No initializer wiring, no registration callbacks, no controller configuration.
+
+## What we're building
+
+Here is what a complete target definition looks like in the host app:
+
+```ruby
+# app/data_porter/targets/guest_target.rb
+class GuestTarget < DataPorter::Target
+  label "Guests"
+  model_name "Guest"
+  icon "fas fa-users"
+  sources :csv, :json
+
+  columns do
+    column :first_name, type: :string, required: true
+    column :last_name,  type: :string, required: true
+    column :email,      type: :email
+  end
+
+  csv_mapping do
+    map "Prenom" => :first_name
+    map "Nom"    => :last_name
+  end
+
+  deduplicate_by :email
+
+  def persist(record, context:)
+    Guest.create!(record.data)
+  end
+end
+```
+
+That is the entire file. The class-level DSL declares metadata and column schema. The instance method `persist` handles the actual write. The engine discovers this class through the Registry and wires it into the UI and orchestration layer automatically.
+
+## Implementation
+
+### Step 1 -- The Column struct
+
+Before the Target itself, we need a value object for columns. Each column has a name, a type for validation, a required flag, a display label, and an open-ended options hash for type-specific settings like date formats.
+
+```ruby
+# lib/data_porter/dsl/column.rb
+module DataPorter
+  module DSL
+    Column = Struct.new(:name, :type, :required, :label, :options, keyword_init: true) do
+      def initialize(name:, type: :string, required: false, label: nil, **options)
+        super(
+          name: name.to_sym,
+          type: type.to_sym,
+          required: required,
+          label: label || name.to_s.humanize,
+          options: options
+        )
+      end
+    end
+  end
+end
+```
+
+A `Struct` gives us equality, `to_h`, `members`, and frozen-by-value semantics for free. The constructor coerces `name` and `type` to symbols so callers can pass strings or symbols without worrying. The `label` falls back to `humanize` -- one less thing to type for the common case, but overridable when the generated label doesn't fit (`column :full_name, label: "Full Name"`). The `**options` splat captures anything else (like `format: "%d/%m/%Y"` for dates) and tucks it into the `options` hash, keeping the struct's interface stable as we add type-specific features.
+
+### Step 2 -- The Target base class
+
+The Target is where the DSL lives. All the declarative methods (`label`, `model_name`, `columns`, etc.) are class methods on a base class that host-app targets inherit from. Instance methods provide hook points for the import lifecycle.
+
+```ruby
+# lib/data_porter/target.rb
+module DataPorter
+  class Target
+    class << self
+      attr_reader :_label, :_model_name, :_icon, :_sources,
+                  :_columns, :_csv_mappings, :_dedup_keys
+
+      def label(value)      = @_label = value
+      def model_name(value) = @_model_name = value
+      def icon(value)       = @_icon = value
+
+      def sources(*types)
+        @_sources = types.map(&:to_sym)
+      end
+
+      def columns(&)
+        @_columns = []
+        instance_eval(&)
+      end
+
+      def column(name, **)
+        @_columns << DSL::Column.new(name: name, **)
+      end
+
+      def csv_mapping(&)
+        @_csv_mappings = {}
+        instance_eval(&)
+      end
+
+      def map(hash)
+        @_csv_mappings.merge!(hash)
+      end
+
+      def deduplicate_by(*keys)
+        @_dedup_keys = keys.map(&:to_sym)
+      end
+    end
+  end
+end
+```
+
+Every DSL method is a class method that stores its value in a class instance variable (`@_label`, not `@@label`). The underscore prefix is a convention to signal "this is DSL storage, not your public API." The `columns` block uses `instance_eval` to execute `column` calls in the class context, which gives us the nested block syntax without requiring the caller to reference `self`.
+
+The instance-level hooks provide the extensibility points for the import lifecycle:
+
+```ruby
+# lib/data_porter/target.rb (instance methods)
+def transform(record)  = record
+def validate(record)   = nil
+def persist(_record, context:) = raise NotImplementedError
+def after_import(_results, context:) = nil
+def on_error(_record, _error, context:) = nil
+```
+
+`transform` and `validate` are no-ops by default -- override them if you need custom data munging or cross-field validation beyond type checking. `persist` raises `NotImplementedError` because every target must define how records get written. `after_import` and `on_error` are optional hooks for cleanup, notifications, or error recovery. The `context:` keyword argument carries the host app context (current user, tenant, etc.) that we set up in the configuration DSL back in part 3.
+
+The split between class methods (declaration) and instance methods (execution) is deliberate. Class methods describe *what* the import is. Instance methods describe *what happens* during the import. The Orchestrator (part 7) will call `target_class._columns` to know the schema, then instantiate the target and call `target.persist(record, context: ctx)` for each row. Keeping these on different layers prevents the declaration phase from depending on runtime state.
+
+### Step 3 -- The Registry
+
+Targets need to be discoverable. The engine's UI shows a dropdown of available import types; the Orchestrator looks up a target by key to process an import. The Registry is the central index.
+
+```ruby
+# lib/data_porter/registry.rb
+module DataPorter
+  class TargetNotFound < Error; end
+
+  module Registry
+    @targets = {}
+
+    class << self
+      def register(key, klass)
+        @targets[key.to_sym] = klass
+      end
+
+      def find(key)
+        @targets.fetch(key.to_sym) do
+          raise TargetNotFound, "Target '#{key}' not found"
+        end
+      end
+
+      def available
+        @targets.map do |key, klass|
+          { key: key, label: klass._label, icon: klass._icon }
+        end
+      end
+
+      def clear
+        @targets = {}
+      end
+    end
+  end
+end
+```
+
+The Registry is a module with class-level state -- essentially a singleton hash. `register` adds a target class under a symbolic key. `find` retrieves it, raising a custom `TargetNotFound` error instead of a generic `KeyError` so the controller can rescue it with a proper 404. `available` returns lightweight summaries for the UI: just the key, label, and icon, without exposing the full class.
+
+Registration happens in the host app's initializer:
+
+```ruby
+# config/initializers/data_porter.rb
+DataPorter::Registry.register(:guests, GuestTarget)
+DataPorter::Registry.register(:products, ProductTarget)
+```
+
+We considered auto-discovery (scanning a directory for Target subclasses), but explicit registration is simpler to reason about: you can see exactly which targets are active, control ordering, and conditionally register based on environment or feature flags. The `clear` and `refresh!` methods support testing and hot-reloading in development.
+
+## Decisions & tradeoffs
+
+| Decision | We chose | Over | Because |
+|----------|----------|------|---------|
+| DSL placement | Class methods on a base class | Instance methods or a configuration hash | Class methods read like declarations, not procedure calls. They execute once at load time, not per-import |
+| State storage | Class instance variables (`@_label`) | `class_attribute` from ActiveSupport | Class instance variables don't leak to subclasses by default, avoiding surprising inheritance behavior. We don't need the per-instance override that `class_attribute` provides |
+| Column definition | `Struct` with keyword init | Plain hash or full ActiveModel class | Struct gives us typed attributes, equality, and `to_h` with no dependencies. A hash would lose the interface; ActiveModel would be overkill for a value object |
+| Hook pattern | Instance methods with default no-ops | Event system or callback chain | Override-and-call is the simplest extension model. No subscription management, no ordering concerns. If you need it, override it |
+| Registry | Explicit `register` calls | Auto-discovery via `inherited` hook or directory scanning | Explicit registration is visible, testable, and doesn't depend on load order or file system conventions. Auto-discovery can be added later as sugar on top |
+
+## Testing it
+
+Target specs verify both the DSL declarations and the default hook behavior:
+
+```ruby
+# spec/data_porter/target_spec.rb
+let(:target_class) do
+  Class.new(DataPorter::Target) do
+    label "Guests"
+    model_name "Guest"
+    sources :csv, :json
+
+    columns do
+      column :first_name, type: :string, required: true
+      column :email,      type: :email
+    end
+
+    csv_mapping do
+      map "Prenom" => :first_name
+    end
+  end
+end
+
+it "sets the label" do
+  expect(target_class._label).to eq("Guests")
+end
+
+it "defines columns" do
+  expect(target_class._columns.size).to eq(2)
+end
+
+it "persist raises NotImplementedError" do
+  expect { target_class.new.persist(nil, context: nil) }
+    .to raise_error(NotImplementedError)
+end
+```
+
+Registry specs confirm lookup, error handling, and the `available` summary:
+
+```ruby
+# spec/data_porter/registry_spec.rb
+before { described_class.clear }
+
+it "stores a target by key" do
+  described_class.register(:guests, target_class)
+  expect(described_class.find(:guests)).to eq(target_class)
+end
+
+it "raises TargetNotFound for unknown keys" do
+  expect { described_class.find(:unknown) }
+    .to raise_error(DataPorter::TargetNotFound)
+end
+
+it "returns target summaries" do
+  described_class.register(:guests, target_class)
+  result = described_class.available
+  expect(result).to contain_exactly(
+    { key: :guests, label: "Guests", icon: "fas fa-users" }
+  )
+end
+```
+
+Both suites run without a database. Anonymous classes (`Class.new(DataPorter::Target)`) let us define fresh targets per test without polluting the class hierarchy.
+
+## Recap
+
+- The **Target base class** uses class methods as a declarative DSL: `label`, `model_name`, `columns`, `csv_mapping`, and `deduplicate_by`. Each stores its value in a class instance variable, keeping subclasses isolated.
+- The **Column struct** is a lightweight value object that captures name, type, required flag, display label, and open-ended options. Struct gives us equality and `to_h` for free.
+- **Instance method hooks** (`transform`, `validate`, `persist`, `after_import`, `on_error`) separate runtime behavior from static declaration. Only `persist` is mandatory; the rest default to no-ops.
+- The **Registry** is an explicit registration system that maps symbolic keys to target classes, providing lookup for the Orchestrator and summaries for the UI.
+
+## Next up
+
+We have data models (part 4) and a target DSL (this part) that describes what each import expects. In part 6, we will wire them together with the **Source layer** -- starting with CSV parsing, ActiveStorage file handling, and automatic column mapping from CSV headers to target columns. That is where the first end-to-end flow comes together: upload a file, parse it, and see structured records.
+
+---
+
+*This is part 5 of the series "Building DataPorter - A Data Import Engine for Rails". [Previous: Modeling import data with StoreModel & JSONB](#) | [Next: Parsing CSV data with Sources](#)*