data_porter 0.1.0

data/docs/blog/006-parsing-csv-sources.md

@@ -0,0 +1,300 @@
+---
+title: "Building DataPorter #6 — Parsing CSV Data with Sources"
+series: "Building DataPorter - A Data Import Engine for Rails"
+part: 6
+tags: [ruby, rails, rails-engine, gem-development, csv, active-storage, source-pattern]
+published: false
+---
+
+# Parsing CSV Data with Sources
+
+> How to model an import record in the database, parse CSV files through a pluggable Source layer, and map headers to target columns -- the first end-to-end flow.
+
+## Context
+
+This is part 6 of the series where we build **DataPorter**, a mountable Rails engine for data import workflows. In [part 5](#), we designed the Target DSL and Registry -- the layer that describes *what* an import looks like: its columns, mappings, and persistence logic.
+
+Now we need the other half: the code that represents an import *in progress* and the code that reads raw data from a file. By the end of this article, we will have a `DataImport` ActiveRecord model to track state, a `Source` abstraction for parsing, and a concrete CSV source that maps headers to target columns. This is where data first flows through the engine.
+
+## The problem
+
+A target declaration tells the engine what columns to expect, but it says nothing about where data comes from or how to track the import lifecycle. We need a database-backed model that records who started the import, what state it is in, what records were parsed, and what errors occurred. We also need a layer that can read a CSV file (today) and a JSON payload or API response (later) through the same interface. Without this separation, the parsing logic would live in the controller or the target itself, coupling the format to the business rules.
+
+## What we're building
+
+Here is the end-to-end flow we are wiring together:
+
+```ruby
+# 1. Create an import record
+import = DataPorter::DataImport.create!(
+  target_key: "guests",
+  source_type: "csv",
+  user: current_user
+)
+
+# 2. Resolve the source and parse the file
+source_class = DataPorter::Sources.resolve(import.source_type)
+source = source_class.new(import, content: csv_string)
+rows = source.fetch
+# => [{ first_name: "Alice", last_name: "Smith", email: "alice@example.com" }, ...]
+
+# 3. The import knows its target
+import.target_class
+# => GuestTarget
+```
+
+Three objects, three concerns: `DataImport` tracks state, `Sources.resolve` picks the parser, and the source turns raw bytes into mapped hashes. The Orchestrator (part 7) will coordinate these pieces, but each works independently.
+
+## Implementation
+
+### Step 1 -- The DataImport model and migration
+
+The `DataImport` model is the central record for every import. It needs to track which target is being imported, what source format the data arrives in, what state the import has reached, and who initiated it.
+
+The migration creates a single table with JSONB columns for records and report data (the StoreModel types from part 4):
+
+```ruby
+# lib/generators/data_porter/install/templates/create_data_porter_imports.rb.erb
+create_table :data_porter_imports do |t|
+  t.string  :target_key,  null: false
+  t.string  :source_type, null: false, default: "csv"
+  t.integer :status,      null: false, default: 0
+  t.jsonb   :records,     null: false, default: []
+  t.jsonb   :report,      null: false, default: {}
+  t.jsonb   :config,      null: false, default: {}
+
+  t.references :user, polymorphic: true, null: false
+
+  t.timestamps
+end
+```
+
+The `user` reference is polymorphic (`user_type` + `user_id`), so the engine works regardless of whether the host app calls its user model `User`, `AdminUser`, or `Account`. The `config` JSONB column stores source-specific options like CSV delimiters or API authentication parameters -- things that vary per import, not per target.
+
+The model itself is compact:
+
+```ruby
+# app/models/data_porter/data_import.rb
+class DataImport < ActiveRecord::Base
+  self.table_name = "data_porter_imports"
+
+  belongs_to :user, polymorphic: true
+
+  enum :status, {
+    pending: 0, parsing: 1, previewing: 2,
+    importing: 3, completed: 4, failed: 5
+  }
+
+  attribute :records, StoreModels::ImportRecord.to_array_type, default: -> { [] }
+  attribute :report, StoreModels::Report.to_type, default: -> { StoreModels::Report.new }
+  attribute :config, :json, default: -> { {} }
+
+  validates :target_key, presence: true
+  validates :source_type, presence: true, inclusion: { in: %w[csv json api] }
+end
+```
+
+The `status` enum defines the import lifecycle as a linear state machine: `pending -> parsing -> previewing -> importing -> completed` (or `failed` at any point). Integer-backed enums keep the database column small and indexable. The `records` and `report` attributes use StoreModel types that we built in part 4 -- they serialize structured data into the JSONB columns while providing typed Ruby objects in memory.
+
+Two convenience methods bridge the model to the rest of the engine:
+
+```ruby
+# app/models/data_porter/data_import.rb
+def target_class
+  Registry.find(target_key)
+end
+
+def importable_records
+  records.select(&:importable?)
+end
+```
+
+`target_class` delegates to the Registry so any code holding a `DataImport` can reach the target's column definitions, mappings, and hooks. `importable_records` filters parsed records down to those that passed validation -- the subset the Orchestrator will actually persist.
+
+### Step 2 -- The Source base class
+
+Sources are responsible for one thing: turning raw input into an array of hashes where keys are target column names. The base class defines the interface and the shared mapping logic:
+
+```ruby
+# lib/data_porter/sources/base.rb
+module DataPorter
+  module Sources
+    class Base
+      def initialize(data_import, **)
+        @data_import = data_import
+        @target_class = data_import.target_class
+      end
+
+      def fetch
+        raise NotImplementedError
+      end
+    end
+  end
+end
+```
+
+Every source receives the `DataImport` record at construction, which gives it access to the target class (for column mappings) and the config hash (for source-specific options). The `**` double splat lets subclasses accept extra keyword arguments without the base class needing to know about them.
+
+The mapping logic lives in the base class because it is shared across all sources that deal with key-value rows:
+
+```ruby
+# lib/data_porter/sources/base.rb (private methods)
+def apply_csv_mapping(row)
+  mappings = @target_class._csv_mappings
+  return auto_map(row) if mappings.nil? || mappings.empty?
+
+  explicit_map(row, mappings)
+end
+
+def auto_map(row)
+  row.to_h.transform_keys { |k| k.parameterize(separator: "_").to_sym }
+end
+
+def explicit_map(row, mappings)
+  mappings.each_with_object({}) do |(header, column), hash|
+    hash[column] = row[header]
+  end
+end
+```
+
+There are two mapping strategies. When a target defines `csv_mapping`, explicit mapping applies: only the declared header-to-column pairs are extracted, and anything else in the row is silently dropped. When no mapping is defined, auto-mapping kicks in: every header is parameterized into a snake_case symbol (`"First Name"` becomes `:first_name`). This lets simple imports work with zero configuration while giving complex imports full control over which columns matter.
+
+### Step 3 -- The CSV source and source resolution
+
+The CSV source implements `fetch` by parsing content through Ruby's standard library `CSV` class:
+
+```ruby
+# lib/data_porter/sources/csv.rb
+class Csv < Base
+  def initialize(data_import, content: nil)
+    super(data_import)
+    @content = content
+  end
+
+  def fetch
+    rows = []
+    ::CSV.parse(csv_content, **csv_options) do |row|
+      rows << apply_csv_mapping(row)
+    end
+    rows
+  end
+
+  private
+
+  def csv_content
+    @content || download_file
+  end
+
+  def download_file
+    @data_import.file.download
+  end
+
+  def csv_options
+    { headers: true }.merge(extra_options)
+  end
+
+  def extra_options
+    config = @data_import.config
+    return {} unless config.is_a?(Hash)
+    config.symbolize_keys.slice(:col_sep, :encoding)
+  end
+end
+```
+
+The `content:` keyword argument enables two usage modes. In production, `content` is nil and the source downloads the file from ActiveStorage via `@data_import.file.download`. In tests, you pass a CSV string directly, avoiding the need for file attachments or storage mocks. The `csv_options` method merges `headers: true` (so `CSV.parse` yields `CSV::Row` objects with named access) with any per-import overrides from the `config` column -- currently `col_sep` for semicolon-delimited files and `encoding` for non-UTF-8 data.
+
+Source resolution ties it together:
+
+```ruby
+# lib/data_porter/sources.rb
+module DataPorter
+  module Sources
+    REGISTRY = {
+      csv: Csv
+    }.freeze
+
+    def self.resolve(type)
+      REGISTRY.fetch(type.to_sym) { raise Error, "Unknown source type: #{type}" }
+    end
+  end
+end
+```
+
+This is intentionally simpler than the Target Registry. Sources are engine-internal (the gem ships them), so a frozen hash with a `resolve` method is sufficient. When we add JSON and API sources in part 12, they get one line each in the registry.
+
+## Decisions & tradeoffs
+
+| Decision | We chose | Over | Because |
+|----------|----------|------|---------|
+| User association | Polymorphic `belongs_to :user` | A configurable foreign key or no association | Polymorphic works with any user model name without configuration; the engine does not need to know the host app's auth setup |
+| State tracking | Integer-backed `enum` | A state machine gem (AASM, Statesman) | Six linear states do not need transition guards or history tracking yet; a gem would add a dependency for no immediate benefit |
+| Auto-mapping fallback | `parameterize` + `to_sym` on headers | Requiring explicit mapping for all imports | Auto-mapping lets simple CSVs work with zero target configuration; explicit mapping is there when headers don't match column names |
+| CSV content injection | `content:` keyword on initialize | Always reading from ActiveStorage | Injecting content makes tests fast and storage-independent; production code passes `nil` and falls through to `download_file` |
+| Source-specific config | JSONB `config` column on DataImport | Separate columns for each option | A single JSONB column absorbs any source's options (col_sep, encoding, API headers) without schema changes |
+
+## Testing it
+
+DataImport specs verify validations, the status enum, and StoreModel integration:
+
+```ruby
+# spec/data_porter/data_import_spec.rb
+it "validates source_type inclusion" do
+  import = described_class.new(target_key: "guests", source_type: "xml")
+
+  expect(import).not_to be_valid
+  expect(import.errors[:source_type]).to include("is not included in the list")
+end
+
+it "saves and reloads with records" do
+  import = described_class.create!(
+    target_key: "guests", source_type: "csv",
+    user_type: "User", user_id: 1
+  )
+  record = DataPorter::StoreModels::ImportRecord.new(line_number: 1, data: { name: "Alice" })
+  import.update!(records: [record])
+
+  reloaded = described_class.find(import.id)
+  expect(reloaded.records.first.data).to eq({ "name" => "Alice" })
+end
+```
+
+CSV source specs exercise both mapping modes and the config override:
+
+```ruby
+# spec/data_porter/sources/csv_spec.rb
+it "parses CSV content and applies mapping" do
+  csv_content = "Prenom,Nom,Email\nAlice,Smith,alice@example.com\n"
+  source = described_class.new(data_import, content: csv_content)
+
+  rows = source.fetch
+
+  expect(rows.size).to eq(2)
+  expect(rows.first).to eq(
+    first_name: "Alice", last_name: "Smith", email: "alice@example.com"
+  )
+end
+
+it "auto-maps when no csv_mapping defined" do
+  csv_content = "First Name,Last Name\nAlice,Smith\n"
+  source = described_class.new(import, content: csv_content)
+
+  expect(source.fetch.first).to eq(first_name: "Alice", last_name: "Smith")
+end
+```
+
+All source specs pass CSV strings directly through the `content:` parameter, so they run without ActiveStorage, without file fixtures, and without I/O.
+
+## Recap
+
+- The **DataImport model** is the database-backed record for every import, tracking target key, source type, status, parsed records (via StoreModel), and the initiating user (via polymorphic association).
+- The **migration** uses JSONB columns for records, report, and config, keeping the schema stable as features evolve.
+- The **Source base class** defines the `fetch` interface and shared column-mapping logic with two strategies: explicit mapping from the target's `csv_mapping` block, or automatic parameterize-based mapping when none is defined.
+- The **CSV source** parses content via Ruby's `CSV` library, supports ActiveStorage file download in production and string injection in tests, and respects per-import config options like custom delimiters.
+
+## Next up
+
+We have targets that describe imports and sources that parse raw data into mapped hashes. But right now, nothing coordinates the flow: reading the file, building ImportRecord objects, running validations, transitioning the status, and persisting results. In part 7, we build the **Orchestrator** -- the class that ties `DataImport`, `Target`, and `Source` together into the complete parse-then-import workflow. That is where state transitions, per-record error handling, and ActiveJob integration come in.
+
+---
+
+*This is part 6 of the series "Building DataPorter - A Data Import Engine for Rails". [Previous: Designing a Target DSL](#) | [Next: The Orchestrator](#)*

data/docs/blog/007-orchestrator.md

@@ -0,0 +1,247 @@
+---
+title: "Building DataPorter #7 -- The Orchestrator: Coordinating the Import Workflow"
+series: "Building DataPorter - A Data Import Engine for Rails"
+part: 7
+tags: [ruby, rails, rails-engine, gem-development, orchestrator, activejob, error-handling]
+published: false
+---
+
+# The Orchestrator: Coordinating the Import Workflow
+
+> How to coordinate parsing, validation, and persistence through a single class that manages state transitions, handles errors per-record, and integrates with ActiveJob.
+
+## Context
+
+This is part 7 of the series where we build **DataPorter**, a mountable Rails engine for data import workflows. In [part 6](#), we built the DataImport model to track import state and the Source layer to parse CSV files into mapped hashes.
+
+We now have all the individual pieces: targets describe imports, sources parse files, the RecordValidator checks column constraints, and DataImport tracks state. But nothing ties them together. In this article, we build the Orchestrator -- the class that coordinates the full parse-then-import workflow.
+
+## The problem
+
+Right now, if you wanted to run an import, you would need to manually resolve the source, call `fetch`, iterate over the rows, build ImportRecord objects, run validations, update the status, and handle failures. That is a lot of procedural logic. If it lived in a controller action, it would be untestable, unreusable, and impossible to run in the background. If it lived in the model, DataImport would become a god object. We need a dedicated coordination layer that knows the *order* of operations but delegates the *details* to the objects that own them.
+
+## What we're building
+
+Here is the Orchestrator in action -- two method calls that drive the entire workflow:
+
+```ruby
+# In a controller or job
+orchestrator = DataPorter::Orchestrator.new(data_import, content: csv_string)
+
+# Step 1: Parse the file, validate records, generate a preview
+orchestrator.parse!
+data_import.status   # => "previewing"
+data_import.records  # => [ImportRecord, ImportRecord, ...]
+
+# Step 2: After user reviews the preview, persist the records
+orchestrator.import!
+data_import.status   # => "completed"
+data_import.report.imported_count  # => 42
+```
+
+Two methods, two phases. `parse!` turns raw data into validated records and stops at `previewing` so the user can review. `import!` takes the importable records and persists them through the target's `persist` method. If anything goes catastrophically wrong, the import transitions to `failed` with an error report.
+
+## Implementation
+
+### Step 1 -- The Orchestrator skeleton and parse phase
+
+The Orchestrator is a plain Ruby object. It receives a DataImport and optional content (for testing), then delegates to the pieces we already built:
+
+```ruby
+# lib/data_porter/orchestrator.rb
+class Orchestrator
+  def initialize(data_import, content: nil)
+    @data_import = data_import
+    @target = data_import.target_class.new
+    @source_options = { content: content }.compact
+  end
+
+  def parse!
+    @data_import.parsing!
+    records = build_records
+    @data_import.update!(records: records, status: :previewing)
+    build_report
+  rescue StandardError => e
+    handle_failure(e)
+  end
+end
+```
+
+The constructor instantiates the target (so we can call its hooks) and compacts the source options (so `content: nil` does not override ActiveStorage downloads). The `parse!` method follows a strict sequence: transition to `parsing`, build and validate records, save them with a `previewing` status, then generate a summary report. The `rescue` at the method level catches any failure -- a malformed CSV, a missing file, an unexpected source error -- and transitions the import to `failed` with the error message preserved in the report.
+
+Notice that `parsing!` is called *before* the work starts. This is intentional: if the job crashes between the status transition and the `update!`, the import is left in `parsing` rather than `pending`, signaling to the user that something went wrong mid-process rather than silently sitting idle.
+
+### Step 2 -- Building and validating records
+
+The `build_records` method is where the Source, Target, and RecordValidator converge:
+
+```ruby
+# lib/data_porter/orchestrator.rb
+def build_records
+  source = @data_import.source_class.new(@data_import, **@source_options)
+  raw_rows = source.fetch
+  columns = @target.class._columns || []
+  validator = RecordValidator.new(columns)
+
+  raw_rows.each_with_index.map do |row, index|
+    build_record(row, index, columns, validator)
+  end
+end
+
+def build_record(row, index, columns, validator)
+  record = StoreModels::ImportRecord.new(
+    line_number: index + 1,
+    data: extract_data(row, columns)
+  )
+  record = @target.transform(record)
+  @target.validate(record)
+  validator.validate(record)
+  record.determine_status!
+  record
+end
+```
+
+Each row goes through a four-step pipeline: extract the data for declared columns, let the target transform it (e.g., normalizing phone numbers), let the target run custom validations, then run the generic column-level validations (required fields, type checks). Finally, `determine_status!` sets each record to `complete`, `partial`, or `missing` based on whether errors were added.
+
+The RecordValidator handles the generic constraints we defined in the column DSL:
+
+```ruby
+# lib/data_porter/record_validator.rb
+class RecordValidator
+  def initialize(columns)
+    @columns = columns
+  end
+
+  def validate(record)
+    @columns.each do |col|
+      value = record.data[col.name]
+      validate_required(record, col, value)
+      validate_type(record, col, value)
+    end
+  end
+end
+```
+
+This separation matters: the target owns business-rule validations ("email must be unique in the system"), while the RecordValidator owns structural validations ("email must look like an email"). Neither knows about the other.
+
+### Step 3 -- The import phase and per-record error handling
+
+Once the user reviews the preview and confirms, `import!` persists the records:
+
+```ruby
+# lib/data_porter/orchestrator.rb
+def import!
+  @data_import.importing!
+  results = import_records
+  update_import_report(results)
+  @target.after_import(results, context: build_context)
+rescue StandardError => e
+  handle_failure(e)
+end
+
+def persist_record(record, context, results)
+  @target.persist(record, context: context)
+  results[:created] += 1
+rescue StandardError => e
+  record.add_error(e.message)
+  @target.on_error(record, e, context: context)
+  results[:errored] += 1
+end
+```
+
+The critical design decision here is the error boundary. Each record is persisted individually, and if `persist` raises -- a uniqueness violation, a foreign key constraint, a custom validation from the host app -- the error is captured *on that record* and the import continues. The import does not wrap everything in a single transaction. This means a 10,000-row file with 3 bad records will successfully import 9,997 records rather than rolling back the entire batch.
+
+The `on_error` hook lets the target react to failures (logging, notifying, skipping related records), while `after_import` runs once after all records are processed, receiving the results hash for summary work like sending a confirmation email.
+
+### Step 4 -- ActiveJob integration
+
+The Orchestrator is designed to be called from anywhere, but its primary consumer is a pair of ActiveJob classes:
+
+```ruby
+# app/jobs/data_porter/parse_job.rb
+class ParseJob < ActiveJob::Base
+  queue_as { DataPorter.configuration.queue_name }
+
+  def perform(import_id)
+    data_import = DataImport.find(import_id)
+    Orchestrator.new(data_import).parse!
+  end
+end
+
+# app/jobs/data_porter/import_job.rb
+class ImportJob < ActiveJob::Base
+  queue_as { DataPorter.configuration.queue_name }
+
+  def perform(import_id)
+    data_import = DataImport.find(import_id)
+    Orchestrator.new(data_import).import!
+  end
+end
+```
+
+Each job is a one-liner: find the import, delegate to the Orchestrator. The queue name comes from the engine's configuration, so the host app controls which queue processes imports. Because the Orchestrator already handles failures internally (transitioning to `failed` and recording the error), the jobs do not need their own error handling -- a crash at the ActiveJob level means something truly unexpected happened, and the adapter's retry mechanism takes over.
+
+## Decisions & tradeoffs
+
+| Decision | We chose | Over | Because |
+|----------|----------|------|---------|
+| Coordination layer | Dedicated Orchestrator class | Controller-level logic or model callbacks | Keeps controllers thin, models focused on data, and the workflow independently testable |
+| Transaction boundaries | Per-record persist (no wrapping transaction) | Single transaction around all records | A failed record should not roll back thousands of successful ones; partial success is more useful than total failure |
+| Error recovery | Capture error on the record, continue importing | Halt on first error | Users expect to see which rows failed and why, not just "import failed"; the report becomes actionable |
+| Two-phase workflow | Separate `parse!` and `import!` methods | A single `run!` method | The preview step between parse and import lets users catch problems before data hits the database |
+| Job design | Thin jobs delegating to Orchestrator | Logic inside the job classes | The Orchestrator is testable without ActiveJob; jobs are just the async trigger |
+
+## Testing it
+
+The Orchestrator specs exercise both phases end-to-end using an anonymous target class and injected CSV content:
+
+```ruby
+# spec/data_porter/orchestrator_spec.rb
+describe "#parse!" do
+  it "transitions to previewing" do
+    orchestrator = described_class.new(data_import, content: csv_content)
+
+    orchestrator.parse!
+
+    expect(data_import.reload.status).to eq("previewing")
+  end
+
+  it "validates required fields" do
+    csv = "First Name,Last Name,Email\n,Smith,alice@example.com\n"
+    orchestrator = described_class.new(data_import, content: csv)
+
+    orchestrator.parse!
+
+    record = data_import.reload.records.first
+    expect(record.status).to eq("missing")
+  end
+end
+
+describe "#import!" do
+  it "handles per-record errors" do
+    # Target that always raises on persist
+    orchestrator = described_class.new(import)
+    orchestrator.import!
+
+    expect(import.reload.status).to eq("completed")
+    expect(import.report.errored_count).to eq(1)
+  end
+end
+```
+
+The key pattern: even when every `persist` call raises, the import still reaches `completed` -- not `failed`. The `failed` status is reserved for catastrophic errors (the source cannot be read, the target cannot be resolved). Per-record errors are expected operational noise, tracked in the report.
+
+## Recap
+
+- The **Orchestrator** is a plain Ruby class that coordinates the parse-validate-persist workflow, keeping controllers thin and models focused.
+- The **two-phase design** (`parse!` then `import!`) creates a natural preview checkpoint where users can review data before it touches the database.
+- **Per-record error handling** means a single bad row never takes down the entire import; errors are captured on individual records and surfaced in the report.
+- **ActiveJob integration** is a thin wrapper: two one-liner jobs that delegate to the Orchestrator, using the engine's configured queue name.
+
+## Next up
+
+The import now runs in the background, but the user has no way to know what is happening. They click "Import" and stare at a static page. In part 8, we build a **real-time progress system** using ActionCable and Stimulus -- a Broadcaster service that pushes status updates and record counts to the browser as the Orchestrator processes each row. No more refreshing to check if it is done.
+
+---
+
+*This is part 7 of the series "Building DataPorter - A Data Import Engine for Rails". [Previous: Parsing CSV Data with Sources](#) | [Next: Real-time Progress with ActionCable & Stimulus](#)*