data_porter 0.1.0

data/docs/blog/SERIES.md

@@ -0,0 +1,154 @@
+---
+series: "Building DataPorter - A Data Import Engine for Rails"
+author: ""
+repo: ""
+status: complete
+---
+
+# Series Plan
+
+## Target audience
+Ruby/Rails developers (intermediate+) interested in gem architecture, Rails engines, and clean DSL design.
+
+## Series structure
+Each article: ~5-8 min read, one clear concept, real code, decisions explained.
+
+---
+
+## Articles
+
+### Part 1 — Why build a data import engine?
+- **Tasks:** None (intro)
+- **Hook:** The repetitive import pattern in Rails apps
+- **Covers:** Problem statement, existing solutions (maintenance_tasks, activeadmin_import), why a new gem, what we'll build (3-step workflow diagram)
+- **Key decisions:** Mountable engine vs. lib, isolate_namespace, target audience
+- **File:** `docs/blog/001-why-build-a-data-import-engine.md`
+- **Status:** draft
+
+### Part 2 — Scaffolding a Rails Engine gem
+- **Tasks:** #1
+- **Hook:** `bundle gem` is just the start
+- **Covers:** Gem structure, Engine setup, isolate_namespace, directory tree, gemspec dependencies
+- **Key decisions:** Why isolate_namespace, directory conventions, dependency choices (store_model, phlex, turbo)
+- **File:** `docs/blog/002-scaffolding-a-rails-engine.md`
+- **Status:** draft
+
+### Part 3 — Configuration DSL: making the gem flexible
+- **Tasks:** #2
+- **Hook:** A good gem adapts to any host app
+- **Covers:** Configuration singleton pattern, DSL with `configure` block, sensible defaults, context_builder lambda
+- **Key decisions:** Why not Rails config, lambda vs block for context, what to make configurable vs. convention
+- **File:** `docs/blog/003-configuration-dsl.md`
+- **Status:** draft
+
+### Part 4 — Modeling import data with StoreModel & JSONB
+- **Tasks:** #3, #4
+- **Hook:** Storing structured data without extra tables
+- **Covers:** StoreModel gem, ImportRecord/Error/Report models, JSONB attributes, TypeValidator
+- **Key decisions:** JSONB vs. separate tables, store_model vs. hand-rolled, validation strategy (column-level vs DB-level)
+- **File:** `docs/blog/004-store-model-jsonb.md`
+- **Status:** draft
+
+### Part 5 — Designing a Target DSL
+- **Tasks:** #5, #6
+- **Hook:** One file = one import type, zero boilerplate
+- **Covers:** Target base class, class-level DSL (label, model, columns, csv_mapping), Registry pattern, auto-discovery
+- **Key decisions:** DSL design (class methods vs. instance), class_attribute vs. class instance vars, hook pattern for extensibility
+- **File:** `docs/blog/005-target-dsl.md`
+- **Status:** draft
+
+### Part 6 — Parsing CSV data with Sources
+- **Tasks:** #7, #8
+- **Hook:** The first end-to-end flow
+- **Covers:** DataImport model, migration, Source base class, CSV source, ActiveStorage integration, column mapping
+- **Key decisions:** Polymorphic user association, enum state machine, auto-mapping vs explicit mapping
+- **File:** `docs/blog/006-parsing-csv-sources.md`
+- **Status:** draft
+
+### Part 7 — The Orchestrator: coordinating the import workflow
+- **Tasks:** #9, #10
+- **Hook:** The brain of the engine
+- **Covers:** Orchestrator class (parse! and import!), state transitions, error handling per-record, ActiveJob integration
+- **Key decisions:** Why an orchestrator (not controller logic), transaction boundaries, error recovery strategy
+- **File:** `docs/blog/007-orchestrator.md`
+- **Status:** draft
+
+### Part 8 — Real-time progress with ActionCable & Stimulus
+- **Tasks:** #11, #14
+- **Hook:** Users hate staring at a spinner with no feedback
+- **Covers:** Broadcaster service, ImportChannel, Stimulus controller, progress bar updates, auto-reload on completion
+- **Key decisions:** ActionCable vs. polling vs. SSE, channel naming strategy, Stimulus vs. Turbo Streams
+- **File:** `docs/blog/008-actioncable-stimulus.md`
+- **Status:** draft
+
+### Part 9 — Building the UI with Phlex & Tailwind
+- **Tasks:** #13
+- **Hook:** Auto-generated preview tables from a DSL
+- **Covers:** Phlex components, scoped Tailwind (dp- prefix), preview table with dynamic columns, status badges, CSS custom properties for theming
+- **Key decisions:** Phlex vs. ERB/ViewComponent, Tailwind prefix strategy, how to not pollute host app styles
+- **File:** `docs/blog/009-phlex-ui-components.md`
+- **Status:** draft
+
+### Part 10 — Controllers & routing in a Rails Engine
+- **Tasks:** #12
+- **Hook:** Engine controllers are tricky — here's the clean way
+- **Covers:** ImportsController, inheriting from host's parent controller, engine routes, strong params, Turbo integration
+- **Key decisions:** Dynamic parent controller inheritance, route namespacing, how auth flows from host to engine
+- **File:** `docs/blog/010-controllers-routing.md`
+- **Status:** draft
+
+### Part 11 — Generators: install & target scaffolding
+- **Tasks:** #15, #16
+- **Hook:** Great gems install in one command
+- **Covers:** Install generator (migration, initializer, routes), Target generator (column parsing, template), Rails::Generators API
+- **Key decisions:** What to generate vs. what to configure, template format (ERB .tt), route injection strategy
+- **File:** `docs/blog/011-generators.md`
+- **Status:** draft
+
+### Part 12 — Adding JSON & API sources
+- **Tasks:** #18, #19
+- **Hook:** CSV is just the beginning
+- **Covers:** JSON source (file + raw text), API source (HTTP client, params injection), source plugin architecture
+- **Key decisions:** Source abstraction design, how to handle auth headers, response_root extraction
+- **File:** `docs/blog/012-json-api-sources.md`
+- **Status:** draft
+
+### Part 13 — Testing a Rails Engine with RSpec
+- **Tasks:** #17
+- **Hook:** Testing an engine is different from testing an app
+- **Covers:** Dummy app setup, spec organization, testing StoreModels, mocking ActiveStorage, testing jobs, controller specs
+- **Key decisions:** Dummy app vs. shared examples, factory vs. fixture, what to unit test vs. integration test
+- **File:** `docs/blog/013-testing-rails-engine.md`
+- **Status:** draft
+
+### Part 14 — Dry Run: validate before you persist
+- **Tasks:** #20
+- **Hook:** Preview catches column errors, dry-run catches DB errors
+- **Covers:** Transaction + rollback pattern, enriching records with DB errors, DryRunJob, UI integration
+- **Key decisions:** Why two validation layers, transaction rollback approach, when to offer dry-run
+- **File:** `docs/blog/014-dry-run.md`
+- **Status:** draft
+
+### Part 15 — Publishing the gem & retrospective
+- **Tasks:** None (wrap-up)
+- **Hook:** From idea to rubygems.org
+- **Covers:** Gemspec final polish, CHANGELOG, versioning, publishing, series recap, what worked, what we'd do differently
+- **Key decisions:** Semantic versioning strategy, open source considerations
+- **File:** `docs/blog/015-publishing-retro.md`
+- **Status:** draft
+
+### Part 16 — ERB View Templates: Composing Phlex Components
+- **Tasks:** None (view layer completion)
+- **Hook:** Phlex components are pure Ruby. ERB is what the browser sees. The glue is `.call`.
+- **Covers:** ERB templates composing Phlex, has_one_attached bugfix, ActiveStorage test setup, view testing in Rails 8, plain CSS stylesheet
+- **Key decisions:** raw component.call vs phlex-rails, plain CSS vs build step, view test infrastructure
+- **File:** `docs/blog/016-erb-view-templates.md`
+- **Status:** draft
+
+### Part 17 — Showcase & Final Retrospective
+- **Tasks:** None (series finale)
+- **Hook:** 17 articles, 22 components, one complete gem. DataPorter in action.
+- **Covers:** Full workflow walkthrough with screenshots, architecture overview, series recap, lessons learned, what's next
+- **Key decisions:** N/A (retrospective)
+- **File:** `docs/blog/017-showcase-final-retro.md`
+- **Status:** draft

data/lib/data_porter/broadcaster.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module DataPorter
+  class Broadcaster
+    def initialize(import_id)
+      prefix = DataPorter.configuration.cable_channel_prefix
+      @channel = "#{prefix}/imports/#{import_id}"
+    end
+
+    def progress(current, total)
+      percentage = ((current.to_f / total) * 100).round
+      broadcast(status: :processing, percentage: percentage, current: current, total: total)
+    end
+
+    def success
+      broadcast(status: :success)
+    end
+
+    def failure(message)
+      broadcast(status: :failure, error: message)
+    end
+
+    private
+
+    def broadcast(message)
+      ActionCable.server.broadcast(@channel, message)
+    end
+  end
+end

data/lib/data_porter/components/base.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "phlex"
+
+module DataPorter
+  module Components
+    class Base < Phlex::HTML
+    end
+  end
+end

data/lib/data_porter/components/failure_alert.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    class FailureAlert < Base
+      def initialize(report:)
+        super()
+        @report = report
+      end
+
+      def view_template
+        div(class: "dp-alert dp-alert--danger") do
+          @report.error_reports.each do |err|
+            p { err.message }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components/preview_table.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    class PreviewTable < Base
+      def initialize(columns:, records:)
+        super()
+        @columns = columns
+        @records = records
+      end
+
+      def view_template
+        div(class: "dp-preview-table") do
+          table(class: "dp-table") do
+            render_header
+            render_body
+          end
+        end
+      end
+
+      private
+
+      def render_header
+        thead do
+          tr do
+            th { "#" }
+            th { "Status" }
+            @columns.each { |col| th { col.label } }
+            th { "Errors" }
+          end
+        end
+      end
+
+      def render_body
+        tbody do
+          @records.each { |record| render_row(record) }
+        end
+      end
+
+      def render_row(record)
+        tr(class: "dp-row--#{record.status}") do
+          td { record.line_number.to_s }
+          td { record.status }
+          @columns.each { |col| td { record.data[col.name.to_s].to_s } }
+          td(class: "dp-errors") { error_messages(record) }
+        end
+      end
+
+      def error_messages(record)
+        record.errors_list.map(&:message).join(", ")
+      end
+    end
+  end
+end

data/lib/data_porter/components/progress_bar.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    class ProgressBar < Base
+      def initialize(import_id:)
+        super()
+        @import_id = import_id
+      end
+
+      def view_template
+        div(class: "dp-progress", **stimulus_controller_attrs) do
+          render_bar
+        end
+      end
+
+      private
+
+      def render_bar
+        div(class: "dp-progress-bar", data_data_porter__progress_target: "bar", style: "width: 0%") do
+          span(data_data_porter__progress_target: "text") { "0%" }
+        end
+      end
+
+      def stimulus_controller_attrs
+        {
+          data_controller: "data-porter--progress",
+          data_data_porter__progress_id_value: @import_id.to_s
+        }
+      end
+    end
+  end
+end

data/lib/data_porter/components/results_summary.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    class ResultsSummary < Base
+      def initialize(report:)
+        super()
+        @report = report
+      end
+
+      def view_template
+        div(class: "dp-results") do
+          p { "Created: #{@report.imported_count}" }
+          p { "Errors: #{@report.errored_count}" }
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components/status_badge.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    class StatusBadge < Base
+      def initialize(status:)
+        super()
+        @status = status.to_s
+      end
+
+      def view_template
+        span(class: "dp-badge dp-badge--#{@status}") { @status.capitalize }
+      end
+    end
+  end
+end

data/lib/data_porter/components/summary_cards.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module Components
+    class SummaryCards < Base
+      def initialize(report:)
+        super()
+        @report = report
+      end
+
+      def view_template
+        div(class: "dp-summary-cards") do
+          card("dp-card--complete", @report.complete_count, "Ready")
+          card("dp-card--partial", @report.partial_count, "Incomplete")
+          card("dp-card--missing", @report.missing_count, "Missing")
+          card("dp-card--duplicate", @report.duplicate_count, "Duplicates")
+        end
+      end
+
+      private
+
+      def card(css_class, count, label)
+        div(class: "dp-card #{css_class}") do
+          strong { count.to_s }
+          plain " #{label}"
+        end
+      end
+    end
+  end
+end

data/lib/data_porter/components.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require_relative "components/base"
+require_relative "components/status_badge"
+require_relative "components/summary_cards"
+require_relative "components/preview_table"
+require_relative "components/progress_bar"
+require_relative "components/results_summary"
+require_relative "components/failure_alert"
+
+module DataPorter
+  module Components
+  end
+end

data/lib/data_porter/configuration.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module DataPorter
+  class Configuration
+    attr_accessor :parent_controller,
+                  :queue_name,
+                  :storage_service,
+                  :cable_channel_prefix,
+                  :context_builder,
+                  :preview_limit,
+                  :enabled_sources,
+                  :scope
+
+    def initialize
+      @parent_controller = "ApplicationController"
+      @queue_name = :imports
+      @storage_service = :local
+      @cable_channel_prefix = "data_porter"
+      @context_builder = nil
+      @preview_limit = 500
+      @enabled_sources = %i[csv json api]
+      @scope = nil
+    end
+  end
+end

data/lib/data_porter/dsl/api_config.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module DSL
+    class ApiConfig
+      def endpoint(value = nil)
+        return @endpoint if value.nil?
+
+        @endpoint = value
+      end
+
+      def headers(value = nil)
+        return @headers if value.nil?
+
+        @headers = value
+      end
+
+      def response_root(value = nil)
+        return @response_root if value.nil?
+
+        @response_root = value
+      end
+    end
+  end
+end

data/lib/data_porter/dsl/column.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module DataPorter
+  module DSL
+    Column = Struct.new(:name, :type, :required, :label, :options, keyword_init: true) do
+      def initialize(name:, type: :string, required: false, label: nil, **options)
+        super(
+          name: name.to_sym,
+          type: type.to_sym,
+          required: required,
+          label: label || name.to_s.humanize,
+          options: options
+        )
+      end
+    end
+  end
+end

data/lib/data_porter/engine.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module DataPorter
+  class Engine < ::Rails::Engine
+    isolate_namespace DataPorter
+
+    initializer "data_porter.assets.precompile" do |app|
+      app.config.assets.precompile += %w[data_porter/application.css] if app.config.respond_to?(:assets)
+    end
+
+    config.to_prepare do
+      Dir[Rails.root.join("app/importers/*_target.rb")].each { |f| require f }
+    end
+  end
+end

data/lib/data_porter/orchestrator.rb

@@ -0,0 +1,141 @@
+# frozen_string_literal: true
+
+module DataPorter
+  class Orchestrator
+    def initialize(data_import, content: nil)
+      @data_import = data_import
+      @target = data_import.target_class.new
+      @source_options = { content: content }.compact
+    end
+
+    def parse!
+      @data_import.parsing!
+      records = build_records
+      @data_import.update!(records: records, status: :previewing)
+      build_report
+    rescue StandardError => e
+      handle_failure(e)
+    end
+
+    def import!
+      @data_import.importing!
+      results = import_records
+      update_import_report(results)
+      @target.after_import(results, context: build_context)
+    rescue StandardError => e
+      handle_failure(e)
+    end
+
+    def dry_run!
+      @data_import.dry_running!
+      run_dry_run_records
+      @data_import.update!(status: :previewing)
+      build_report
+    rescue StandardError => e
+      handle_failure(e)
+    end
+
+    private
+
+    def build_records
+      source = @data_import.source_class.new(@data_import, **@source_options)
+      raw_rows = source.fetch
+      columns = @target.class._columns || []
+      validator = RecordValidator.new(columns)
+
+      raw_rows.each_with_index.map do |row, index|
+        build_record(row, index, columns, validator)
+      end
+    end
+
+    def build_record(row, index, columns, validator)
+      record = StoreModels::ImportRecord.new(
+        line_number: index + 1,
+        data: extract_data(row, columns)
+      )
+      record = @target.transform(record)
+      @target.validate(record)
+      validator.validate(record)
+      record.determine_status!
+      record
+    end
+
+    def extract_data(row, columns)
+      columns.each_with_object({}) do |col, hash|
+        hash[col.name] = row[col.name] || row[col.name.to_s]
+      end
+    end
+
+    def run_dry_run_records
+      records = @data_import.records
+      importable = records.select(&:importable?)
+      context = build_context
+
+      importable.each do |record|
+        dry_run_record(record, context)
+      end
+
+      @data_import.records_will_change!
+      @data_import.update!(records: records)
+    end
+
+    def dry_run_record(record, context)
+      @target.persist(record, context: context)
+      record.dry_run_passed = true
+    rescue StandardError => e
+      record.dry_run_passed = false
+      record.add_error(e.message)
+    end
+
+    def import_records
+      importable = @data_import.importable_records
+      context = build_context
+      results = { created: 0, errored: 0 }
+
+      importable.each do |record|
+        persist_record(record, context, results)
+      end
+
+      @data_import.update!(status: :completed)
+      results
+    end
+
+    def persist_record(record, context, results)
+      @target.persist(record, context: context)
+      results[:created] += 1
+    rescue StandardError => e
+      record.add_error(e.message)
+      @target.on_error(record, e, context: context)
+      results[:errored] += 1
+    end
+
+    def update_import_report(results)
+      report = @data_import.report || StoreModels::Report.new
+      report.imported_count = results[:created]
+      report.errored_count = results[:errored]
+      @data_import.update!(report: report)
+    end
+
+    def build_report
+      summary = @data_import.records_summary
+      report = StoreModels::Report.new(
+        records_count: @data_import.records.size,
+        complete_count: summary["complete"] || 0,
+        partial_count: summary["partial"] || 0,
+        missing_count: summary["missing"] || 0
+      )
+      @data_import.update!(report: report)
+    end
+
+    def build_context
+      DataPorter.configuration.context_builder&.call(@data_import)
+    end
+
+    def handle_failure(error)
+      report = StoreModels::Report.new(
+        error_reports: [StoreModels::Error.new(message: error.message)]
+      )
+      @data_import.update!(status: :failed, report: report)
+    end
+  end
+end

data/lib/data_porter/record_validator.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module DataPorter
+  class RecordValidator
+    def initialize(columns)
+      @columns = columns
+    end
+
+    def validate(record)
+      @columns.each do |col|
+        value = record.data[col.name]
+        validate_required(record, col, value)
+        validate_type(record, col, value)
+      end
+    end
+
+    private
+
+    def validate_required(record, col, value)
+      return unless col.required && value.to_s.strip.empty?
+
+      record.add_error("#{col.label} is required")
+    end
+
+    def validate_type(record, col, value)
+      return if value.to_s.strip.empty?
+      return if TypeValidator.valid?(value, col.type, col.options)
+
+      record.add_error("#{col.label}: invalid #{col.type}")
+    end
+  end
+end