data_porter 0.1.0

data/.taskmaster/state.json

@@ -0,0 +1,6 @@
+{
+  "currentTag": "master",
+  "lastSwitched": "2026-02-06T11:26:19.556Z",
+  "branchTagMapping": {},
+  "migrationNoticeShown": false
+}

data/.taskmaster/tasks/task_001.md

@@ -0,0 +1,19 @@
+# Task ID: 1
+
+**Title:** Setup gem structure and Rails Engine
+
+**Status:** pending
+
+**Dependencies:** None
+
+**Priority:** high
+
+**Description:** Set up the core gem directory structure as defined in the spec (app/, lib/, config/, spec/) and configure the Rails Engine with isolate_namespace DataPorter. Include engine.rb with asset precompilation initializer, ActiveStorage initializer, and auto-discovery of target files from app/importers/.
+
+**Details:**
+
+Create the full directory tree: app/{channels,controllers,jobs,models,views}/data_porter/, lib/data_porter/{sources,store_models,dsl}/, config/, spec/. The Engine class should inherit from ::Rails::Engine, call isolate_namespace DataPorter, and set up initializers for assets, active_storage, and target auto-discovery via config.to_prepare. Update data_porter.gemspec with dependencies: rails (>= 7.0), store_model, phlex, turbo-rails, stimulus-rails. Update lib/data_porter.rb to require all sub-modules.
+
+**Test Strategy:**
+
+Verify the engine loads without errors. Test that isolate_namespace is correctly applied. Test that the initializer discovers target files.

data/.taskmaster/tasks/task_002.md

@@ -0,0 +1,19 @@
+# Task ID: 2
+
+**Title:** Implement Configuration module
+
+**Status:** pending
+
+**Dependencies:** 1
+
+**Priority:** high
+
+**Description:** Create DataPorter::Configuration class with all config options and DataPorter.configure DSL. Options: parent_controller, queue_name, storage_service, cable_channel_prefix, context_builder, preview_limit, enabled_sources, scope.
+
+**Details:**
+
+Create lib/data_porter/configuration.rb with attr_accessors for: parent_controller (default: 'ApplicationController'), queue_name (default: :imports), storage_service (default: :local), cable_channel_prefix (default: 'data_porter'), context_builder (default: nil, accepts a lambda), preview_limit (default: 500), enabled_sources (default: [:csv, :json, :api]), scope (default: nil). Add DataPorter.configure class method yielding the config singleton, and DataPorter.configuration accessor.
+
+**Test Strategy:**
+
+Test default values. Test that configure block sets values correctly. Test that configuration is accessible via DataPorter.configuration.

data/.taskmaster/tasks/task_003.md

@@ -0,0 +1,19 @@
+# Task ID: 3
+
+**Title:** Create StoreModels (ImportRecord, Error, Report)
+
+**Status:** pending
+
+**Dependencies:** 1
+
+**Priority:** high
+
+**Description:** Implement the three StoreModel classes for JSONB storage: ImportRecord (line_number, status, data, errors_list, warnings, target_id), Error (message), Report (records_count, complete_count, partial_count, missing_count, duplicate_count, imported_count, errored_count, error_reports).
+
+**Details:**
+
+Create lib/data_porter/store_models/error.rb with StoreModel::Model including attribute :message. Create lib/data_porter/store_models/import_record.rb with attributes: line_number (:integer), status (:string, default: 'pending'), data (:json, default: {}), errors_list (Error.to_array_type), warnings (Error.to_array_type), target_id (:integer), dry_run_passed (:boolean, default: false). Include STATUSES constant, helper methods (complete?, importable?, add_error, add_warning, attributes), and run_validations! method with validate_required_columns!, validate_types!, validate_inclusions!, check_duplicates!, determine_status!. Create lib/data_porter/store_models/report.rb with all count attributes.
+
+**Test Strategy:**
+
+Test each StoreModel initializes with defaults. Test ImportRecord validation methods. Test status determination logic. Test add_error and add_warning methods.

data/.taskmaster/tasks/task_004.md

@@ -0,0 +1,19 @@
+# Task ID: 4
+
+**Title:** Implement TypeValidator
+
+**Status:** pending
+
+**Dependencies:** 1
+
+**Priority:** high
+
+**Description:** Create DataPorter::TypeValidator module with .valid?(value, type, options) class method supporting types: :string, :integer, :decimal, :date, :datetime, :email, :phone, :url, :boolean.
+
+**Details:**
+
+Create lib/data_porter/type_validator.rb. Implement validation for each type: string (always valid if present), integer (Integer(value) rescue false), decimal (Float(value) rescue false), date (Date.parse or strptime with format option), datetime (DateTime.parse), email (regex validation), phone (regex for common formats), url (URI.parse with scheme check), boolean (true/false/'true'/'false'/0/1).
+
+**Test Strategy:**
+
+Test each type with valid and invalid values. Test date with custom format option. Test edge cases (nil, empty string, weird types).

data/.taskmaster/tasks/task_005.md

@@ -0,0 +1,19 @@
+# Task ID: 5
+
+**Title:** Build Target DSL base class
+
+**Status:** pending
+
+**Dependencies:** 3, 4
+
+**Priority:** high
+
+**Description:** Implement DataPorter::Target base class with all DSL class methods: label, model, icon, sources, columns, csv_mapping, deduplicate_by, params, api_config. Include overridable instance hooks: transform, validate, persist, after_import, on_error.
+
+**Details:**
+
+Create lib/data_porter/target.rb. Implement class-level DSL using class_attribute or class instance variables with accessor methods: _label, _model, _icon, _sources, _columns, _csv_mappings, _dedup_keys, _params, _api_config, _dry_run_enabled, _rollback_enabled, _rollback_window, _rollback_strategy. Create DSL sub-modules in lib/data_porter/dsl/: columns_dsl.rb (Column struct with name, type, required, options, label), csv_mapping_dsl.rb, api_config_dsl.rb, params_dsl.rb. Default hook implementations: transform returns record unchanged, validate is no-op, persist raises NotImplementedError, after_import is no-op, on_error is no-op.
+
+**Test Strategy:**
+
+Test DSL methods set class attributes correctly. Test a sample target definition. Test default hooks. Test column definition with all options. Test inheritance of DSL attributes.

data/.taskmaster/tasks/task_006.md

@@ -0,0 +1,19 @@
+# Task ID: 6
+
+**Title:** Implement Registry and auto-discovery
+
+**Status:** pending
+
+**Dependencies:** 5
+
+**Priority:** high
+
+**Description:** Create DataPorter::Registry singleton with class methods: register, find, available, refresh!. Targets auto-register on inheritance using Target.descendants.
+
+**Details:**
+
+Create lib/data_porter/registry.rb. The Registry stores targets in a Hash keyed by parameterized label. register(key, klass) adds to the hash. find(key) looks up or raises DataPorter::TargetNotFound. available returns array of {key:, label:, icon:} hashes. refresh! clears and re-registers all Target descendants that have a _label set. Define DataPorter::TargetNotFound error class.
+
+**Test Strategy:**
+
+Test register and find. Test TargetNotFound error. Test available returns correct format. Test refresh! discovers target subclasses.

data/.taskmaster/tasks/task_007.md

@@ -0,0 +1,19 @@
+# Task ID: 7
+
+**Title:** Create DataImport model and migration
+
+**Status:** pending
+
+**Dependencies:** 3, 6
+
+**Priority:** high
+
+**Description:** Implement DataPorter::DataImport ActiveRecord model with enum status, JSONB attributes (records, report, config), ActiveStorage file attachment, polymorphic user association. Create the migration template.
+
+**Details:**
+
+Create app/models/data_porter/data_import.rb with: table_name 'data_porter_imports', belongs_to :user (polymorphic), has_one_attached :file, enum :status (pending:0, parsing:1, previewing:2, importing:3, completed:4, failed:5, dry_running:6, rolling_back:7, rolled_back:8), StoreModel attributes for records and report, serialized config. Add validations for target_key and source_type. Implement helper methods: target_class, source_class, previewable?, importable_records, records_summary. Create migration template in lib/generators/data_porter/templates/ with proper schema (target_key, source_type, status, records JSONB, report JSONB, config JSONB, user polymorphic refs, timestamps, indexes).
+
+**Test Strategy:**
+
+Test enum values. Test validations. Test helper methods. Test StoreModel attribute types. Test associations.

data/.taskmaster/tasks/task_008.md

@@ -0,0 +1,19 @@
+# Task ID: 8
+
+**Title:** Implement Source base class and CSV source
+
+**Status:** pending
+
+**Dependencies:** 5, 7
+
+**Priority:** high
+
+**Description:** Create DataPorter::Sources::Base abstract class and DataPorter::Sources::Csv implementation. Include DataPorter::Sources.resolve class method for source type lookup.
+
+**Details:**
+
+Create lib/data_porter/sources/base.rb with initialize(data_import), abstract fetch method, and private apply_csv_mapping helper (uses target's _csv_mappings or falls back to parameterize auto-mapping). Create lib/data_porter/sources/csv.rb extending Base: fetch reads the attached file via ActiveStorage, parses with Ruby CSV (headers: true + target's csv_options), applies csv_mapping to each row. Create lib/data_porter/sources.rb with resolve(type) method that maps 'csv' -> Csv, 'json' -> Json, 'api' -> Api.
+
+**Test Strategy:**
+
+Test CSV parsing with headers. Test csv_mapping application (explicit and auto). Test resolve method returns correct class. Test with various CSV formats (different separators, encodings).

data/.taskmaster/tasks/task_009.md

@@ -0,0 +1,19 @@
+# Task ID: 9
+
+**Title:** Build Orchestrator (parse and import flows)
+
+**Status:** pending
+
+**Dependencies:** 7, 8
+
+**Priority:** high
+
+**Description:** Implement DataPorter::Orchestrator that coordinates source + target for parse! and import! operations. Handles the full workflow: fetch data, transform, validate, build records, persist.
+
+**Details:**
+
+Create lib/data_porter/orchestrator.rb. initialize(data_import) sets up @source and @target instances. parse!: transitions to parsing, fetches raw rows from source, maps to ImportRecord StoreModels (applying transform and validate hooks, running run_validations!), broadcasts progress, saves records and transitions to previewing, builds report. import!: transitions to importing, iterates importable_records, calls target.persist for each with context, handles per-record errors via on_error hook, updates counts, transitions to completed, calls after_import. Private methods: broadcast_progress, broadcast_success, handle_failure (transitions to failed with error report), build_context (calls context_builder), build_report.
+
+**Test Strategy:**
+
+Test parse! creates ImportRecords from source data. Test import! calls persist for each importable record. Test error handling during parse and import. Test status transitions. Test progress broadcasting.

data/.taskmaster/tasks/task_010.md

@@ -0,0 +1,19 @@
+# Task ID: 10
+
+**Title:** Create ActiveJob classes (ParseJob, ImportJob)
+
+**Status:** pending
+
+**Dependencies:** 9
+
+**Priority:** high
+
+**Description:** Implement DataPorter::ParseJob and DataPorter::ImportJob that delegate to the Orchestrator. Both use configurable queue_name.
+
+**Details:**
+
+Create app/jobs/data_porter/parse_job.rb: inherits ActiveJob::Base, queue_as from DataPorter.configuration.queue_name, perform(import_id) finds DataImport and calls Orchestrator.new(data_import).parse!. Create app/jobs/data_porter/import_job.rb: same pattern but calls orchestrator.import!.
+
+**Test Strategy:**
+
+Test jobs enqueue correctly. Test they find the DataImport and call the right orchestrator method. Test queue configuration.

data/.taskmaster/tasks/task_011.md

@@ -0,0 +1,19 @@
+# Task ID: 11
+
+**Title:** Implement ActionCable Broadcaster and ImportChannel
+
+**Status:** pending
+
+**Dependencies:** 1
+
+**Priority:** medium
+
+**Description:** Create DataPorter::Broadcaster for real-time progress updates and DataPorter::ImportChannel for client subscriptions.
+
+**Details:**
+
+Create lib/data_porter/broadcaster.rb: initialize(import_id) builds channel name from cable_channel_prefix config. progress(current, total) broadcasts percentage. success() broadcasts success status. failure(message) broadcasts failure with error. Create app/channels/data_porter/import_channel.rb: subscribes to the import-specific channel stream.
+
+**Test Strategy:**
+
+Test broadcaster formats messages correctly. Test channel subscribes to correct stream. Test progress percentage calculation.

data/.taskmaster/tasks/task_012.md

@@ -0,0 +1,19 @@
+# Task ID: 12
+
+**Title:** Build ImportsController with all actions
+
+**Status:** pending
+
+**Dependencies:** 7, 10
+
+**Priority:** high
+
+**Description:** Implement DataPorter::ImportsController inheriting from configured parent_controller. Actions: index, new, create, show, parse, confirm, cancel. Wire up routes.
+
+**Details:**
+
+Create app/controllers/data_porter/imports_controller.rb inheriting from DataPorter.configuration.parent_controller.constantize. before_action :set_import for show/parse/confirm/cancel. index: list imports ordered by created_at desc. new: build new DataImport, load available targets. create: build DataImport from params, assign current_user, enqueue ParseJob on save. show: load target, records, grouped records. parse: reset to pending and re-enqueue ParseJob. confirm: enqueue ImportJob. cancel: set status to failed. Create config/routes.rb with DataPorter::Engine.routes.draw: resources :imports with member routes for parse, confirm, cancel.
+
+**Test Strategy:**
+
+Test each controller action. Test strong params. Test job enqueuing on create and confirm. Test status transitions on parse and cancel.

data/.taskmaster/tasks/task_013.md

@@ -0,0 +1,19 @@
+# Task ID: 13
+
+**Title:** Create Phlex view components for the UI
+
+**Status:** pending
+
+**Dependencies:** 12
+
+**Priority:** high
+
+**Description:** Build all UI components using Phlex with Tailwind CSS (dp- prefix, .data-porter scope). Components: Layout, ImportsList, NewImportForm, ImportShow (with Preview table, Progress bar, Summary cards, Actions, Results, Failure alert).
+
+**Details:**
+
+Create Phlex components in lib/data_porter/components/ or app/views/data_porter/components/. All Tailwind classes use dp- prefix. Root wrapper uses .data-porter class for scoping. Components: ImportsList (table with status badges, links), NewImportForm (target select, source type, file upload, config fields), ImportShow (delegates to sub-components based on status), PreviewTable (dynamic columns from target, status icons, error display, row coloring by status), ProgressBar (Stimulus-connected, percentage display), SummaryCards (complete/partial/missing/duplicate counts), ActionButtons (cancel, confirm with count), ResultsSummary (created/errored counts), FailureAlert (error messages, retry button). Use CSS custom properties for theming: --dp-color-complete, --dp-color-partial, --dp-color-missing, --dp-color-primary.
+
+**Test Strategy:**
+
+Test each component renders correct HTML structure. Test conditional rendering based on import status. Test dynamic columns from target. Test CSS class prefixing.

data/.taskmaster/tasks/task_014.md

@@ -0,0 +1,19 @@
+# Task ID: 14
+
+**Title:** Implement Stimulus progress controller
+
+**Status:** pending
+
+**Dependencies:** 11, 13
+
+**Priority:** medium
+
+**Description:** Create the Stimulus controller for real-time progress updates via ActionCable. Connects to ImportChannel, updates progress bar and auto-reloads on completion.
+
+**Details:**
+
+Create app/javascript/data_porter/progress_controller.js (or app/assets/javascript/). Stimulus controller with targets: bar, text. Values: id (Number). On connect: subscribe to ImportChannel with the import id. On received: if status is 'processing', update bar width and text with percentage. If status is 'success' or 'failure', reload page. On disconnect: unsubscribe.
+
+**Test Strategy:**
+
+Test controller connects to correct channel. Test progress bar updates. Test page reload on completion.

data/.taskmaster/tasks/task_015.md

@@ -0,0 +1,19 @@
+# Task ID: 15
+
+**Title:** Create install generator
+
+**Status:** pending
+
+**Dependencies:** 7, 2
+
+**Priority:** medium
+
+**Description:** Implement rails generate data_porter:install generator that creates migration, initializer, importers directory, and mounts the engine in routes.
+
+**Details:**
+
+Create lib/generators/data_porter/install_generator.rb. It should: copy migration template (create_data_porter_imports) with proper timestamp, create config/initializers/data_porter.rb with commented configuration options, create empty app/importers/ directory, inject mount DataPorter::Engine at: '/imports' into config/routes.rb. Use Rails::Generators::Base with source_root pointing to templates/.
+
+**Test Strategy:**
+
+Test generator creates all expected files. Test migration has correct schema. Test initializer has all config options. Test route injection.

data/.taskmaster/tasks/task_016.md

@@ -0,0 +1,19 @@
+# Task ID: 16
+
+**Title:** Create target generator
+
+**Status:** pending
+
+**Dependencies:** 5
+
+**Priority:** medium
+
+**Description:** Implement rails generate data_porter:target NAME [columns...] generator that scaffolds a target class in app/importers/.
+
+**Details:**
+
+Create lib/generators/data_porter/target_generator.rb. Arguments: name (required), columns (optional, format: name:type or name:type:required). Generate app/importers/{name}_target.rb with: class inheriting DataPorter::Target, label, model (inferred from name), sources :csv, columns block from arguments, empty persist method. Template in lib/generators/data_porter/templates/target.rb.tt.
+
+**Test Strategy:**
+
+Test generator creates target file with correct class name. Test column parsing (type, required flag). Test generated file is valid Ruby.

data/.taskmaster/tasks/task_017.md

@@ -0,0 +1,19 @@
+# Task ID: 17
+
+**Title:** Write RSpec test suite for core modules
+
+**Status:** pending
+
+**Dependencies:** 9
+
+**Priority:** high
+
+**Description:** Create comprehensive RSpec tests for all core modules: Configuration, StoreModels, TypeValidator, Target DSL, Registry, Sources, Orchestrator, DataImport model.
+
+**Details:**
+
+Set up spec/spec_helper.rb with proper requires and test configuration. Create specs: spec/data_porter/configuration_spec.rb, spec/data_porter/store_models/import_record_spec.rb, spec/data_porter/store_models/error_spec.rb, spec/data_porter/store_models/report_spec.rb, spec/data_porter/type_validator_spec.rb, spec/data_porter/target_spec.rb (test DSL), spec/data_porter/registry_spec.rb, spec/data_porter/sources/csv_spec.rb, spec/data_porter/orchestrator_spec.rb, spec/models/data_porter/data_import_spec.rb. Use fixtures or factories for test data. Mock ActiveStorage for file tests.
+
+**Test Strategy:**
+
+Run bundle exec rspec and verify all tests pass. Aim for high coverage of business logic.

data/.taskmaster/tasks/task_018.md

@@ -0,0 +1,19 @@
+# Task ID: 18
+
+**Title:** Implement JSON source
+
+**Status:** pending
+
+**Dependencies:** 8
+
+**Priority:** medium
+
+**Description:** Create DataPorter::Sources::Json for JSON file upload and raw JSON text input. Supports configurable json_root for nested data extraction.
+
+**Details:**
+
+Create lib/data_porter/sources/json.rb extending Base. fetch method: reads from attached file (download) or from config['raw_json']. Parses JSON, extracts records from json_root path (dot-separated) or root array. Transforms keys to parameterized symbols. Add _json_root DSL to Target class.
+
+**Test Strategy:**
+
+Test parsing from file attachment. Test parsing from raw_json config. Test json_root extraction. Test key transformation.

data/.taskmaster/tasks/task_019.md

@@ -0,0 +1,19 @@
+# Task ID: 19
+
+**Title:** Implement API source
+
+**Status:** pending
+
+**Dependencies:** 8
+
+**Priority:** medium
+
+**Description:** Create DataPorter::Sources::Api for HTTP endpoint fetching. Uses target's api_config (endpoint, headers, response_root) with params injection.
+
+**Details:**
+
+Create lib/data_porter/sources/api.rb extending Base. fetch method: resolves endpoint URL (lambda or string) with params from data_import.config, resolves headers (lambda or hash), makes HTTP GET with Net::HTTP, parses JSON response, extracts records from response_root, transforms keys. Add api_config DSL block to Target with endpoint, headers, response_root setters.
+
+**Test Strategy:**
+
+Test with mocked HTTP responses. Test endpoint lambda resolution. Test header resolution. Test response_root extraction. Test error handling for failed requests.

data/.taskmaster/tasks/task_020.md

@@ -0,0 +1,19 @@
+# Task ID: 20
+
+**Title:** Implement Dry Run mode
+
+**Status:** pending
+
+**Dependencies:** 9, 10
+
+**Priority:** low
+
+**Description:** Add dry_run! method to Orchestrator that wraps persist calls in a transaction and rolls back. Enriches ImportRecords with DB-level validation errors. Add DryRunJob and controller action.
+
+**Details:**
+
+Add dry_run! to Orchestrator: transitions to dry_running, wraps all persist calls in ActiveRecord::Base.transaction, catches RecordInvalid/RecordNotUnique/StandardError per record, adds DB errors to import records, raises ActiveRecord::Rollback at end, resets target_ids, transitions back to previewing, rebuilds report. Create app/jobs/data_porter/dry_run_job.rb. Add dry_run action to controller. Add dry_run_enabled DSL flag to Target. Add dry_run_passed attribute to ImportRecord StoreModel.
+
+**Test Strategy:**
+
+Test that dry_run! rolls back all DB changes. Test that DB validation errors are captured on records. Test status transitions (previewing -> dry_running -> previewing). Test dry_run_enabled flag controls UI button visibility.