good_pipeline 0.1.0

data/docs/architecture.md

@@ -0,0 +1,184 @@
+# Architecture
+
+This page describes GoodPipeline's internal architecture for contributors and advanced users who want to understand how the system works.
+
+## Layer diagram
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                        DSL Layer                            │
+│    Pipeline.configure defines the DAG topology              │
+│    Step keys are graph identity; job classes are impl       │
+└─────────────────────────┬───────────────────────────────────┘
+                          │
+┌─────────────────────────▼───────────────────────────────────┐
+│                  Validation Layer                            │
+│    DAG validated at instantiation time, before DB writes    │
+│    Cycles, unknown keys, duplicates rejected upfront        │
+└─────────────────────────┬───────────────────────────────────┘
+                          │
+┌─────────────────────────▼───────────────────────────────────┐
+│                     State Layer                             │
+│    Postgres tables are the authoritative source of truth    │
+│    coordination_status is the sole input for decisions      │
+│    halt_triggered flag drives :halted derivation            │
+└─────────────────────────┬───────────────────────────────────┘
+                          │
+┌─────────────────────────▼───────────────────────────────────┐
+│                   Execution Layer                            │
+│    One GoodJob::Batch per step                              │
+│    User jobs enqueued via perform_later — fully untouched   │
+│    Batch on_finish is the sole terminal signal              │
+│    Enqueue is transactionally coupled to row transition     │
+└─────────────────────────┬───────────────────────────────────┘
+                          │
+┌─────────────────────────▼───────────────────────────────────┐
+│                  Coordination Layer                          │
+│    Coordinator owns ALL coordination_status transitions     │
+│    Atomic row-locked transitions (FOR UPDATE SKIP LOCKED)   │
+│    Explicit transaction boundaries per atomic unit          │
+│    recompute_pipeline_status is the sole derivation path    │
+└─────────────────────────┬───────────────────────────────────┘
+                          │
+┌─────────────────────────▼───────────────────────────────────┐
+│                    Chain Layer                               │
+│    .then() wires pipeline-level DAG dependencies            │
+│    Same coordinator pattern, one level up                   │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Data model
+
+GoodPipeline uses four Postgres tables:
+
+### `good_pipeline_pipelines`
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | uuid | Primary key |
+| `type` | string | Pipeline class name (e.g. `"VideoProcessingPipeline"`) |
+| `params` | jsonb | Arguments passed to `.run()` |
+| `status` | string | `pending`, `running`, `succeeded`, `failed`, `halted`, `skipped` |
+| `halt_triggered` | boolean | Set to `true` when `:halt` strategy is applied |
+| `good_job_batch_id` | uuid | Pipeline-level GoodJob::Batch for grouping |
+| `on_failure_strategy` | string | `halt`, `continue`, or `ignore` |
+| `callbacks_dispatched_at` | timestamp | Exactly-once callback dispatch guard |
+| `created_at` | timestamp | |
+| `updated_at` | timestamp | |
+
+### `good_pipeline_steps`
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | uuid | Primary key |
+| `pipeline_id` | uuid | Foreign key |
+| `key` | string | Step key — graph identity |
+| `job_class` | string | ActiveJob class name |
+| `params` | jsonb | Arguments passed to `with:` |
+| `coordination_status` | string | `pending`, `enqueued`, `succeeded`, `failed`, `skipped` |
+| `on_failure_strategy` | string | Step-level override (nullable) |
+| `queue` | string | Optional queue override |
+| `priority` | integer | Optional priority override |
+| `good_job_batch_id` | uuid | Step's own GoodJob::Batch |
+| `good_job_id` | uuid | GoodJob record ID (nil until enqueued) |
+| `attempts` | integer | Execution attempt count |
+| `error_class` | string | Terminal failure error class |
+| `error_message` | text | Terminal failure error message |
+| `created_at` | timestamp | |
+| `updated_at` | timestamp | |
+
+Unique constraint: `(pipeline_id, key)` — enforces step key uniqueness within a pipeline.
+
+### `good_pipeline_dependencies`
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | bigint | Primary key |
+| `pipeline_id` | uuid | Denormalized for fast querying |
+| `step_id` | uuid | The dependent step |
+| `depends_on_step_id` | uuid | The step that must complete first |
+
+### `good_pipeline_chains`
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | uuid | Primary key |
+| `upstream_pipeline_id` | uuid | The pipeline that must finish first |
+| `downstream_pipeline_id` | uuid | The pipeline to start after |
+
+## One batch per step
+
+Each step has its own `GoodJob::Batch`. The user's job is enqueued via `perform_later` into that batch, preserving all ActiveJob semantics (instrumentation, callbacks, serialization, queue routing, retries, `discard_on`).
+
+When the batch's `on_finish` fires, `StepFinishedJob` receives the signal and delegates to the coordinator. `StepFinishedJob` is a thin dispatcher — it does not own any state transitions.
+
+## The coordinator
+
+The `Coordinator` class is the sole owner of all `coordination_status` transitions. Its `complete_step` method uses explicit transaction boundaries around three atomic units:
+
+1. **Terminal step transition** — marks the step `succeeded` or `failed` with metadata
+2. **Halt propagation** — sets `halt_triggered` and skips all pending steps (if `:halt` strategy)
+3. **Downstream unblocking** — checks and enqueues each downstream step independently via `try_enqueue_step`, which acquires a per-step row lock
+
+These three units are intentionally **not** wrapped in a single outer transaction to avoid holding locks across multiple downstream step enqueues under high-parallelism pipelines.
+
+After the three units complete, `complete_step` calls `recompute_pipeline_status` to derive the pipeline's terminal state from the current database state.
+
+## Terminal state derivation
+
+Pipeline terminal status is **never inferred from a single event**. It is always derived by `recompute_pipeline_status`, which reads all step `coordination_status` values and the `halt_triggered` flag:
+
+| Condition | Derived status |
+|---|---|
+| Any step is `pending` or `enqueued` | Not terminal — still running |
+| All steps terminal, none `failed` | `succeeded` |
+| All steps terminal, at least one `failed`, `halt_triggered` is `true` | `halted` |
+| All steps terminal, at least one `failed`, `halt_triggered` is `false` | `failed` |
+
+This function is safe to call from multiple code paths (coordinator, batch reconciliation) because it is idempotent on terminal pipelines.
+
+## Enqueue transaction contract
+
+The transition of a step from `pending` to `enqueued` and the insertion of the corresponding GoodJob record happen inside a **single database transaction**. This is possible because GoodJob stores jobs in Postgres — the same database as GoodPipeline's tables.
+
+If the transaction rolls back, both the step status revert and the GoodJob record insertion are cancelled atomically. No stuck-enqueued steps, no ghost jobs.
+
+## Concurrency safety
+
+### The double-enqueue problem
+
+If two upstream steps of a shared downstream complete simultaneously, both coordinator invocations may try to enqueue the downstream step. GoodPipeline prevents this with:
+
+1. **`FOR UPDATE SKIP LOCKED`** — one coordinator acquires the row lock; the other skips silently
+2. **`good_job_id` null guard** — a non-null `good_job_id` is conclusive proof the step was already enqueued (only valid inside a row lock)
+3. **Status guard** — the coordinator checks `coordination_status == "pending"` inside the lock
+
+### Callback exactly-once
+
+`dispatch_callbacks_once` uses a `FOR UPDATE` locked transaction with the `callbacks_dispatched_at` timestamp as a guard. Even if `recompute_pipeline_status` is called concurrently from multiple code paths, the callback bundle fires exactly once.
+
+## Retry model
+
+GoodPipeline never inspects exceptions during retry attempts. It only responds to the **terminal signal** from the step batch's `on_finish` callback:
+
+| GoodJob outcome | GoodPipeline response |
+|---|---|
+| Job completed successfully | Step → `succeeded` |
+| Job raised, retries remaining | No action — `on_finish` hasn't fired |
+| Job exhausted retries | Step → `failed` |
+| Job discarded via `discard_on` | Step → `failed` |
+
+This ensures a step is never prematurely marked `failed` on attempt 1 of 5.
+
+## Key design decisions
+
+1. **Postgres only** — all state in Postgres, enabling atomic enqueue transactions
+2. **One batch per step** — user jobs enqueued via `perform_later`, preserving all ActiveJob semantics
+3. **Terminal signal via `batch.succeeded?`** — not exception rescue
+4. **`coordination_status` is the sole decision input** — the coordinator reads only this column when making decisions
+5. **`:halted` is policy-driven** — set imperatively via `halt_triggered` flag, not pattern-derived
+6. **Coordinator owns all transitions** — `StepFinishedJob` is a thin dispatcher
+7. **Explicit transaction boundaries** — separate atomic units to minimize lock contention
+8. **DAG validation at instantiation** — before any database writes
+9. **`run` is the only DSL verb** — all topology from `after:`
+10. **`failure_strategy` and `on_failure` are distinct** — strategy vs. callback, no naming collision

data/docs/callbacks.md

@@ -0,0 +1,66 @@
+# Lifecycle Callbacks
+
+GoodPipeline supports three lifecycle callbacks that fire when a pipeline reaches a terminal state.
+
+## Defining callbacks
+
+Register callbacks as class-level methods that name instance methods to invoke:
+
+```ruby
+class VideoProcessingPipeline < GoodPipeline::Pipeline
+  on_complete :notify_complete   # fires on any terminal state
+  on_success  :notify_success    # fires on succeeded
+  on_failure  :notify_failure    # fires on failed or halted
+
+  def configure(video_id:)
+    run :download, DownloadJob, with: { video_id: video_id }
+  end
+
+  private
+
+  def notify_complete
+    Rails.logger.info("Pipeline #{id} finished with status: #{status}")
+  end
+
+  def notify_success
+    Slack.notify("Pipeline #{id} succeeded for video #{params[:video_id]}")
+  end
+
+  def notify_failure
+    Slack.notify("Pipeline #{id} failed for video #{params[:video_id]}")
+  end
+end
+```
+
+## When each callback fires
+
+| Callback | Fires when pipeline status is |
+|---|---|
+| `on_complete` | `succeeded`, `failed`, `halted`, or `skipped` |
+| `on_success` | `succeeded` |
+| `on_failure` | `failed` or `halted` |
+
+Note: `on_failure` does **not** fire for `skipped` pipelines. Being skipped by a chain is not considered a failure — only `on_complete` fires in that case.
+
+## Asynchronous dispatch
+
+Callbacks are dispatched via `PipelineCallbackJob`, a GoodJob job enqueued after the terminal state transaction commits. This means:
+
+- A slow external call (Slack, webhooks) cannot stall the coordinator
+- Callback execution cannot corrupt pipeline state
+- Callbacks benefit from GoodJob's retry mechanism if they fail
+
+## Exactly-once guarantee
+
+The callback bundle (`on_complete` + one of `on_success`/`on_failure`) is dispatched as a **single unit**. A `callbacks_dispatched_at` timestamp is set atomically inside a `FOR UPDATE` locked transaction, ensuring the bundle fires exactly once even if `recompute_pipeline_status` is called from multiple code paths (coordinator or batch reconciliation).
+
+## Callback failure isolation
+
+If a callback method raises an error:
+
+- The `PipelineCallbackJob` fails and is retried by GoodJob
+- Pipeline status and step statuses are **not** affected
+- The pipeline remains in its terminal state
+- Other callback methods in the same bundle are still attempted
+
+Callback delivery failure is an isolated concern that never reopens or alters the terminal pipeline record.

data/docs/cleanup.md

@@ -0,0 +1,45 @@
+# Cleanup
+
+GoodPipeline automatically cleans up old terminal pipelines when GoodJob runs its own cleanup cycle. No configuration is needed beyond what GoodJob already provides.
+
+## Automatic cleanup
+
+GoodPipeline subscribes to GoodJob's `cleanup_preserved_jobs` ActiveSupport notification. When GoodJob cleans its old job records, GoodPipeline deletes terminal pipelines older than the same timestamp.
+
+This means cleanup is:
+
+- **Zero-config** — it uses GoodJob's existing retention period (default 14 days)
+- **Automatic** — it runs whenever GoodJob's cleanup runs
+- **Safe** — only terminal pipelines (`succeeded`, `failed`, `halted`, `skipped`) are deleted; running and pending pipelines are never touched
+
+## What gets cleaned
+
+When a pipeline is cleaned up, the following records are deleted:
+
+1. `good_pipeline_dependencies` — step dependency edges
+2. `good_pipeline_steps` — step records
+3. `good_pipeline_chains` — pipeline chain links
+4. `good_pipeline_pipelines` — pipeline records
+
+Records are deleted in dependency order using `delete_all` (no callbacks) for performance.
+
+## Configuring the retention period
+
+The retention period is controlled by GoodJob's configuration:
+
+```ruby
+# config/application.rb
+config.good_job.cleanup_preserved_jobs_before_seconds_ago = 30.days.to_i
+```
+
+The default is 14 days. GoodPipeline uses the same threshold.
+
+## Manual cleanup
+
+You can trigger cleanup manually at any time:
+
+```ruby
+GoodPipeline.cleanup_preserved_pipelines(older_than: 7.days.ago)
+```
+
+This deletes all terminal pipelines (and their associated steps, dependencies, and chains) created before the given timestamp.

data/docs/dag-validation.md

@@ -0,0 +1,88 @@
+# DAG Validation
+
+GoodPipeline validates the directed acyclic graph at instantiation time — **before** any step records are persisted or any jobs are enqueued. If validation fails, a `GoodPipeline::InvalidPipelineError` is raised with a descriptive message and nothing is written to the database.
+
+## What is validated
+
+### 1. Empty pipelines
+
+A pipeline with no steps is rejected:
+
+```ruby
+class EmptyPipeline < GoodPipeline::Pipeline
+  def configure(id:); end
+end
+
+EmptyPipeline.run(id: 1)
+# => GoodPipeline::InvalidPipelineError: pipeline has no steps
+```
+
+### 2. Duplicate step keys
+
+Each step key must be unique within a pipeline:
+
+```ruby
+run :download, DownloadJob
+run :download, AnotherJob   # same key
+# => GoodPipeline::InvalidPipelineError: duplicate step key :download
+```
+
+### 3. Unknown `after:` references
+
+Every key in `after:` must correspond to a declared step:
+
+```ruby
+run :publish, PublishJob, after: :missing
+# => GoodPipeline::InvalidPipelineError: unknown step key :missing
+```
+
+Forward references are allowed — the full graph is validated after all `run` calls are collected, not incrementally.
+
+### 4. Self-dependencies
+
+A step cannot depend on itself:
+
+```ruby
+run :transcode, TranscodeJob, after: :transcode
+# => GoodPipeline::InvalidPipelineError: step :transcode depends on itself
+```
+
+### 5. Cycles
+
+The graph is checked for cycles using depth-first search. Any cycle causes validation to fail with the cycle path in the error message:
+
+```ruby
+run :a, JobA, after: :b
+run :b, JobB, after: :a
+# => GoodPipeline::InvalidPipelineError: cycle detected: :a -> :b -> :a
+```
+
+Indirect cycles are also detected:
+
+```ruby
+run :a, JobA, after: :c
+run :b, JobB, after: :a
+run :c, JobC, after: :b
+# => GoodPipeline::InvalidPipelineError: cycle detected: :a -> :c -> :b -> :a
+```
+
+## Cycle detection algorithm
+
+GoodPipeline uses a recursive depth-first search with a three-color marking scheme:
+
+- **White** — unvisited
+- **Grey** — currently in the DFS stack (an ancestor of the current node)
+- **Black** — fully processed, confirmed acyclic
+
+If a grey node is encountered during traversal, a cycle is present. The algorithm collects the path for the error message.
+
+## No database writes on failure
+
+Validation runs before any database interaction. If validation fails:
+
+- No `PipelineRecord` is created
+- No `StepRecord` rows are inserted
+- No `DependencyRecord` edges are written
+- No GoodJob batches or jobs are enqueued
+
+This is an intentional design decision: a DAG pipeline gem that allows invalid graphs is not useful.

data/docs/dashboard.md

@@ -0,0 +1,66 @@
+# Web Dashboard
+
+GoodPipeline includes a mountable Rails engine that provides a web dashboard for inspecting pipeline executions. No build step is required — it uses Pico CSS and Mermaid.js from CDN.
+
+## Mounting the engine
+
+```ruby
+# config/routes.rb
+mount GoodPipeline::Engine => "/good_pipeline"
+```
+
+The dashboard is then available at `/good_pipeline`.
+
+## Pages
+
+### Pipeline Executions
+
+The index page lists all pipeline executions with:
+
+- Status filter tabs with per-status counts
+- Pipeline type dropdown filter
+- Execution ID, pipeline name, status badge, start time, and duration
+- Keyset pagination (25 per page)
+
+![Pipeline Executions](/screenshots/index.png)
+
+### Pipeline Details
+
+The show page displays a single pipeline execution with:
+
+- Pipeline metadata: status, failure strategy, params, created time, duration
+- **Mermaid DAG visualization** with color-coded step statuses
+- Steps table with coordination status, job class, duration, error info, and links to the GoodJob dashboard
+- Upstream and downstream chain links (if the pipeline is part of a chain)
+
+![Pipeline Details](/screenshots/show.png)
+
+### Pipeline Definitions
+
+The definitions page catalogs all pipeline types found in the database:
+
+- Sidebar listing all pipeline types with step counts
+- Selected pipeline shows its Mermaid DAG structure, failure strategy, and a link to filter executions by that type
+
+![Pipeline Definitions](/screenshots/definitions.png)
+
+## GoodJob integration
+
+The dashboard links to GoodJob's dashboard for individual step jobs when a `good_job_id` is present on a step. It automatically discovers GoodJob's mount path from your application's routes.
+
+## Securing the dashboard
+
+GoodPipeline's engine is a standard Rails engine mount. Secure it the same way you would any admin interface:
+
+```ruby
+# config/routes.rb
+
+# With Devise
+authenticate :user, ->(user) { user.admin? } do
+  mount GoodPipeline::Engine => "/good_pipeline"
+end
+
+# With Rails routing constraints
+mount GoodPipeline::Engine => "/good_pipeline",
+  constraints: AdminConstraint.new
+```

data/docs/defining-pipelines.md

@@ -0,0 +1,167 @@
+# Defining Pipelines
+
+## Pipeline class structure
+
+Every pipeline is a subclass of `GoodPipeline::Pipeline` that implements `configure`:
+
+```ruby
+class VideoProcessingPipeline < GoodPipeline::Pipeline
+  description "Downloads, transcodes and publishes a video"
+  failure_strategy :halt
+
+  on_complete :notify
+  on_success  :celebrate
+  on_failure  :alert
+
+  def configure(video_id:)
+    run :download,  DownloadJob,  with: { video_id: video_id }
+    run :transcode, TranscodeJob, after: :download
+    run :thumbnail, ThumbnailJob, after: :download
+    run :publish,   PublishJob,   after: %i[transcode thumbnail]
+    run :cleanup,   CleanupJob,   after: :publish
+  end
+
+  private
+
+  def notify = Rails.logger.info("Pipeline complete")
+  def celebrate = Rails.logger.info("All steps succeeded!")
+  def alert = Rails.logger.warn("Pipeline had failures")
+end
+```
+
+**Class-level methods:**
+
+| Method | Purpose | Default |
+|---|---|---|
+| `description` | Human-readable label for the dashboard | `nil` |
+| `failure_strategy` | Pipeline-level failure handling policy | `:halt` |
+| `on_complete` | Callback for any terminal state | `nil` |
+| `on_success` | Callback for succeeded | `nil` |
+| `on_failure` | Callback for failed or halted | `nil` |
+
+## The `run` DSL verb
+
+`run` is the **only** DSL verb. All DAG topology is expressed through `after:` edges:
+
+```ruby
+run :step_key, JobClass,
+  with:       { keyword: args },   # keyword args passed to job's perform method
+  after:      :other_step,         # single dependency (symbol or array of symbols)
+  on_failure: :ignore,             # step-level failure strategy override
+  queue:      :media,              # optional queue override
+  priority:   10                   # optional priority override
+```
+
+### Parameters
+
+| Parameter | Type | Required | Description |
+|---|---|---|---|
+| `key` | Symbol | Yes | Unique step identifier within this pipeline |
+| `job_class` | Class | Yes | The ActiveJob class to execute |
+| `with:` | Hash | No | Keyword arguments forwarded to the job's `perform` method |
+| `after:` | Symbol or Array | No | Step key(s) this step depends on |
+| `on_failure:` | Symbol | No | Override: `:halt`, `:continue`, or `:ignore` |
+| `queue:` | String | No | Queue name for this step's job |
+| `priority:` | Integer | No | Priority for this step's job |
+
+## Step keys vs job classes
+
+The **step key** is the graph identity — it's what you reference in `after:`. The **job class** is the implementation — what code runs:
+
+| Concept | Purpose | Example |
+|---|---|---|
+| Step key | Graph identity — used in `after:` | `:transcode_720p` |
+| Job class | Implementation — what code runs | `TranscodeJob` |
+
+The same job class can appear multiple times in one pipeline under different keys:
+
+```ruby
+run :resize_small, ResizeImageJob, with: { size: "small" }
+run :resize_large, ResizeImageJob, with: { size: "large" }
+run :combine,      CombineJob,     after: [:resize_small, :resize_large]
+```
+
+## DAG topology via `after:`
+
+All topology flows from `after:`. Steps with no `after:` are **root steps** and start immediately. Steps with `after:` wait for all listed dependencies to be satisfied.
+
+### Fan-out
+
+One step feeds into multiple parallel steps:
+
+```ruby
+run :download,  DownloadJob
+run :transcode, TranscodeJob, after: :download
+run :thumbnail, ThumbnailJob, after: :download
+```
+
+```
+   :download
+      ↓
+ ┌────┴────┐
+ ↓         ↓
+:transcode :thumbnail
+```
+
+### Fan-in
+
+Multiple steps converge into one:
+
+```ruby
+run :transcode, TranscodeJob, after: :download
+run :thumbnail, ThumbnailJob, after: :download
+run :publish,   PublishJob,   after: [:transcode, :thumbnail]
+```
+
+`:publish` waits for **both** `:transcode` and `:thumbnail` to succeed.
+
+### Complex DAGs
+
+Combine fan-out and fan-in freely:
+
+```ruby
+def configure(video_id:)
+  run :download,  DownloadJob,  with: { video_id: video_id }
+  run :transcode, TranscodeJob, after: :download
+  run :thumbnail, ThumbnailJob, after: :download
+  run :publish,   PublishJob,   after: [:transcode, :thumbnail]
+  run :cleanup,   CleanupJob,   after: :publish
+end
+```
+
+```
+   :download
+      ↓
+ ┌────┴────┐
+ ↓         ↓
+:transcode :thumbnail
+ └────┬────┘
+      ↓
+  :publish
+      ↓
+  :cleanup
+```
+
+## Running a pipeline
+
+```ruby
+# Fire and forget
+VideoProcessingPipeline.run(video_id: 123)
+
+# Capture the result for monitoring or chaining
+pipeline = VideoProcessingPipeline.run(video_id: 123)
+pipeline.id      # => "uuid-string"
+pipeline.status  # => "running"
+```
+
+`.run` returns a `GoodPipeline::Chain` object that delegates common methods (`id`, `status`, `params`, `steps`, etc.) to the underlying pipeline record.
+
+## Definition freeze semantics
+
+Once a pipeline instance is created and validated:
+
+1. The graph topology is **immutable** — steps and edges are written to the database and cannot be modified
+2. All step definitions are frozen
+3. The params hash is frozen
+
+This ensures the pipeline definition is a snapshot — later changes to the pipeline class don't affect running instances.