good_pipeline 0.1.0

data/docs/failure-strategies.md

@@ -0,0 +1,138 @@
+# Failure Strategies
+
+GoodPipeline provides three failure strategies that control what happens when a step fails. Strategies can be set at the pipeline level and overridden per step.
+
+## Pipeline-level strategy
+
+Set with `failure_strategy` in the pipeline class body:
+
+```ruby
+class MyPipeline < GoodPipeline::Pipeline
+  failure_strategy :continue  # :halt (default), :continue, or :ignore
+end
+```
+
+### `:halt` (default)
+
+When any step fails, the coordinator sets `halt_triggered = true` and marks all remaining `pending` steps as `skipped`. The pipeline derives to `halted`.
+
+```ruby
+class HaltPipeline < GoodPipeline::Pipeline
+  failure_strategy :halt
+
+  def configure(id:)
+    run :a, JobA, with: { id: id }
+    run :b, JobB, with: { id: id }  # independent of :a
+    run :c, JobC, after: :a
+  end
+end
+```
+
+If `:a` fails: `:b` is skipped (even though it's independent), `:c` is skipped, pipeline status is `halted`.
+
+### `:continue`
+
+The coordinator applies skip propagation only to permanently unsatisfied descendants. Independent branches continue executing. The pipeline derives to `failed`.
+
+```ruby
+class ContinuePipeline < GoodPipeline::Pipeline
+  failure_strategy :continue
+
+  def configure(id:)
+    run :a, JobA, with: { id: id }
+    run :b, JobB, with: { id: id }  # independent of :a
+    run :c, JobC, after: :a
+  end
+end
+```
+
+If `:a` fails: `:c` is skipped (depends on `:a`), `:b` still runs, pipeline status is `failed`.
+
+### `:ignore`
+
+Treats all failed steps as satisfied for dependency resolution. Nothing is skipped. The pipeline derives to `failed` if any step actually failed.
+
+```ruby
+class IgnorePipeline < GoodPipeline::Pipeline
+  failure_strategy :ignore
+
+  def configure(id:)
+    run :a, JobA, with: { id: id }
+    run :b, JobB, after: :a
+  end
+end
+```
+
+If `:a` fails: `:b` is still enqueued (failure treated as success for dependencies), pipeline status is `failed`.
+
+## Step-level override
+
+Override the failure strategy for a specific step's **outgoing edges** using `on_failure:` in the `run` call:
+
+```ruby
+run :thumbnail, ThumbnailJob,
+  after:      :download,
+  on_failure: :ignore   # thumbnail failure never blocks downstream steps
+```
+
+Step-level `on_failure` takes precedence over the pipeline-level strategy for that step's outgoing edges only.
+
+## Effective strategy resolution
+
+The coordinator resolves the effective strategy for each step's outgoing edges:
+
+1. If the step has a step-level `on_failure:` override, use that
+2. Otherwise, fall back to the pipeline-level `failure_strategy`
+
+## The `:halt` + step `:ignore` interaction
+
+::: warning Important
+When a step fails with step-level `on_failure: :ignore` under a pipeline-level `:halt` strategy, the behavior may be surprising:
+
+- That step's **outgoing edges** are treated as non-blocking — its dependents remain eligible
+- The pipeline `:halt` policy **still fires** for all other unrelated pending steps
+- `halt_triggered` is still set to `true`
+- The pipeline still derives to `halted`
+:::
+
+Step-level `:ignore` scopes only to that step's outgoing edges, not to the global halt behavior of the pipeline.
+
+```ruby
+class MixedPipeline < GoodPipeline::Pipeline
+  failure_strategy :halt
+
+  def configure(id:)
+    run :optional, OptionalJob, with: { id: id }, on_failure: :ignore
+    run :required, RequiredJob, with: { id: id }
+    run :after_optional, AfterOptionalJob, after: :optional
+  end
+end
+```
+
+If `:optional` fails: `:after_optional` remains eligible (ignore override), `:required` is skipped (halt policy), pipeline is `halted`.
+
+## Dependency satisfaction rules
+
+A dependency edge (upstream → downstream) is **satisfied** when:
+
+| Upstream status | Upstream strategy | Edge satisfied? |
+|---|---|---|
+| `succeeded` | any | Yes |
+| `failed` | `:ignore` | Yes — treated as non-blocking |
+| `failed` | `:continue` or `:halt` | No |
+| `skipped` | any | No |
+| `pending` or `enqueued` | any | No — not yet terminal |
+
+A downstream step is eligible for enqueue when **all** of its incoming edges are satisfied.
+
+A downstream step is marked `skipped` when it's still `pending` and at least one incoming edge is **permanently unsatisfied** — the upstream is terminal, cannot satisfy the edge, and no future event can change that.
+
+## Failure resolution table
+
+| Pipeline strategy | Step override | Effect when step fails |
+|---|---|---|
+| `:halt` | none | `halt_triggered = true`; all pending steps skipped; pipeline → `halted` |
+| `:halt` | `:ignore` on failed step | That step's dependents still eligible; all other pending steps still skipped; pipeline → `halted` |
+| `:continue` | none | Permanently unsatisfied descendants skipped; pipeline → `failed` |
+| `:continue` | `:ignore` on failed step | That step's dependents still eligible |
+| `:ignore` | none | Nothing skipped; pipeline → `failed` if any step failed |

data/docs/getting-started.md

@@ -0,0 +1,77 @@
+# Installation & Setup
+
+## Install the gem
+
+Add GoodPipeline to your Gemfile:
+
+```ruby
+gem "good_pipeline"
+```
+
+Then install:
+
+```bash
+bundle install
+```
+
+## Run the install generator
+
+GoodPipeline provides a generator that creates the necessary database migration:
+
+```bash
+bin/rails generate good_pipeline:install
+bin/rails db:migrate
+```
+
+This creates four tables: `good_pipeline_pipelines`, `good_pipeline_steps`, `good_pipeline_dependencies`, and `good_pipeline_chains`.
+
+## Configure GoodJob
+
+GoodPipeline requires GoodJob to preserve job records so it can read terminal failure metadata:
+
+```ruby
+# config/initializers/good_job.rb
+GoodJob.preserve_job_records = true
+```
+
+GoodPipeline will raise `GoodPipeline::ConfigurationError` at boot if this is not set.
+
+## Mount the dashboard (optional)
+
+```ruby
+# config/routes.rb
+mount GoodPipeline::Engine => "/good_pipeline"
+```
+
+See the [Web Dashboard](/dashboard) page for details.
+
+## Your first pipeline
+
+Define a pipeline by subclassing `GoodPipeline::Pipeline` and implementing `configure`:
+
+```ruby
+class DataIngestionPipeline < GoodPipeline::Pipeline
+  description "Fetches, transforms, and loads data"
+
+  def configure(source_id:)
+    run :fetch,     FetchJob,     with: { source_id: source_id }
+    run :transform, TransformJob, with: { source_id: source_id }, after: :fetch
+    run :load,      LoadJob,      with: { source_id: source_id }, after: :transform
+  end
+end
+```
+
+Run it:
+
+```ruby
+DataIngestionPipeline.run(source_id: 42)
+```
+
+This enqueues `:fetch` immediately. When it succeeds, `:transform` is enqueued. When that succeeds, `:load` is enqueued. If any step fails, the pipeline halts by default.
+
+## Next steps
+
+- [Defining Pipelines](/defining-pipelines) — full DSL reference and DAG patterns
+- [Failure Strategies](/failure-strategies) — control what happens when steps fail
+- [Pipeline Chaining](/pipeline-chaining) — wire pipelines together
+- [Monitoring](/monitoring) — inspect pipeline and step state

data/docs/index.md

@@ -0,0 +1,23 @@
+---
+layout: home
+
+hero:
+  name: GoodPipeline
+  text: DAG-based job pipelines for Rails
+  tagline: Postgres-only workflow orchestration built on GoodJob. Define multi-step workflows as directed acyclic graphs with dependency resolution, parallel execution, failure strategies, and a built-in dashboard.
+  actions:
+    - theme: brand
+      text: Get Started
+      link: /getting-started
+    - theme: alt
+      text: View on GitHub
+      link: https://github.com/milkstrawai/good_pipeline
+
+features:
+  - title: Postgres Only
+    details: All state lives in Postgres — no Redis, no external dependencies. Step transitions and job enqueues are atomically coupled in a single database transaction.
+  - title: DAG Orchestration
+    details: Define pipelines as directed acyclic graphs with the run DSL. Steps run in parallel when possible and wait for dependencies automatically. Fan-out, fan-in, and chaining are all built in.
+  - title: Built-in Dashboard
+    details: A mountable Rails engine with pipeline executions, step details with DAG visualization, and a pipeline definitions catalog. No build step — uses CDN assets.
+---

data/docs/introduction.md

@@ -0,0 +1,42 @@
+# Introduction
+
+GoodPipeline is a Ruby gem that brings DAG-based (Directed Acyclic Graph) workflow orchestration to Rails applications using [GoodJob](https://github.com/bensheldon/good_job) as the job backend. It allows you to define pipelines of jobs that run in parallel or with explicit dependencies, chain multiple pipelines together, and monitor execution — all without any infrastructure beyond Postgres.
+
+## Why GoodPipeline?
+
+### The gap in the ecosystem
+
+The two prominent DAG workflow gems in Ruby are:
+
+- **[Gush](https://github.com/chaps-io/gush)** — graph-based with a clean DSL, but requires **Sidekiq + Redis**
+- **[Jongleur](https://gitlab.com/RedFred7/Jongleur)** — DAG-based, but runs jobs as **OS processes**, not ActiveJob workers
+
+Neither integrates with GoodJob. Teams that have chosen GoodJob for its Postgres-only simplicity have no DAG workflow option that stays within that constraint.
+
+### Why GoodJob::Batch isn't enough
+
+GoodJob's Batch feature fires a single `on_finish` callback when all jobs in a batch complete. This is powerful for fan-out/fan-in patterns but insufficient for DAGs because:
+
+- There is no per-job completion hook
+- There is no concept of edges (dependencies) between individual jobs
+- There is no way to express "enqueue Job C only after Job A and Job B both succeed"
+
+GoodPipeline solves this by building a formal coordination state machine, DAG validation, and atomic coordination layer on top of Batch.
+
+## Key features
+
+- **DAG topology via `run` DSL** — define steps and their dependencies with a single verb
+- **Parallel execution** — steps without dependencies run concurrently
+- **Three failure strategies** — `:halt`, `:continue`, and `:ignore` at pipeline and step level
+- **Pipeline chaining** — serial chains, fan-out, fan-in, and parallel start
+- **Lifecycle callbacks** — `on_complete`, `on_success`, `on_failure` with exactly-once dispatch
+- **Built-in dashboard** — mountable Rails engine with execution list, DAG visualization, and definitions catalog
+- **Automatic cleanup** — piggybacks on GoodJob's cleanup cycle
+- **Postgres-only** — all state in Postgres, no Redis, atomic enqueue transactions
+
+## Requirements
+
+- Ruby >= 3.2
+- Rails >= 7.1
+- PostgreSQL
+- GoodJob >= 3.10 with `preserve_job_records = true`

data/docs/monitoring.md

@@ -0,0 +1,103 @@
+# Monitoring & Introspection
+
+GoodPipeline records are standard ActiveRecord models. You can query and inspect them using familiar Rails patterns.
+
+## Pipeline instance methods
+
+```ruby
+pipeline = VideoProcessingPipeline.run(video_id: 123)
+
+pipeline.id                  # => "uuid-string"
+pipeline.status              # => "running"
+pipeline.type                # => "VideoProcessingPipeline"
+pipeline.params              # => { "video_id" => 123 }
+pipeline.halt_triggered?     # => false
+pipeline.terminal?           # => false
+pipeline.on_failure_strategy # => "halt"
+pipeline.created_at
+pipeline.updated_at
+```
+
+## Step instance methods
+
+```ruby
+step = pipeline.steps.find_by(key: "transcode")
+
+step.key                     # => "transcode"
+step.job_class               # => "TranscodeJob"
+step.coordination_status     # => "succeeded"
+step.params                  # => { "video_id" => 123 }
+step.queue                   # => nil (or custom queue name)
+step.priority                # => nil (or custom priority)
+step.good_job_id             # => "uuid" of the GoodJob record
+step.attempts                # => 3
+step.error_class             # => "TransientError" (on failure)
+step.error_message           # => "Connection timed out" (on failure)
+step.duration                # => 12.34 (Float seconds, from GoodJob record)
+```
+
+## Pipeline statuses
+
+| Status | Meaning |
+|---|---|
+| `pending` | Created but root steps not yet enqueued — waiting in a chain |
+| `running` | At least one step is enqueued or executing |
+| `succeeded` | All steps terminal, none failed |
+| `failed` | One or more steps failed; `:continue` or `:ignore` strategy was used |
+| `halted` | `:halt` strategy was applied — `halt_triggered` is `true` |
+| `skipped` | Skipped because an upstream pipeline in a chain failed |
+
+## Step statuses
+
+The `coordination_status` column is the authoritative step state:
+
+| Status | Meaning |
+|---|---|
+| `pending` | Waiting for upstream dependencies to be satisfied |
+| `enqueued` | Dependencies satisfied; job enqueued |
+| `succeeded` | Job completed successfully — terminal |
+| `failed` | Job exhausted retries or was discarded — terminal |
+| `skipped` | Skipped due to upstream failure propagation — terminal |
+
+## Querying with ActiveRecord
+
+```ruby
+# Find all failed pipelines in the last 24 hours
+GoodPipeline::PipelineRecord.where(status: "failed")
+  .where("created_at > ?", 24.hours.ago)
+
+# Find all pipelines of a specific type
+GoodPipeline::PipelineRecord.where(type: "VideoProcessingPipeline")
+
+# Find pipelines where a specific job class failed
+GoodPipeline::PipelineRecord
+  .joins(:steps)
+  .where(good_pipeline_steps: {
+    job_class: "TranscodeJob",
+    coordination_status: "failed"
+  })
+
+# Running pipelines
+GoodPipeline::PipelineRecord.where(status: "running")
+```
+
+## Step associations
+
+Steps expose their dependency graph through associations:
+
+```ruby
+step = pipeline.steps.find_by(key: "publish")
+
+step.upstream_steps    # => steps that must complete before this one
+step.downstream_steps  # => steps waiting on this one
+```
+
+## Step duration
+
+The `duration` method calculates how long a step took to execute by reading timing data from the associated GoodJob record:
+
+```ruby
+step.duration  # => 12.34 (seconds as Float), or nil if not available
+```
+
+Duration is `nil` if the step hasn't run yet or if the GoodJob record is unavailable.