good_pipeline 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9f30546b22291c0898027eb5a9458771ab943c424c20312f3ea3a03b954e9f30
+  data.tar.gz: 965deb9b03bfca479696dbf1c6baa980e57c332c4f75752912abb5027d31b44a
+SHA512:
+  metadata.gz: 8faa2b2c58f1a6dce5a5b10b3498a24f70d36516b4a5d949afa594297eae4f28b10806f4484dad65645a0c7807f9daea0277d36ff127452ccfd845b610978749
+  data.tar.gz: 250813f051c7a5566666e11f73e260251b10b3a1c6957d7da52c2a4b79faa8d38bbf720741dce429ec09fa31a69cf50d23b5eb87d75d6c80e35bbb75643ae0ea

data/CHANGELOG.md

@@ -0,0 +1,16 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-03-20
+
+### Added
+
+- **Pipeline DSL** — `GoodPipeline::Pipeline` base class with `configure` method and `run` as the sole DSL verb
+- **DAG validation** — cycle detection (three-color DFS), duplicate keys, unknown references, self-dependencies, and empty pipeline checks — all at instantiation time before any database writes
+- **Three failure strategies** — `:halt` (default), `:continue`, and `:ignore` at pipeline level, with per-step `on_failure:` override
+- **Coordinator** — sole owner of all `coordination_status` transitions with explicit transaction boundaries, `FOR UPDATE SKIP LOCKED` row locking, and `good_job_id` null guard to prevent double-enqueue
+- **Atomic enqueue** — step status transition and GoodJob record insertion in a single Postgres transaction
+- **Pipeline chaining** — `.then()` API for serial chains, fan-out, fan-in, and `GoodPipeline.run()` for parallel start
+- **Lifecycle callbacks** — `on_complete`, `on_success`, `on_failure` with asynchronous dispatch via `PipelineCallbackJob` and exactly-once guarantee via `callbacks_dispatched_at` guard
+- **Web dashboard** — mountable Rails engine with pipeline executions list (status tabs, type filter, keyset pagination), pipeline details page (steps table, Mermaid DAG visualization, chain links), and pipeline definitions catalog
+- **Automatic cleanup** — subscribes to GoodJob's `cleanup_preserved_jobs` notification to delete terminal pipelines using the same retention period
+- **Install generator** — `rails generate good_pipeline:install` creates migration for four tables (`good_pipeline_pipelines`, `good_pipeline_steps`, `good_pipeline_dependencies`, `good_pipeline_chains`)

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 MilkStraw AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,217 @@
+# GoodPipeline
+
+DAG-based job pipeline orchestration for Rails, built on [GoodJob](https://github.com/bensheldon/good_job).
+
+Define multi-step workflows as directed acyclic graphs, where each step is a GoodJob job. GoodPipeline handles dependency resolution, parallel execution, failure strategies, pipeline chaining, lifecycle callbacks, and provides a built-in web dashboard.
+
+## Requirements
+
+- Ruby >= 3.2
+- Rails >= 7.1
+- PostgreSQL
+- GoodJob >= 3.10 with `preserve_job_records = true`
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "good_pipeline"
+```
+
+Then install the migrations:
+
+```bash
+bin/rails generate good_pipeline:install
+bin/rails db:migrate
+```
+
+GoodPipeline requires GoodJob to preserve job records. Add this to your GoodJob configuration:
+
+```ruby
+# config/initializers/good_job.rb
+GoodJob.preserve_job_records = true
+```
+
+GoodPipeline will raise `GoodPipeline::ConfigurationError` at boot if this is not set.
+
+## Usage
+
+### Defining a pipeline
+
+Subclass `GoodPipeline::Pipeline` and implement `configure`. Use `run` to declare steps and `after:` to express dependencies:
+
+```ruby
+class VideoProcessingPipeline < GoodPipeline::Pipeline
+  description "Downloads, transcodes and publishes a video"
+  failure_strategy :halt
+
+  on_complete :notify
+  on_success :celebrate
+  on_failure :alert
+
+  def configure(video_id:)
+    run :download,  DownloadJob,  with: { video_id: video_id }
+    run :transcode, TranscodeJob, after: :download
+    run :thumbnail, ThumbnailJob, after: :download
+    run :publish,   PublishJob,   after: %i[transcode thumbnail]
+    run :cleanup,   CleanupJob,   after: :publish
+  end
+
+  private
+
+  def notify = Rails.logger.info("Pipeline complete")
+  def celebrate = Rails.logger.info("All steps succeeded!")
+  def alert = Rails.logger.warn("Pipeline had failures")
+end
+```
+
+This produces the following DAG:
+
+```mermaid
+graph TD
+  download --> transcode
+  download --> thumbnail
+  transcode --> publish
+  thumbnail --> publish
+  publish --> cleanup
+```
+
+### Running a pipeline
+
+```ruby
+VideoProcessingPipeline.run(video_id: 123)
+```
+
+### Step options
+
+```ruby
+run :step_key, JobClass,
+  with:       { key: "value" },  # keyword args passed to the job
+  after:      :other_step,       # dependency (symbol or array of symbols)
+  on_failure: :ignore,           # step-level failure strategy override
+  queue:      :media,            # optional queue override
+  priority:   10                 # optional priority override
+```
+
+### Failure strategies
+
+Set at the pipeline level with `failure_strategy`:
+
+| Strategy | Behaviour |
+|---|---|
+| `:halt` (default) | Stop all pending steps when any step fails |
+| `:continue` | Let independent branches continue; skip only blocked downstream steps |
+| `:ignore` | Treat failures as successes for dependency resolution |
+
+Per-step overrides via `on_failure:` in `run` apply to that step's outgoing edges only.
+
+### Pipeline chaining
+
+Chain pipelines together with `.then()`:
+
+```ruby
+# Serial chain
+VideoProcessingPipeline
+  .run(video_id: 123)
+  .then(NotificationPipeline, with: { video_id: 123 })
+
+# Fan-out
+VideoProcessingPipeline
+  .run(video_id: 123)
+  .then(
+    [NotificationPipeline, with: { video_id: 123 }],
+    [AnalyticsPipeline,    with: { video_id: 123 }]
+  )
+
+# Parallel start with fan-in
+GoodPipeline.run(
+  [VideoProcessingPipeline, with: { video_id: 123 }],
+  [AudioProcessingPipeline, with: { audio_id: 456 }]
+).then(MergeMediaPipeline, with: { video_id: 123, audio_id: 456 })
+```
+
+If an upstream pipeline fails or halts, downstream pipelines are automatically skipped.
+
+### Monitoring
+
+```ruby
+pipeline = VideoProcessingPipeline.run(video_id: 123)
+
+pipeline.status     # => "running"
+pipeline.terminal?  # => false
+pipeline.steps      # => all step records
+pipeline.params     # => { "video_id" => 123 }
+
+# Query across pipelines
+GoodPipeline::PipelineRecord.where(status: "failed")
+GoodPipeline::PipelineRecord.where(type: "VideoProcessingPipeline")
+```
+
+### Lifecycle callbacks
+
+```ruby
+class MyPipeline < GoodPipeline::Pipeline
+  on_complete :always_runs    # any terminal state
+  on_success  :only_success   # pipeline succeeded
+  on_failure  :only_failure   # pipeline failed or halted
+end
+```
+
+Callbacks are dispatched asynchronously via a separate GoodJob job. They never block the coordinator or affect pipeline state.
+
+## Dashboard
+
+GoodPipeline includes a mountable web dashboard for inspecting pipeline executions:
+
+```ruby
+# config/routes.rb
+mount GoodPipeline::Engine => "/good_pipeline"
+```
+
+The dashboard provides:
+
+- Pipeline Executions: filterable list with status tabs and pipeline type dropdown
+- Pipeline Details: steps table, DAG visualization, chain links, error info
+- Pipeline Definitions: catalog of all pipeline types with their DAG structure
+
+### Pipeline Executions
+
+![Pipeline Executions](docs/screenshots/index.png)
+
+### Pipeline Details
+
+![Pipeline Details](docs/screenshots/show.png)
+
+### Pipeline Definitions
+
+![Pipeline Definitions](docs/screenshots/definitions.png)
+
+No build step required. Uses Pico CSS and Mermaid.js from CDN.
+
+## Cleanup
+
+GoodPipeline automatically cleans up old terminal pipelines when GoodJob runs its own cleanup cycle. No configuration needed, it uses GoodJob's retention period (default 14 days).
+
+To configure the retention period, set GoodJob's option:
+
+```ruby
+# config/application.rb
+config.good_job.cleanup_preserved_jobs_before_seconds_ago = 30.days.to_i
+```
+
+## Development
+
+```bash
+bin/setup
+mise docker:start  # PostgreSQL
+rake test
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/milkstrawai/good_pipeline.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+require "rubocop/rake_task"
+
+Minitest::TestTask.create(:test_unit) do |task|
+  task.test_globs = ["test/**/*test*.rb"]
+end
+
+desc "Run integration tests"
+task :test_integration do
+  sh "ruby -Idemo/test -e 'Dir[\"demo/test/**/*test*.rb\"].sort.each { |file| require File.expand_path(file) }'"
+end
+
+task test: %i[test_unit test_integration]
+
+RuboCop::RakeTask.new
+
+task default: %i[test rubocop]

data/app/controllers/good_pipeline/application_controller.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module GoodPipeline
+  class ApplicationController < ActionController::Base
+    protect_from_forgery with: :exception
+
+    layout "good_pipeline/application"
+  end
+end

data/app/controllers/good_pipeline/frontends_controller.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module GoodPipeline
+  class FrontendsController < ApplicationController
+    def static
+      file_name = "#{sanitized_id}.#{sanitized_format}"
+      file_path = assets_directory.join(file_name)
+
+      if file_path.exist?
+        expires_in 1.year, public: true
+        send_file file_path, disposition: :inline
+      else
+        head :not_found
+      end
+    end
+
+    private
+
+    def sanitized_id
+      params[:id].to_s.gsub(/[^a-zA-Z0-9_-]/, "")
+    end
+
+    def sanitized_format
+      params[:format].to_s.gsub(/[^a-zA-Z0-9]/, "")
+    end
+
+    def assets_directory
+      GoodPipeline::Engine.root.join("app", "frontend", "good_pipeline")
+    end
+  end
+end

data/app/controllers/good_pipeline/pipelines_controller.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+module GoodPipeline
+  class PipelinesController < ApplicationController
+    PAGE_SIZE = 25
+
+    def index
+      @status = params[:status].presence
+      @pipeline_type = params[:pipeline_type].presence
+      @pipeline_types = PipelineRecord.distinct.pluck(:type).sort
+      counts_scope = @pipeline_type ? PipelineRecord.where(type: @pipeline_type) : PipelineRecord
+      @status_counts = counts_scope.group(:status).count
+      @total_count = @status_counts.values.sum
+      load_pipelines
+    end
+
+    def definitions
+      pipeline_types = PipelineRecord.distinct.pluck(:type).sort
+      pipeline_ids = pipeline_types.filter_map do |type|
+        PipelineRecord.where(type: type).order(created_at: :desc).pick(:id)
+      end
+      @pipelines = PipelineRecord.includes(:steps, dependencies: %i[step depends_on_step])
+                                 .where(id: pipeline_ids)
+                                 .sort_by(&:type)
+    end
+
+    def show
+      scope = PipelineRecord.includes(
+        :upstream_pipelines, :downstream_pipelines, :steps,
+        dependencies: %i[step depends_on_step]
+      )
+      @pipeline = scope.find(params[:id])
+    end
+
+    private
+
+    def load_pipelines
+      scope = PipelineRecord.order(created_at: :desc, id: :desc)
+      scope = scope.where(status: @status) if @status
+      scope = scope.where(type: @pipeline_type) if @pipeline_type
+      scope = apply_keyset_pagination(scope)
+      records = scope.limit(PAGE_SIZE + 1).to_a
+      @has_next_page = records.size > PAGE_SIZE
+      @pipelines = records.first(PAGE_SIZE)
+    end
+
+    def apply_keyset_pagination(scope)
+      return scope unless params[:after_created_at].present? && params[:after_id].present?
+
+      scope.where(
+        "(created_at, id) < (?, ?)",
+        params[:after_created_at],
+        params[:after_id]
+      )
+    end
+  end
+end