good_pipeline 0.2.2 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e52af813bd86f0de3d905a3f591209e0a155e0b5c28e3a76f963f1c2cdbd8232
-  data.tar.gz: 3a48213db593949e7f9afff0f341cf29dfdb90d4972b20b4c5e48091cb86bab1
+  metadata.gz: e9df8b5fbd57895f53adf1d3c5804ca9bd64ca792e880c2d4941957e4b8ca368
+  data.tar.gz: c4e2a7c4edbe27a0e40e7ff62a061f5ade53c44a74c0b824ce21bdf019cb8781
 SHA512:
-  metadata.gz: 7bc3d7e1181b852bbc795f1e65ef9b2764645ffc13d7d860f444074158c741202ddae188253c50cb12a44da9958fe6403b2435fef6ec71d9172a35b46816da55
-  data.tar.gz: 22b59ba7dba841a6d55559c27a7427032bad94ed1fd3e336f2d53997ee4a24bf2f8c193d77e743095e4c5c73b948975b0f85228d63839eba367af1dc441f8456
+  metadata.gz: e3c6e6940034efbc8679ede33b352022deb64a36b7a16a9dd3f02fd6dc32a837caf545407541bd0ca629f690d38cfa60342e6aa756e53ea321fb754eebbd772c
+  data.tar.gz: 878566f6d9ddc7b9ce6b4201f5bd241b21f2758f5475e3fa60e6c3fd629651bff06ff973af9f3f29ac4624bee58e78dc0f9628e8a1cdb503836e923a09e3944e

data/CHANGELOG.md

@@ -1,5 +1,43 @@
 ## [Unreleased]
 
+## [0.3.1] - 2026-03-26
+
+### Added
+
+- **`halt_pipeline!`** — call from any job to stop the pipeline early with a `succeeded` status. The halting step is marked `halted`, remaining pending steps are `skipped`, and the `on_success` callback fires. The GoodJob record completes as succeeded (no error, no discard). Available in all jobs via `GoodPipeline::Haltable`, included automatically by the Engine.
+- **`halted` coordination status** — new terminal step status for steps that called `halt_pipeline!`. Treated as satisfied for downstream dependency resolution.
+- **`halt_requested` column** — boolean column on steps table, set by `halt_pipeline!` and checked by the coordinator on step completion.
+- **`good_job_id` index** — partial unique index on `good_job_id` for fast step lookup from within jobs.
+
+## [0.3.0] - 2026-03-25
+
+### Performance
+
+- **Bulk insert steps and dependencies** — `Runner` uses `insert_all!` with `RETURNING` for steps and dependencies instead of individual `create!` calls, reducing pipeline creation from N+M queries to 2.
+- **Pre-generated pipeline UUID** — `Runner` generates the pipeline UUID upfront, folding batch ID and initial status into a single INSERT instead of separate UPDATEs.
+- **Atomic upstream counter** — new `pending_upstream_count` column on steps tracks how many upstreams remain. `unblock_downstream_steps` atomically decrements via `UPDATE ... RETURNING` and only calls `try_enqueue_step` when the count reaches zero, eliminating O(N) wasted lock acquisitions for fan-in and diamond topologies.
+- **Merged UPDATE round-trips** — `enqueue_user_job` folds status transition, batch ID, and job ID into one `update_columns`. `record_step_failure` merges status and error metadata into one `update_columns`.
+- **Removed redundant transaction** — `record_step_outcome` no longer wraps a single `update_columns` in an explicit transaction.
+- **`update_columns` in transition methods** — `transition_coordination_status_to!` and `transition_to!` use `update_columns` instead of `update!`, skipping AR dirty tracking overhead.
+- **SQL EXISTS for status checks** — `recompute_pipeline_status` and `derive_terminal_status` use `EXISTS` queries instead of loading all step records.
+- **Pipeline load with EXISTS** — `load_pipeline_with_active_check` combines pipeline load with active-step and downstream-chain EXISTS checks in a single query.
+- **Conditional callback dispatch** — `dispatch_callbacks_once` uses `UPDATE WHERE callbacks_dispatched_at IS NULL` instead of `SELECT FOR UPDATE` + `UPDATE`.
+- **Early return on active pipeline** — `complete_step` skips pipeline status recomputation when `unblock_downstream_steps` enqueued any downstream step.
+- **Bulk skip on halt** — `skip_all_pending_steps` uses `update_all` instead of iterating with individual updates.
+- **Single-pass graph validation** — `GraphValidator` merges duplicate-key check, self-dependency check, steps-by-key index, and forward-edges construction into one O(n) pass and returns `steps_by_key` for reuse by `Pipeline`.
+- **Frozen constant defaults** — `EMPTY_HASH` and `EMPTY_ARRAY` shared constants avoid allocating fresh empty containers on every `StepDefinition` and `Pipeline#run` call.
+- **Fast-path shortcuts** — `validate_enqueue_options!` returns immediately for empty options. `expand_branch_aliases` skips `flat_map` when no branches are defined.
+
+### Added
+
+- **Benchmarking scripts** — `bench/memory_bench.rb` (in-memory, no DB) and `bench/database_bench.rb` (PostgreSQL) with `--json` flag for structured output. Covers pipeline construction, graph validation, cycle detection, step enqueue, step completion, status recomputation, halt propagation, and full pipeline run across linear, fan-out, fan-in, and diamond topologies.
+- **`pending_upstream_count` column** — integer column on steps table, set by `Runner` at creation time, decremented atomically by `Coordinator` on step completion.
+
+### Changed
+
+- **`Runner` refactored** — `call` method extracted into `create_pipeline_batch`, `create_pipeline_record`, `insert_steps`, `insert_dependencies`, and `enqueue_root_steps` for readability. Pipeline record is a local variable passed to methods instead of an instance variable.
+- **`Coordinator` method reordering** — private methods grouped by concern (outcome recording, downstream unblocking, step resolution, pipeline status) rather than call order.
+
 ## [0.2.2] - 2026-03-24
 
 ### Fixed

data/app/helpers/good_pipeline/mermaid_diagram_builder.rb

@@ -15,6 +15,7 @@ module GoodPipeline
       "  classDef failed fill:#f44336,color:#fff",
       "  classDef skipped fill:#bdbdbd,color:#333",
       "  classDef skipped_by_branch fill:#bdbdbd,color:#333",
+      "  classDef halted fill:#8bc34a,color:#fff",
       "  classDef branch fill:#ff9800,color:#fff,stroke:#f57c00",
       "  classDef terminal fill:#1a1a2e,color:#fff,stroke:#1a1a2e"
     ].freeze

data/app/models/good_pipeline/pipeline_record.rb

@@ -1,6 +1,9 @@
 # frozen_string_literal: true
 
 module GoodPipeline
+  # This model intentionally has no AR callbacks or validations. Status transitions
+  # use update_columns throughout the coordinator layer. If you need lifecycle hooks,
+  # ensure all update_columns call sites are updated accordingly.
   class PipelineRecord < ActiveRecord::Base
     self.table_name = "good_pipeline_pipelines"
     self.inheritance_column = nil
@@ -67,7 +70,7 @@ module GoodPipeline
         raise InvalidTransition, "cannot transition pipeline from '#{status}' to '#{new_status}'"
       end
 
-      update!(status: new_status)
+      update_columns(status: new_status, updated_at: Time.current)
     end
   end
 end

data/app/models/good_pipeline/step_record.rb

@@ -1,14 +1,17 @@
 # frozen_string_literal: true
 
 module GoodPipeline
+  # This model intentionally has no AR callbacks or validations. Status transitions
+  # use update_columns throughout the coordinator layer. If you need lifecycle hooks,
+  # ensure all update_columns call sites are updated accordingly.
   class StepRecord < ActiveRecord::Base
     self.table_name = "good_pipeline_steps"
 
-    TERMINAL_COORDINATION_STATUSES = %w[succeeded failed skipped skipped_by_branch].freeze
+    TERMINAL_COORDINATION_STATUSES = %w[succeeded failed skipped skipped_by_branch halted].freeze
 
     VALID_COORDINATION_TRANSITIONS = {
-      "pending" => %w[enqueued skipped skipped_by_branch succeeded failed],
-      "enqueued" => %w[succeeded failed]
+      "pending" => %w[enqueued skipped skipped_by_branch succeeded failed halted],
+      "enqueued" => %w[succeeded failed halted]
     }.freeze
 
     enum :coordination_status, {
@@ -17,7 +20,8 @@ module GoodPipeline
       succeeded: "succeeded",
       failed: "failed",
       skipped: "skipped",
-      skipped_by_branch: "skipped_by_branch"
+      skipped_by_branch: "skipped_by_branch",
+      halted: "halted"
     }
 
     enum :on_failure_strategy, { halt: "halt", continue: "continue", ignore: "ignore" }
@@ -74,7 +78,7 @@ module GoodPipeline
               "cannot transition step '#{key}' coordination_status from '#{coordination_status}' to '#{new_status}'"
       end
 
-      update!(coordination_status: new_status)
+      update_columns(coordination_status: new_status, updated_at: Time.current)
     end
   end
 end

data/demo/app/jobs/halt_execution_job.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+class HaltExecutionJob < ApplicationJob
+  def perform(**)
+    halt_pipeline!
+  end
+end

data/demo/db/migrate/20260319205326_create_good_pipeline_tables.rb

@@ -30,6 +30,8 @@ class CreateGoodPipelineTables < ActiveRecord::Migration[8.1]
       t.jsonb :branch, null: false, default: {}
       t.uuid :good_job_batch_id
       t.uuid :good_job_id
+      t.integer :pending_upstream_count, null: false, default: 0
+      t.boolean :halt_requested, null: false, default: false
       t.integer :attempts
       t.string :error_class
       t.text :error_message
@@ -39,6 +41,7 @@ class CreateGoodPipelineTables < ActiveRecord::Migration[8.1]
 
     add_index :good_pipeline_steps, %i[pipeline_id key], unique: true
     add_index :good_pipeline_steps, :coordination_status
+    add_index :good_pipeline_steps, :good_job_id, unique: true, where: "good_job_id IS NOT NULL"
 
     create_table :good_pipeline_dependencies do |t|
       t.references :pipeline, null: false, foreign_key: { to_table: :good_pipeline_pipelines }, type: :uuid

data/demo/test/integration/test_halt_execution.rb

@@ -0,0 +1,113 @@
+# frozen_string_literal: true
+
+require "test_helper"
+
+class TestHaltExecution < ActiveSupport::TestCase
+  def run_pipeline_to_completion(pipeline_record, timeout: 15)
+    deadline = Time.current + timeout
+    loop do
+      perform_enqueued_jobs_inline
+      pipeline_record.reload
+      return pipeline_record if pipeline_record.terminal?
+
+      raise "Pipeline did not reach terminal state within #{timeout}s (status: #{pipeline_record.status})" if Time.current > deadline
+
+      sleep 0.05
+    end
+  end
+
+  def test_halt_pipeline_marks_step_halted
+    pipeline_class = Class.new(GoodPipeline::Pipeline) do
+      failure_strategy :halt
+      define_method(:configure) do |**|
+        run :halt_step, HaltExecutionJob
+        run :after_step, DownloadJob, after: :halt_step
+      end
+    end
+    Object.const_set(:HaltSucceededPipeline, pipeline_class) unless defined?(::HaltSucceededPipeline)
+
+    chain = HaltSucceededPipeline.run
+    result = run_pipeline_to_completion(chain)
+
+    halt_step = result.steps.find_by(key: "halt_step")
+    assert_equal "halted", halt_step.coordination_status
+    assert halt_step.halt_requested?, "halt_requested should be true"
+  end
+
+  def test_halt_pipeline_skips_remaining_steps
+    pipeline_class = Class.new(GoodPipeline::Pipeline) do
+      failure_strategy :halt
+      define_method(:configure) do |**|
+        run :halt_step, HaltExecutionJob
+        run :after_step, DownloadJob, after: :halt_step
+      end
+    end
+    Object.const_set(:HaltSkipsPipeline, pipeline_class) unless defined?(::HaltSkipsPipeline)
+
+    chain = HaltSkipsPipeline.run
+    result = run_pipeline_to_completion(chain)
+
+    after_step = result.steps.find_by(key: "after_step")
+    assert_equal "skipped", after_step.coordination_status
+  end
+
+  def test_halt_pipeline_pipeline_succeeds
+    pipeline_class = Class.new(GoodPipeline::Pipeline) do
+      failure_strategy :halt
+      define_method(:configure) do |**|
+        run :halt_step, HaltExecutionJob
+        run :after_step, DownloadJob, after: :halt_step
+      end
+    end
+    Object.const_set(:HaltSucceedsPipeline, pipeline_class) unless defined?(::HaltSucceedsPipeline)
+
+    chain = HaltSucceedsPipeline.run
+    result = run_pipeline_to_completion(chain)
+
+    assert_equal "succeeded", result.status
+  end
+
+  def test_halt_pipeline_job_succeeds_in_good_job
+    pipeline_class = Class.new(GoodPipeline::Pipeline) do
+      failure_strategy :halt
+      define_method(:configure) do |**|
+        run :halt_step, HaltExecutionJob
+      end
+    end
+    Object.const_set(:HaltJobSucceedsPipeline, pipeline_class) unless defined?(::HaltJobSucceedsPipeline)
+
+    chain = HaltJobSucceedsPipeline.run
+    run_pipeline_to_completion(chain)
+
+    halt_step = chain.steps.find_by(key: "halt_step")
+    good_job = GoodJob::Job.find(halt_step.good_job_id)
+
+    assert_equal 1, good_job.executions_count
+    assert_nil good_job.error, "GoodJob record should have no error"
+    assert_not_nil good_job.finished_at
+  end
+
+  def test_halt_pipeline_with_parallel_steps
+    pipeline_class = Class.new(GoodPipeline::Pipeline) do
+      failure_strategy :continue
+      define_method(:configure) do |**|
+        run :halt_step, HaltExecutionJob
+        run :normal_step, DownloadJob
+        run :after_both, CleanupJob, after: %i[halt_step normal_step]
+      end
+    end
+    Object.const_set(:HaltParallelPipeline, pipeline_class) unless defined?(::HaltParallelPipeline)
+
+    chain = HaltParallelPipeline.run
+    result = run_pipeline_to_completion(chain)
+
+    halt_step = result.steps.find_by(key: "halt_step")
+    normal_step = result.steps.find_by(key: "normal_step")
+    after_both = result.steps.find_by(key: "after_both")
+
+    assert_equal "halted", halt_step.coordination_status
+    assert_equal "succeeded", normal_step.coordination_status
+    assert_equal "skipped", after_both.coordination_status
+    assert_equal "succeeded", result.status
+  end
+end

data/demo/test/test_helper.rb

@@ -65,6 +65,7 @@ module ActiveSupport
       dependencies.each do |dependency_step|
         GoodPipeline::DependencyRecord.create!(pipeline: pipeline, step: step, depends_on_step: dependency_step)
       end
+      step.update_column(:pending_upstream_count, dependencies.size)
       step
     end
   end

data/docs/failure-strategies.md

@@ -127,6 +127,41 @@ A downstream step is eligible for enqueue when **all** of its incoming edges are
 
 A downstream step is marked `skipped` when it's still `pending` and at least one incoming edge is **permanently unsatisfied** — the upstream is terminal, cannot satisfy the edge, and no future event can change that.
 
+## Early termination with success
+
+Sometimes a job determines there is nothing to do — the account is deactivated, the resource was already processed, etc. Call `halt_pipeline!` to stop the pipeline early and mark it as `succeeded`:
+
+```ruby
+class FetchDataJob < ApplicationJob
+  def perform(account_id:)
+    account = Account.find(account_id)
+    return halt_pipeline! if account.deactivated?
+
+    # ... normal work
+  end
+end
+```
+
+The behavior:
+
+| Aspect | Value |
+|---|---|
+| Halting step status | `halted` |
+| Remaining pending steps | `skipped` |
+| Pipeline status | `succeeded` |
+| Callback triggered | `on_success` |
+| GoodJob record | Succeeded (no error, no discard) |
+
+No configuration or module includes are required. The Engine includes `GoodPipeline::Haltable` into `ActiveJob::Base` at boot, so `halt_pipeline!` is available in any job. For non-pipeline jobs, it's a no-op.
+
+::: tip Return early
+Remember to use `return halt_pipeline!` — without `return`, the job continues executing after the call.
+:::
+
+::: warning Parallel steps
+If another step is already running when `halt_pipeline!` is called, that step continues to completion. Only `pending` steps are skipped. If the running step fails, the pipeline will derive to `failed`, not `succeeded`.
+:::
+
 ## Failure resolution table
 
 | Pipeline strategy | Step override | Effect when step fails |

data/lib/generators/good_pipeline/install/templates/create_good_pipeline_tables.rb.erb

@@ -28,6 +28,8 @@ class CreateGoodPipelineTables < ActiveRecord::Migration[<%= ActiveRecord::Migra
       t.jsonb :branch, null: false, default: {}
       t.uuid :good_job_batch_id
       t.uuid :good_job_id
+      t.integer :pending_upstream_count, null: false, default: 0
+      t.boolean :halt_requested, null: false, default: false
       t.integer :attempts
       t.string :error_class
       t.text :error_message
@@ -37,6 +39,7 @@ class CreateGoodPipelineTables < ActiveRecord::Migration[<%= ActiveRecord::Migra
 
     add_index :good_pipeline_steps, %i[pipeline_id key], unique: true
     add_index :good_pipeline_steps, :coordination_status
+    add_index :good_pipeline_steps, :good_job_id, unique: true, where: "good_job_id IS NOT NULL"
 
     create_table :good_pipeline_dependencies do |t|
       t.references :pipeline, null: false, foreign_key: { to_table: :good_pipeline_pipelines }, type: :uuid

data/lib/good_pipeline/constants.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module GoodPipeline
+  EMPTY_HASH  = {}.freeze
+  EMPTY_ARRAY = [].freeze
+end

data/lib/good_pipeline/coordinator.rb

@@ -3,25 +3,39 @@
 module GoodPipeline
   class Coordinator # rubocop:disable Metrics/ClassLength
     class << self
-      def complete_step(step, succeeded:)
+      def complete_step(step, succeeded:) # rubocop:disable Metrics/MethodLength
         return if step.terminal_coordination_status?
 
+        if succeeded && step.halt_requested?
+          handle_halt_execution(step)
+          return
+        end
+
         record_step_outcome(step, succeeded)
         propagate_halt(step) if !succeeded && step.pipeline.halt?
-        unblock_downstream_steps(step)
-        recompute_pipeline_status(step.pipeline.reload)
+        return if unblock_downstream_steps(step)
+
+        pipeline = load_pipeline_with_active_check(step.pipeline_id)
+
+        recompute_pipeline_status(
+          pipeline,
+          has_active_steps: pipeline["has_active_steps"],
+          has_downstream_chains: pipeline["has_downstream_chains"]
+        )
       end
 
       def try_enqueue_step(step_id) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
+        step_was_enqueued = false
         skipped_downstream_ids = nil
         recompute_pipeline = nil
 
         StepRecord.transaction do
           locked_step = StepRecord.lock("FOR UPDATE").find_by(id: step_id)
-          return unless locked_step&.pending?
-          return if locked_step.good_job_id.present?
+          return false unless locked_step&.pending?
+          return false if locked_step.good_job_id.present?
 
           skipped_downstream_ids = resolve_step(locked_step)
+          step_was_enqueued = skipped_downstream_ids.nil?
         rescue ConfigurationError => error
           fail_step_with_error(locked_step, error)
           propagate_halt(locked_step) if locked_step.pipeline.halt?
@@ -29,48 +43,68 @@ module GoodPipeline
           recompute_pipeline = locked_step.pipeline
         end
 
-        skipped_downstream_ids&.each { |downstream_step_id| try_enqueue_step(downstream_step_id) }
+        downstream_enqueued = false
+        skipped_downstream_ids&.each { |id| downstream_enqueued = true if try_enqueue_step(id) }
         recompute_pipeline_status(recompute_pipeline.reload) if recompute_pipeline
+        step_was_enqueued || downstream_enqueued
       end
 
-      def recompute_pipeline_status(pipeline)
-        steps = pipeline.steps.reload
-
-        return if steps.any? { |step| step.pending? || step.enqueued? }
+      def recompute_pipeline_status(pipeline, has_active_steps: nil, has_downstream_chains: nil) # rubocop:disable Metrics/MethodLength
         return if pipeline.terminal?
 
-        new_status = derive_terminal_status(steps, pipeline)
+        active = if has_active_steps.nil?
+                   pipeline.steps.where(coordination_status: %w[pending enqueued]).exists?
+                 else
+                   has_active_steps
+                 end
+
+        return if active
+
+        new_status = derive_terminal_status(pipeline)
         pipeline.transition_to!(new_status)
         dispatch_callbacks_once(pipeline, new_status)
-        ChainCoordinator.propagate_terminal_state(pipeline)
+        ChainCoordinator.propagate_terminal_state(pipeline) unless has_downstream_chains == false
       end
 
       def dispatch_callbacks_once(pipeline, new_status)
         PipelineRecord.transaction do
-          locked = PipelineRecord.lock("FOR UPDATE").find(pipeline.id)
-          return if locked.callbacks_dispatched_at.present?
+          rows_updated = PipelineRecord.where(id: pipeline.id, callbacks_dispatched_at: nil)
+                                       .update_all(callbacks_dispatched_at: Time.current)
+
+          return if rows_updated.zero?
 
-          locked.update!(callbacks_dispatched_at: Time.current)
-          PipelineCallbackJob.perform_later(locked.id, new_status.to_s)
+          PipelineCallbackJob.perform_later(pipeline.id, new_status.to_s)
         end
       end
 
       private
 
+      def handle_halt_execution(step)
+        step.transition_coordination_status_to!(:halted)
+        step.pipeline.steps.pending.update_all(coordination_status: "skipped")
+
+        pipeline = load_pipeline_with_active_check(step.pipeline_id)
+
+        recompute_pipeline_status(
+          pipeline,
+          has_active_steps: pipeline["has_active_steps"],
+          has_downstream_chains: pipeline["has_downstream_chains"]
+        )
+      end
+
       def record_step_outcome(step, succeeded)
-        StepRecord.transaction do
-          if succeeded
-            step.transition_coordination_status_to!(:succeeded)
-          else
-            record_step_failure(step)
-          end
+        if succeeded
+          step.transition_coordination_status_to!(:succeeded)
+        else
+          record_step_failure(step)
         end
       end
 
       def record_step_failure(step)
         metadata = FailureMetadata.extract(step)
-        step.transition_coordination_status_to!(:failed)
         step.update_columns(
+          coordination_status: "failed",
+          updated_at: Time.current,
           error_class: metadata.error_class,
           error_message: metadata.error_message,
           attempts: metadata.attempts
@@ -78,28 +112,71 @@ module GoodPipeline
       end
 
       def propagate_halt(step)
-        pipeline = step.pipeline
         StepRecord.transaction do
-          pipeline.update_column(:halt_triggered, true)
-          skip_all_pending_steps(pipeline, except_dependents_of: step)
+          step.pipeline.update_column(:halt_triggered, true)
+          skip_all_pending_steps(step.pipeline, except_dependents_of: step)
+        end
+      end
+
+      def skip_all_pending_steps(pipeline, except_dependents_of:)
+        scope = pipeline.steps.pending
+
+        if effective_failure_strategy(except_dependents_of) == :ignore
+          exempt_ids = transitive_downstream_ids(except_dependents_of)
+          scope = scope.where.not(id: exempt_ids.to_a) if exempt_ids.any?
         end
+
+        scope.update_all(coordination_status: "skipped")
       end
 
       def unblock_downstream_steps(step)
-        step.downstream_steps.each do |downstream_step|
-          try_enqueue_step(downstream_step.id)
+        sql = <<~SQL
+          UPDATE good_pipeline_steps
+             SET pending_upstream_count = pending_upstream_count - 1
+          WHERE id IN (
+            SELECT step_id FROM good_pipeline_dependencies
+             WHERE depends_on_step_id = $1
+          )
+            AND coordination_status = 'pending'
+          RETURNING id, pending_upstream_count
+        SQL
+
+        any_enqueued = false
+        StepRecord.connection.exec_query(sql, "SQL", [step.id]).each do |row|
+          any_enqueued = true if row["pending_upstream_count"].zero? && try_enqueue_step(row["id"])
         end
+        any_enqueued
       end
 
-      def resolve_step(locked_step) # rubocop:disable Metrics/MethodLength
+      def load_pipeline_with_active_check(pipeline_id)
+        sql = <<~SQL.squish
+          good_pipeline_pipelines.*,
+          EXISTS(
+            SELECT 1 FROM good_pipeline_steps
+             WHERE pipeline_id = good_pipeline_pipelines.id
+               AND coordination_status IN ('pending', 'enqueued')
+          ) AS has_active_steps,
+          EXISTS(
+            SELECT 1 FROM good_pipeline_chains
+             WHERE upstream_pipeline_id = good_pipeline_pipelines.id
+          ) AS has_downstream_chains
+        SQL
+
+        PipelineRecord.select(sql).where(id: pipeline_id).first!
+      end
+
+      def resolve_step(locked_step) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
         if should_skip?(locked_step)
           locked_step.transition_coordination_status_to!(:skipped)
+          decrement_upstream_counts_for_terminal_step(locked_step.id)
           locked_step.downstream_steps.pluck(:id)
         elsif locked_step.branch_step? && all_upstreams_satisfied?(locked_step)
           BranchResolver.resolve(locked_step)
+          decrement_upstream_counts_for_terminal_step(locked_step.id)
           locked_step.downstream_steps.pluck(:id)
         elsif BranchResolver.skipped_by_branch?(locked_step)
           locked_step.transition_coordination_status_to!(:skipped_by_branch)
+          decrement_upstream_counts_for_terminal_step(locked_step.id)
           locked_step.downstream_steps.pluck(:id)
         else
           enqueue_user_job(locked_step) if all_upstreams_satisfied?(locked_step)
@@ -107,12 +184,43 @@ module GoodPipeline
         end
       end
 
-      def enqueue_user_job(step)
-        step.transition_coordination_status_to!(:enqueued)
+      def should_skip?(step)
+        step.pending? && step.upstream_steps.any? { |upstream| permanently_unsatisfied?(upstream) }
+      end
+
+      def permanently_unsatisfied?(upstream)
+        upstream.terminal_coordination_status? &&
+          !upstream.succeeded? &&
+          !upstream.halted? &&
+          !upstream.skipped_by_branch? &&
+          effective_failure_strategy(upstream) != :ignore
+      end
+
+      def decrement_upstream_counts_for_terminal_step(step_id)
+        downstream_ids = DependencyRecord.where(depends_on_step_id: step_id).select(:step_id)
+        StepRecord.where(id: downstream_ids, coordination_status: "pending")
+                  .update_all("pending_upstream_count = pending_upstream_count - 1")
+      end
+
+      def all_upstreams_satisfied?(step)
+        step.upstream_steps.all? do |upstream|
+          upstream.succeeded? ||
+            upstream.halted? ||
+            upstream.skipped_by_branch? ||
+            (upstream.failed? && effective_failure_strategy(upstream) == :ignore)
+        end
+      end
 
+      def enqueue_user_job(step)
         batch = build_step_batch(step)
-        batch.enqueue { enqueue_step_job(step) }
-        step.update_column(:good_job_batch_id, batch.id)
+        good_job_id = nil
+        batch.enqueue { good_job_id = enqueue_step_job(step) }
+        step.update_columns(
+          coordination_status: "enqueued",
+          good_job_batch_id: batch.id,
+          good_job_id: good_job_id,
+          updated_at: Time.current
+        )
       end
 
       def build_step_batch(step)
@@ -125,11 +233,11 @@ module GoodPipeline
       def enqueue_step_job(step)
         job = step.job_class.constantize.new(**step.params.symbolize_keys)
         enqueued_job = job.enqueue(**step.enqueue_options.symbolize_keys)
-        step.update_column(:good_job_id, enqueued_job.provider_job_id || enqueued_job.job_id)
+        enqueued_job.provider_job_id || enqueued_job.job_id
       end
 
-      def derive_terminal_status(steps, pipeline)
-        has_failures = steps.any?(&:failed?)
+      def derive_terminal_status(pipeline)
+        has_failures = pipeline.steps.where(coordination_status: "failed").exists?
 
         return :succeeded unless has_failures
         return :halted if pipeline.halt_triggered?
@@ -137,40 +245,6 @@ module GoodPipeline
         :failed
       end
 
-      def all_upstreams_satisfied?(step)
-        step.upstream_steps.all? do |upstream|
-          upstream.succeeded? ||
-            upstream.skipped_by_branch? ||
-            (upstream.failed? && effective_failure_strategy(upstream) == :ignore)
-        end
-      end
-
-      def should_skip?(step)
-        step.pending? &&
-          step.upstream_steps.any? { |upstream| permanently_unsatisfied?(upstream) }
-      end
-
-      def permanently_unsatisfied?(upstream)
-        upstream.terminal_coordination_status? &&
-          !upstream.succeeded? &&
-          !upstream.skipped_by_branch? &&
-          effective_failure_strategy(upstream) != :ignore
-      end
-
-      def skip_all_pending_steps(pipeline, except_dependents_of:)
-        exempt_step_ids = if effective_failure_strategy(except_dependents_of) == :ignore
-                            transitive_downstream_ids(except_dependents_of)
-                          else
-                            Set.new
-                          end
-
-        pipeline.steps.pending.find_each do |pending_step|
-          next if exempt_step_ids.include?(pending_step.id)
-
-          pending_step.transition_coordination_status_to!(:skipped)
-        end
-      end
-
       def transitive_downstream_ids(step)
         visited = Set.new
         queue = step.downstream_steps.pluck(:id)

data/lib/good_pipeline/engine.rb

@@ -13,6 +13,12 @@ module GoodPipeline
       end
     end
 
+    initializer "good_pipeline.haltable" do
+      ActiveSupport.on_load(:active_job) do
+        include GoodPipeline::Haltable
+      end
+    end
+
     initializer "good_pipeline.cleanup_hook" do
       ActiveSupport::Notifications.subscribe("cleanup_preserved_jobs.good_job") do |event|
         timestamp = event.payload[:timestamp]

data/lib/good_pipeline/graph_validator.rb

@@ -12,11 +12,10 @@ module GoodPipeline
 
     def validate!
       check_empty_pipeline!
-      check_duplicate_keys!
-      build_steps_by_key!
-      check_self_dependencies!
+      build_index!
       check_unknown_references!
       check_cycles!
+      @steps_by_key
     end
 
     private
@@ -25,22 +24,20 @@ module GoodPipeline
       raise InvalidPipelineError, "pipeline has no steps" if @step_definitions.empty?
     end
 
-    def check_duplicate_keys!
-      seen = {}
+    def build_index! # rubocop:disable Metrics/AbcSize
+      @steps_by_key = {}
+      @forward_edges = Hash.new { |h, k| h[k] = [] }
+
       @step_definitions.each do |step|
-        raise InvalidPipelineError, "duplicate step key :#{step.key}" if seen.key?(step.key)
+        raise InvalidPipelineError, "duplicate step key :#{step.key}" if @steps_by_key.key?(step.key)
 
-        seen[step.key] = true
-      end
-    end
+        step.dependencies.each do |dependency_key|
+          raise InvalidPipelineError, "step :#{step.key} depends on itself" if dependency_key == step.key
 
-    def build_steps_by_key!
-      @steps_by_key = @step_definitions.to_h { |step| [step.key, step] }
-    end
+          @forward_edges[dependency_key] << step.key
+        end
 
-    def check_self_dependencies!
-      @steps_by_key.each_value do |step|
-        raise InvalidPipelineError, "step :#{step.key} depends on itself" if step.dependencies.include?(step.key)
+        @steps_by_key[step.key] = step
       end
     end
 
@@ -55,17 +52,7 @@ module GoodPipeline
     end
 
     def check_cycles!
-      CycleDetector.check!(@steps_by_key, build_forward_edges)
-    end
-
-    def build_forward_edges
-      edges = Hash.new { |h, k| h[k] = [] }
-      @steps_by_key.each_value do |step|
-        step.dependencies.each do |dependency_key|
-          edges[dependency_key] << step.key
-        end
-      end
-      edges
+      CycleDetector.check!(@steps_by_key, @forward_edges)
     end
   end
 end

data/lib/good_pipeline/haltable.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module GoodPipeline
+  module Haltable
+    def halt_pipeline!
+      step = GoodPipeline::StepRecord.find_by(good_job_id: provider_job_id)
+      step&.update_columns(halt_requested: true, updated_at: Time.current)
+    end
+  end
+end

data/lib/good_pipeline/pipeline.rb

@@ -103,11 +103,10 @@ module GoodPipeline
       @branch_context_stack = []
       @building = true
       configure(**kwargs)
-      GraphValidator.validate!(@step_definitions)
+      @steps_by_key = GraphValidator.validate!(@step_definitions).freeze
       @step_definitions.freeze
       @branch_aliases.freeze
       @building = false
-      @steps_by_key = @step_definitions.to_h { |step| [step.key, step] }.freeze
       @root_steps = @step_definitions.select { |step| step.dependencies.empty? }.freeze
       freeze
     end
@@ -118,7 +117,7 @@ module GoodPipeline
       raise NotImplementedError, "#{self.class} must implement #configure"
     end
 
-    def run(key, job_class, with: {}, after: [], on_failure: nil, enqueue: {}) # rubocop:disable Metrics/MethodLength
+    def run(key, job_class, with: EMPTY_HASH, after: EMPTY_ARRAY, on_failure: nil, enqueue: EMPTY_HASH) # rubocop:disable Metrics/MethodLength
       raise ConfigurationError, "run can only be called inside configure" unless @building
 
       expanded_after = expand_branch_aliases(after)
@@ -186,6 +185,8 @@ module GoodPipeline
     # NOTE: Single-level expansion only. If nested branches are added in the future,
     # this must become recursive to expand inner branch aliases.
     def expand_branch_aliases(dependencies)
+      return Array(dependencies) if @branch_aliases.empty?
+
       Array(dependencies).flat_map { |dependency| @branch_aliases.fetch(dependency, [dependency]) }
     end
   end

data/lib/good_pipeline/runner.rb

@@ -10,61 +10,81 @@ module GoodPipeline
       @pipeline = pipeline_instance
     end
 
-    def call(start: true) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength
+    def call(start: true) # rubocop:disable Metrics/MethodLength
+      pipeline_id = SecureRandom.uuid
       pipeline_record = nil
-      step_records = {}
-
-      PipelineRecord.transaction do # rubocop:disable Metrics/BlockLength
-        pipeline_record = PipelineRecord.create!(
-          type: @pipeline.class.name,
-          params: @pipeline.params,
-          status: :pending,
-          on_failure_strategy: @pipeline.failure_strategy.to_s
-        )
-
-        # Two passes: create all step records first, then dependencies.
-        # Branch steps may appear after their dependents in step_definitions.
-        @pipeline.step_definitions.each do |step_definition|
-          step_records[step_definition.key] = StepRecord.create!(
-            pipeline: pipeline_record,
-            key: step_definition.key.to_s,
-            job_class: resolve_job_class(step_definition),
-            params: step_definition.params,
-            on_failure_strategy: step_definition.failure_strategy&.to_s,
-            enqueue_options: step_definition.enqueue_options,
-            branch: build_branch_hash(step_definition)
-          )
-        end
+      step_id_by_key = {}
 
-        @pipeline.step_definitions.each do |step_definition| # rubocop:disable Style/CombinableLoops
-          step_definition.dependencies.each do |dependency_key|
-            DependencyRecord.create!(
-              pipeline: pipeline_record,
-              step: step_records[step_definition.key],
-              depends_on_step: step_records[dependency_key]
-            )
-          end
-        end
+      PipelineRecord.transaction do
+        batch = create_pipeline_batch(pipeline_id)
+        pipeline_record = create_pipeline_record(pipeline_id, batch.id, start: start)
+        step_id_by_key = insert_steps(pipeline_record)
+        insert_dependencies(pipeline_record, step_id_by_key)
+      end
+
+      enqueue_root_steps(step_id_by_key) if start
+
+      pipeline_record
+    end
+
+    private
 
-        pipeline_batch = GoodJob::Batch.new
-        pipeline_batch.on_finish = "GoodPipeline::PipelineReconciliationJob"
-        pipeline_batch.properties = { pipeline_id: pipeline_record.id }
-        pipeline_batch.save
-        pipeline_record.update_column(:good_job_batch_id, pipeline_batch.id)
+    def create_pipeline_batch(pipeline_id)
+      batch = GoodJob::Batch.new
+      batch.on_finish = "GoodPipeline::PipelineReconciliationJob"
+      batch.properties = { pipeline_id: pipeline_id }
+      batch.save
+      batch
+    end
+
+    def create_pipeline_record(pipeline_id, batch_id, start:)
+      PipelineRecord.create!(
+        id: pipeline_id,
+        type: @pipeline.class.name,
+        params: @pipeline.params,
+        status: start ? :running : :pending,
+        on_failure_strategy: @pipeline.failure_strategy.to_s,
+        good_job_batch_id: batch_id
+      )
+    end
 
-        pipeline_record.transition_to!(:running) if start
+    def insert_steps(pipeline_record) # rubocop:disable Metrics/AbcSize,Metrics/MethodLength
+      step_rows = @pipeline.step_definitions.map do |step_definition|
+        {
+          pipeline_id: pipeline_record.id,
+          key: step_definition.key.to_s,
+          job_class: resolve_job_class(step_definition),
+          params: step_definition.params,
+          on_failure_strategy: step_definition.failure_strategy&.to_s,
+          enqueue_options: step_definition.enqueue_options,
+          branch: build_branch_hash(step_definition),
+          pending_upstream_count: step_definition.dependencies.size
+        }
       end
 
-      if start
-        @pipeline.root_steps.each do |step_definition|
-          Coordinator.try_enqueue_step(step_records[step_definition.key].id)
+      result = StepRecord.insert_all!(step_rows, returning: %w[id key])
+      result.rows.each_with_object({}) { |(id, key), hash| hash[key.to_sym] = id }
+    end
+
+    def insert_dependencies(pipeline_record, step_id_by_key)
+      dependency_rows = @pipeline.step_definitions.flat_map do |step_definition|
+        step_definition.dependencies.map do |dependency_key|
+          {
+            pipeline_id: pipeline_record.id,
+            step_id: step_id_by_key[step_definition.key],
+            depends_on_step_id: step_id_by_key[dependency_key]
+          }
         end
       end
 
-      pipeline_record
+      DependencyRecord.insert_all!(dependency_rows) if dependency_rows.any?
     end
 
-    private
+    def enqueue_root_steps(step_id_by_key)
+      @pipeline.root_steps.each do |step_definition|
+        Coordinator.try_enqueue_step(step_id_by_key[step_definition.key])
+      end
+    end
 
     def resolve_job_class(step_definition)
       step_definition.job_class.is_a?(String) ? step_definition.job_class : step_definition.job_class.name

data/lib/good_pipeline/step_definition.rb

@@ -4,11 +4,29 @@ module GoodPipeline
   class StepDefinition
     SUPPORTED_ENQUEUE_OPTIONS = %i[queue priority wait good_job_labels good_job_notify].freeze
 
-    attr_reader :key, :job_class, :params, :dependencies, :failure_strategy, :enqueue_options,
-                :branch_key, :branch_arm, :decides, :empty_arms
+    attr_reader :key,
+                :job_class,
+                :params,
+                :dependencies,
+                :failure_strategy,
+                :enqueue_options,
+                :branch_key,
+                :branch_arm,
+                :decides,
+                :empty_arms
 
-    def initialize(key:, job_class:, params: {}, dependencies: [], failure_strategy: nil, enqueue_options: {}, # rubocop:disable Metrics/MethodLength
-                   branch_key: nil, branch_arm: nil, decides: nil, empty_arms: [])
+    def initialize( # rubocop:disable Metrics/MethodLength
+      key:,
+      job_class:,
+      params: EMPTY_HASH,
+      dependencies: EMPTY_ARRAY,
+      failure_strategy: nil,
+      enqueue_options: EMPTY_HASH,
+      branch_key: nil,
+      branch_arm: nil,
+      decides: nil,
+      empty_arms: EMPTY_ARRAY
+    )
       @key = key
       @job_class = job_class
       @params = params.freeze
@@ -37,6 +55,8 @@ module GoodPipeline
     end
 
     def validate_enqueue_options!(options)
+      return if options.empty?
+
       unsupported = options.keys.map(&:to_sym) - SUPPORTED_ENQUEUE_OPTIONS
       return if unsupported.empty?
 

data/lib/good_pipeline/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module GoodPipeline
-  VERSION = "0.2.2"
+  VERSION = "0.3.1"
 end

data/lib/good_pipeline.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "good_pipeline/version"
+require_relative "good_pipeline/constants"
 require_relative "good_pipeline/errors"
 require_relative "good_pipeline/step_definition"
 require_relative "good_pipeline/branch_builder"
@@ -12,6 +13,7 @@ require_relative "good_pipeline/branch_resolver"
 require_relative "good_pipeline/coordinator"
 require_relative "good_pipeline/chain_coordinator"
 require_relative "good_pipeline/runner"
+require_relative "good_pipeline/haltable"
 require_relative "good_pipeline/chain"
 require_relative "good_pipeline/engine" if defined?(Rails::Engine)
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: good_pipeline
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.3.1
 platform: ruby
 authors:
 - Ali Hamdi Ali Fadel
@@ -93,6 +93,7 @@ files:
 - demo/app/jobs/cleanup_job.rb
 - demo/app/jobs/download_job.rb
 - demo/app/jobs/failing_job.rb
+- demo/app/jobs/halt_execution_job.rb
 - demo/app/jobs/publish_job.rb
 - demo/app/jobs/retryable_job.rb
 - demo/app/jobs/thumbnail_job.rb
@@ -135,6 +136,7 @@ files:
 - demo/test/integration/test_concurrent_fan_in.rb
 - demo/test/integration/test_end_to_end.rb
 - demo/test/integration/test_enqueue_atomicity.rb
+- demo/test/integration/test_halt_execution.rb
 - demo/test/integration/test_halt_ignore_chain.rb
 - demo/test/integration/test_ignore_transitive_exemption.rb
 - demo/test/integration/test_late_chain_registration.rb
@@ -177,12 +179,14 @@ files:
 - lib/good_pipeline/branch_resolver.rb
 - lib/good_pipeline/chain.rb
 - lib/good_pipeline/chain_coordinator.rb
+- lib/good_pipeline/constants.rb
 - lib/good_pipeline/coordinator.rb
 - lib/good_pipeline/cycle_detector.rb
 - lib/good_pipeline/engine.rb
 - lib/good_pipeline/errors.rb
 - lib/good_pipeline/failure_metadata.rb
 - lib/good_pipeline/graph_validator.rb
+- lib/good_pipeline/haltable.rb
 - lib/good_pipeline/pipeline.rb
 - lib/good_pipeline/runner.rb
 - lib/good_pipeline/step_definition.rb