ductwork 1.1.0 → 1.1.2

data/CHANGELOG-PRO.md

@@ -1,6 +1,22 @@
 # Ductwork Pro Changelog
 
-## [1.0.0] (Unreleased)
+## [1.1.1]
+
+- fix: use `Ductwork::DatabaseClock` instead of `Time.current` for every `started_at` Pro stamps, mirroring the OSS fix
+- fix: evaluate the step-timeout deadline in the database (`DatabaseClock.ago_sql`) instead of subtracting `attempt.started_at` from the worker's `Time.current`, so an NTP correction mid-job cannot stretch or collapse a step's timeout
+- fix: temporarily handle multiple return types from `Branch.with_latest_claimed` - this is a result of changes in OSS from v1.1.0 and v1.1.1
+- fix: log the claimed job/branch of a dead thread before the restart clears it
+- fix: add rubocop cop for ensuring `updated_at` is updated (copied from OSS)
+- fix: explicitly set `updated_at` for queries that update the record directly
+- fix: mirror OSS `halt_branch_and_resolve_run_without_rescue!` implementation
+- fix: walk the `collapse` fan-in with an explicit keyset scan so each batch is one indexed range seek, instead of a per-batch subquery plus a second `in_batches` query that re-ran the whole scope
+- fix: don't use `unique_by` argument for `insert_all` for mysql and trilogy database adapters
+
+## [1.1.0]
+
+- chore: update CLI banner to match OSS
+
+## [1.0.0]
 
 - chore: add multiple database adapters to CI test matrix
 - fix: inspect `in_batches` parameters for passing `cursor` option

data/CHANGELOG.md

@@ -1,5 +1,31 @@
 # Ductwork Changelog
 
+## [1.1.2]
+
+- fix: lazily create configuration
+- fix: prevent step's idempotency key from returning `nil`
+
+## [1.1.1]
+
+- fix: use database clock instead of `Time.current` to avoid clock skew in `Branch#latest_step` paths
+- fix: record error metadata on `process_crashed` results — `Execution#crashed!` now takes the exception that killed the work loop (writing its class, message, and backtrace, as `errored!` already did) or one of the new `Ductwork::Crash` markers (`ProcessCrash`, `ThreadCrash`, `AbandonedClaim`, `OrphanedClaim`) naming which recovery path noticed the lost claim; previously all five call sites collapsed into a single unlabeled result row and the work loop logged the error class and message before discarding them, leaving a crash loop undiagnosable from the database; recorded backtraces are now capped at the 10 frames nearest the raise, since `text` tops out at 64KB on MySQL and an insert failure inside `crashed!` would leak the claim
+- fix: use `Ductwork::DatabaseClock` instead of `Time.current` in claiming queries
+- fix: set `updated_at` timestamp when touching record through `update_all` or `update_columns`
+- fix: reap the forking supervisor's own process record on shutdown, as the threaded supervisor already did, so a clean stop no longer leaves a record behind for a peer to sweep up `supervisor.reaper_timeout` later
+- fix: skip the reaping process's own record in `Process.reap_all!` so a supervisor whose work loop stalled past `supervisor.reaper_timeout` cannot declare its own in-flight work crashed — in threaded mode the advancer and worker threads share the supervisor's process record, so self-reaping released branches and re-enqueued jobs that live threads were still running; a peer supervisor still reaps the record when the process genuinely dies
+- fix: report a heartbeat from the forking supervisor's own work loop — its process record went stale after `supervisor.reaper_timeout` and its own reaper sweep destroyed it, so a healthy supervisor disappeared from the dashboard and the health check reported it `dead` five minutes into every boot, failing any container liveness probe wired to it
+- fix: read the run and pipeline status before writing when marking a claimed job `in_progress`, so a no-op conditional `UPDATE` no longer takes an exclusive row lock held for the rest of the claim transaction — InnoDB at REPEATABLE READ locks the row during the primary key lookup even when the qualification fails, so on MySQL and Trilogy every worker claiming a step of the same run serialized on the two rows the whole run shares, capping per-run claim throughput regardless of worker count
+- fix: jitter the idle polling sleep of advancer and worker threads by ±25% of the configured `polling_timeout` so threads spawned together stop waking in lockstep and piling onto the same claim candidates — the configured timeout remains the average poll latency
+- fix: distinguish a contended claim (candidates existed but every sampled one lost its race) from an idle queue so an advancer that loses a race re-enters the work loop immediately instead of sleeping a full polling interval — `Branch.with_latest_claimed` now returns `:claimed`/`:contended`/`:idle` rather than a boolean
+- fix: sample the branch to claim from a window of the oldest candidates (sized off `pipeline_advancer.count`) and walk to the next sampled ID when a claim loses its race, instead of every advancer thread in every process contending for the single oldest row — raising the advancer count or adding advancer processes now increases advancement throughput instead of degrading it
+- fix: re-assert the full candidate predicate (branch status and an `advancing`/`failed` step, not just a null claim) in the claim `UPDATE` query so an advancer whose candidate went stale can no longer claim a branch that another advancer already advanced and released — which advanced the branch off a freshly enqueued step whose job had not run, routing on a nil return value
+- fix: release the claim instead of routing or halting when `Branch#advance!` finds no step to advance
+- fix: confirm a stuck thread actually died before restarting it, so a `Thread#kill` that cannot interrupt a blocking C-level database call no longer leaves two threads sharing one worker's claim state
+- fix: give each worker/advancer thread a fresh running context on `start` so a thread restarted after a kill is not immediately shut down by the context the kill tore down
+- fix: reset the heartbeat timestamp on `start` so a restarted thread is not judged stuck on the stale timestamp that got it killed
+- fix: log the claimed job/branch of a dead thread before the restart clears it
+- fix: let the nested halt path (`divert` no-match, `divide`/`expand` fan-out cap) propagate errors to the advancement rescue instead of committing a step as `completed` alongside an unclaimable branch
+
 ## [1.1.0]
 
 - chore: add color to CLI banner

data/CLAUDE.md

@@ -1,4 +1,4 @@
-# Ductwork OSS Architecture Context
+# Ductwork OSS Context
 
 This is the OSS `ductwork` gem (LGPL v3). The paid `ductwork-pro` gem
 extends it via `prepend` and adds features that MUST NOT be reimplemented

data/lib/ductwork/abandoned_claim.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Ductwork
+  # A work loop exited an iteration with an uncommitted claim without raising
+  # a StandardError -- a non-StandardError exception (a signal, Thread#kill),
+  # a non-local exit, or a rescue path that itself failed to commit.
+  class AbandonedClaim < Ductwork::Crash; end
+end

data/lib/ductwork/branch_claim.rb

@@ -1,12 +1,19 @@
 # frozen_string_literal: true
 
 module Ductwork
-  class BranchClaim
+  class BranchClaim # rubocop:disable Metrics/ClassLength
     attr_reader :transition, :advancement, :token
 
+    MAX_CLAIM_ATTEMPTS = 3
+
     def initialize(pipeline_klass)
       @pipeline_klass = pipeline_klass
       @claimed_for_advancing_at = nil
+      @contended = false
+    end
+
+    def contended?
+      @contended
     end
 
     def latest
@@ -22,16 +29,20 @@ module Ductwork
 
       return log_no_process if process.nil?
 
-      id = find_candidate_branch_id
+      ids = find_candidate_branch_ids
 
-      return log_no_branches if id.blank?
+      return log_no_branches if ids.blank?
 
-      rows_updated = claim_and_setup_records(id, process)
+      Ductwork::FaultInjection.checkpoint(:after_branch_candidate_select)
 
-      if rows_updated == 1
-        Ductwork::Branch.find(id)
+      claimed_id = claim_and_setup_records(ids, process)
+
+      if claimed_id
+        Ductwork::Branch.find(claimed_id)
       else
-        log_race_condition(id)
+        @contended = true
+
+        log_lost_claim_races
       end
     rescue ActiveRecord::InvalidForeignKey => e
       # NOTE: our own `process` record was reaped (heartbeat-stale, destroyed)
@@ -41,49 +52,95 @@ module Ductwork
       # the whole claim transaction (branch + transition + advancement) --
       # nothing is left half-committed. Treat it like any other lost claim
       # race instead of letting it propagate and kill the advancer thread.
-      log_process_reaped_mid_claim(id, e)
+      log_process_reaped_mid_claim(e)
     end
 
     private
 
-    attr_reader :pipeline_klass, :claimed_for_advancing_at
+    attr_reader :pipeline_klass, :claimed_for_advancing_at, :attempted_branch_id
+
+    def candidate_window_size
+      @candidate_window_size ||= begin
+        pipeline_advancer_count = Ductwork
+                                  .configuration
+                                  .pipeline_advancer_count(pipeline_klass)
 
-    def find_candidate_branch_id
+        (pipeline_advancer_count * 4).clamp(20, 100)
+      end
+    end
+
+    # NOTE: the single source of truth for "this branch has work to advance",
+    # shared by the candidate `SELECT` and the claiming `UPDATE` so the two can
+    # never drift. Without a `SELECT ... FOR UPDATE`, candidate selection and
+    # claiming are separate statements, so the claim MUST re-assert every
+    # predicate the selection relied on -- not just `claimed_for_advancing_at IS
+    # NULL`. Between the two, the advancer that already owned this branch can
+    # finish and `release!` it (back to `in_progress`, claim nulled), which
+    # leaves the null check satisfied by a branch whose latest step is now a
+    # freshly enqueued `in_progress` one. Claiming that would advance the branch
+    # off a step whose job has not run yet -- routing on a nil return value and
+    # minting a duplicate downstream step. The window is normally microseconds,
+    # but the candidate `SELECT` scans the branch/step tables, so at scale
+    # (~1M live branches) it stretches into the tens of milliseconds and the
+    # race becomes routine. Sampling from a window rather than always taking
+    # the single oldest row spreads advancers across distinct candidates, but
+    # it also means an id can sit unattempted while earlier attempts in the
+    # walk run -- so the ids reaching the claiming `UPDATE` are, if anything,
+    # staler than before and the re-assertion matters more, not less.
+    # Re-asserting the status keeps a `completed`/`halted` branch from being
+    # resurrected the same way.
+    def claimable_branches
       Ductwork::Branch
         .in_progress
         .where(pipeline_klass:, claimed_for_advancing_at:)
-        .where(steps: Ductwork::Step.where(status: %w[advancing failed]))
+        .where(steps: Ductwork::Step.where(status: Ductwork::Step::ADVANCEABLE_STATUSES))
+    end
+
+    def find_candidate_branch_ids
+      claimable_branches
         .order(:last_advanced_at)
-        .limit(1)
+        .limit(candidate_window_size)
         .pluck(:id)
-        .first
+        .sample(MAX_CLAIM_ATTEMPTS)
     end
 
-    def claim_and_setup_records(id, process)
-      now = Time.current
+    def claim_and_setup_records(ids, process)
+      ids.each do |id|
+        @attempted_branch_id = id
+        claimed_id = attempt_claim(id, process)
+
+        return claimed_id if claimed_id
+      end
+
+      nil
+    end
+
+    def attempt_claim(id, process)
+      now = Ductwork::DatabaseClock.now
       @token = SecureRandom.uuid
 
       Ductwork::Record.transaction do
-        rows_updated = Ductwork::Branch
-                       .where(id:, claimed_for_advancing_at:)
+        rows_updated = claimable_branches
+                       .where(id:)
                        .update_all(
                          claimed_for_advancing_at: now,
                          claim_token: token,
-                         status: :advancing
+                         status: :advancing,
+                         updated_at: now
                        )
 
-        if rows_updated == 1
-          branch = Branch.find(id)
-          @transition = find_or_create_transition(branch, now)
-          Ductwork::FaultInjection.checkpoint(:before_advancement_create)
-          @advancement = transition.advancements.create!(
-            process: process,
-            started_at: now,
-            crash_count: next_crash_count(transition)
-          )
-        end
-
-        rows_updated
+        next nil unless rows_updated == 1
+
+        branch = Branch.find(id)
+        @transition = find_or_create_transition(branch, now)
+        Ductwork::FaultInjection.checkpoint(:before_advancement_create)
+        @advancement = transition.advancements.create!(
+          process: process,
+          started_at: now,
+          crash_count: next_crash_count(transition)
+        )
+
+        id
       end
     end
 
@@ -133,7 +190,7 @@ module Ductwork
         .first
         &.update!(
           completed_at: now,
-          error_klass: "Ductwork::ProcessCrash",
+          error_klass: Ductwork::ProcessCrash.name,
           error_message: "Advancement was abandoned from a process crash"
         )
     end
@@ -158,10 +215,9 @@ module Ductwork
       nil
     end
 
-    def log_race_condition(id)
+    def log_lost_claim_races
       Ductwork.logger.debug(
-        msg: "Did not claim branch, avoided race condition",
-        branch_id: id,
+        msg: "Did not claim branch, lost races on all sampled IDs",
         pipeline_klass: pipeline_klass,
         role: :pipeline_advancer
       )
@@ -169,10 +225,10 @@ module Ductwork
       nil
     end
 
-    def log_process_reaped_mid_claim(id, error)
+    def log_process_reaped_mid_claim(error)
       Ductwork.logger.warn(
         msg: "Did not claim branch, our process record was reaped mid-claim",
-        branch_id: id,
+        branch_id: attempted_branch_id,
         pipeline_klass: pipeline_klass,
         error_klass: error.class.to_s,
         error_message: error.message,

data/lib/ductwork/claimed_state.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+module Ductwork
+  # NOTE: normalizes the display state of a freshly claimed job, moving its
+  # step, run, and pipeline to `in_progress`.
+  #
+  # The step row belongs to the one claim, but the run and pipeline rows are
+  # shared by every step of the run, so a million-wide expand funnels every
+  # concurrent claim through the same two rows. Each write is already
+  # conditional and matches nothing once the run is under way, but a no-op
+  # `UPDATE` is not free everywhere: PostgreSQL never locks a tuple that fails
+  # the qualification, while InnoDB at REPEATABLE READ takes the exclusive lock
+  # during the primary key lookup and only releases it on a non-matching row
+  # under READ COMMITTED's semi-consistent read. On MySQL and Trilogy that left
+  # every worker holding both shared rows for the rest of its claim
+  # transaction, capping claim throughput for a single run regardless of how
+  # many workers were running.
+  #
+  # Reading first keeps the steady state lock free, since a plain `SELECT` is a
+  # consistent-snapshot read on every supported adapter. The writes stay
+  # conditional, so losing the guard's race is benign: a stale read costs one
+  # redundant no-op `UPDATE` on the single claim that actually transitions.
+  class ClaimedState
+    def self.mark_in_progress!(step)
+      new(step).mark_in_progress!
+    end
+
+    def initialize(step)
+      @step = step
+    end
+
+    def mark_in_progress!
+      transition!(Ductwork::Step, step.id)
+
+      run_status, pipeline_id = Ductwork::Run.where(id: step.run_id).pick(:status, :pipeline_id)
+
+      return if run_status.nil?
+
+      transition!(Ductwork::Run, step.run_id) unless run_status == "in_progress"
+
+      return if Ductwork::Pipeline.where(id: pipeline_id, status: "in_progress").exists?
+
+      transition!(Ductwork::Pipeline, pipeline_id)
+    end
+
+    private
+
+    attr_reader :step
+
+    def transition!(klass, id)
+      klass
+        .where(id:)
+        .where.not(status: "in_progress")
+        .update_all(status: "in_progress", updated_at: Time.current)
+    end
+  end
+end

data/lib/ductwork/crash.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Ductwork
+  class Crash < StandardError; end
+end

data/lib/ductwork/engine.rb

@@ -20,7 +20,6 @@ module Ductwork
     end
 
     initializer "ductwork.configure" do
-      Ductwork.configuration ||= Ductwork::Configuration.new
       Ductwork.logger ||= Ductwork::Configuration::DEFAULT_LOGGER
     end
   end

data/lib/ductwork/models/advancement.rb

@@ -7,19 +7,22 @@ module Ductwork
 
     validates :started_at, presence: true
 
-    CRASH_ERROR_KLASSES = %w[Ductwork::ProcessCrash Ductwork::ThreadCrash].freeze
+    CRASH_ERROR_KLASSES = [Ductwork::ProcessCrash.name, Ductwork::ThreadCrash.name].freeze
 
     def crash?
       CRASH_ERROR_KLASSES.include?(error_klass)
     end
 
     def process_crashed!
+      completed_at = updated_at = Time.current
+
       Ductwork::Record.transaction do
         rows_updated = self.class
                            .where(id: id, completed_at: nil)
                            .update_all(
-                             completed_at: Time.current,
-                             error_klass: "Ductwork::ProcessCrash",
+                             completed_at: completed_at,
+                             updated_at: updated_at,
+                             error_klass: Ductwork::ProcessCrash.name,
                              error_message: "Reaped from orphaned process"
                            )
 
@@ -30,12 +33,15 @@ module Ductwork
     end
 
     def thread_crashed!(expected_token)
+      completed_at = updated_at = Time.current
+
       Ductwork::Record.transaction do
         rows_updated = self.class
                            .where(id: id, completed_at: nil)
                            .update_all(
-                             completed_at: Time.current,
-                             error_klass: "Ductwork::ThreadCrash",
+                             completed_at: completed_at,
+                             updated_at: updated_at,
+                             error_klass: Ductwork::ThreadCrash.name,
                              error_message: "Advancement abandoned from a thread crash"
                            )
 

data/lib/ductwork/models/branch.rb

@@ -56,9 +56,11 @@ module Ductwork
 
         yield branch, branch_claim.transition, branch_claim.advancement
 
-        true
+        :claimed
+      elsif branch_claim.contended?
+        :contended
       else
-        false
+        :idle
       end
     ensure
       advancement = branch_claim.advancement
@@ -107,6 +109,19 @@ module Ductwork
       step = latest_step
       max_crash = Ductwork.configuration.pipeline_advancer_max_crash
 
+      # NOTE: backstop for a stale claim, checked before anything else because a
+      # branch with no advanceable step must be neither routed NOR halted. Every
+      # legitimate claim arrives with its latest step in `advancing` (job
+      # succeeded, or `revive!`'s re-advance) or `failed` (budgets exhausted);
+      # `BranchClaim` re-asserts exactly that in its claiming UPDATE. If we
+      # nonetheless hold a branch whose latest step is still running, the
+      # previous advancer already handled that step and enqueued the next one --
+      # there is nothing to advance and no failure to attribute. A poison branch
+      # that keeps crashing the advancer is unaffected: its step stays
+      # `advancing` (the crash rolled the transition back), so the crash cap
+      # below still fires.
+      return release_stale_claim!(transition, advancement, step) unless advanceable_step?(step)
+
       # NOTE: the crash cap is checked first as the true backstop against a
       # poison branch that repeatedly crashes the advancer process/thread. In a
       # normal failed-step halt no advancer crashes have accrued, so this only
@@ -168,18 +183,49 @@ module Ductwork
     # silently leave the branch stranded in `advancing`. Callers outside an
     # advancement (fresh branch objects, explicit tokens) are unaffected.
     def release!(expected_token = @claim_fence_token || claim_token)
+      last_advanced_at = updated_at = Time.current
+
       Ductwork::Branch
         .where(id: id, claim_token: expected_token, status: :advancing)
         .update_all(
           claimed_for_advancing_at: nil,
           claim_token: nil,
           status: :in_progress,
-          last_advanced_at: Time.current
+          last_advanced_at: last_advanced_at,
+          updated_at: updated_at
         )
     end
 
     private
 
+    def advanceable_step?(step)
+      step.present? && Ductwork::Step::ADVANCEABLE_STATUSES.include?(step.status)
+    end
+
+    # NOTE: close our bookkeeping and hand the branch back untouched. The
+    # advancement is closed WITHOUT an `error_klass` on purpose: this is not an
+    # advancer failure, and stamping one would burn either the retry budget
+    # (`too_many_failed_attempts?`) or the crash budget (`next_crash_count`) and
+    # eventually halt a perfectly healthy branch. `release!` restores
+    # `in_progress`, so the branch no longer matches the candidate predicate
+    # until its running step lands -- no reclaim loop.
+    def release_stale_claim!(transition, advancement, step)
+      with_claim_fence do
+        now = Time.current
+        advancement&.update!(completed_at: now)
+        transition.update!(completed_at: now)
+        release!
+      end
+
+      Ductwork.logger.warn(
+        msg: "Branch claimed with no step to advance, released",
+        branch_id: id,
+        step_id: step&.id,
+        step_status: step&.status,
+        role: :pipeline_advancer
+      )
+    end
+
     def log_claim_diverged
       Ductwork.logger.info(
         msg: "Branch claim no longer held",
@@ -201,13 +247,28 @@ module Ductwork
       end
     end
 
+    # NOTE: the unrescued form of the halt, for callers that are ALREADY inside a
+    # `with_claim_fence`. Those callers must NOT use the rescuing wrapper below:
+    # `Ductwork::Record.transaction` nests by joining (no savepoint), so a rescue
+    # here swallows the error without unwinding the caller's still-open outer
+    # transaction. The fence would then commit the caller's partial work (a step
+    # already flipped to `completed`) alongside a branch the rescue put back to
+    # `in_progress` — a branch `find_candidate_branch_id` can never select again
+    # (no step in `advancing`/`failed`) and no reaper covers (the advancement was
+    # closed). Letting the error propagate instead lets `route_by_edge`'s rescue,
+    # which sits outside the fence, roll the whole thing back and retry.
+    def halt_branch_and_resolve_run_without_rescue!(transition, advancement, halt_reason)
+      now = Time.current
+      advancement.update!(completed_at: now)
+      transition.update!(completed_at: now)
+      halt!(halt_reason)
+      run.resolve_terminal_state!
+    end
+
+    # NOTE: only for callers outside a claim fence (`advance!`'s direct halts).
     def halt_branch_and_resolve_run!(transition, advancement, halt_reason)
       with_claim_fence do
-        now = Time.current
-        advancement.update!(completed_at: now)
-        transition.update!(completed_at: now)
-        halt!(halt_reason)
-        run.resolve_terminal_state!
+        halt_branch_and_resolve_run_without_rescue!(transition, advancement, halt_reason)
       end
     rescue StandardError => e
       # NOTE: re-enter the claim fence before mutating branch/advancement state.
@@ -346,7 +407,7 @@ module Ductwork
       input_arg = Ductwork::Job.find_by(step: latest_step).return_value
       node = edge[:to].sole
       klass = run.parsed_definition.dig(:edges, node, :klass)
-      started_at = Time.current
+      started_at = Ductwork::DatabaseClock.now
 
       with_claim_fence do
         latest_step.update!(status: :completed, completed_at: Time.current)
@@ -396,7 +457,7 @@ module Ductwork
                       .map(&:return_value)
           next_node = edge[:to].sole
           klass = run.parsed_definition.dig(:edges, next_node, :klass)
-          started_at = Time.current
+          started_at = Ductwork::DatabaseClock.now
           next_branch = run.branches.create!(
             started_at: started_at,
             status: "in_progress",
@@ -537,7 +598,7 @@ module Ductwork
                       .map(&:return_value)
           next_node = edge[:to].sole
           klass = run.parsed_definition.dig(:edges, next_node, :klass)
-          started_at = Time.current
+          started_at = Ductwork::DatabaseClock.now
           next_branch = run.branches.create!(
             started_at: started_at,
             status: "in_progress",
@@ -573,7 +634,7 @@ module Ductwork
       input_arg = Ductwork::Job.find_by(step: latest_step).return_value
       node = edge[:to].sole
       klass = run.parsed_definition.dig(:edges, node, :klass)
-      started_at = Time.current
+      started_at = Ductwork::DatabaseClock.now
 
       with_claim_fence do
         latest_step.update!(status: :completed, completed_at: Time.current)
@@ -599,12 +660,16 @@ module Ductwork
       input_arg = Ductwork::Job.find_by(step: latest_step).return_value
       node = edge[:to][input_arg.to_s] || edge[:to]["otherwise"]
       klass = run.parsed_definition.dig(:edges, node, :klass)
-      started_at = Time.current
+      started_at = Ductwork::DatabaseClock.now
 
       if node.nil?
         with_claim_fence do
           latest_step.update!(status: :completed, completed_at: Time.current)
-          halt_branch_and_resolve_run!(transition, advancement, "condition_unmatched")
+          halt_branch_and_resolve_run_without_rescue!(
+            transition,
+            advancement,
+            "condition_unmatched"
+          )
         end
       else
         with_claim_fence do
@@ -628,7 +693,7 @@ module Ductwork
     end
 
     def divide_branch(edge, transition, advancement) # rubocop:todo Metrics
-      started_at = Time.current
+      started_at = Ductwork::DatabaseClock.now
       input_arg = Ductwork::Job.find_by(step: latest_step).return_value
       too_many = edge[:to].tally.any? do |to_klass, count|
         depth = Ductwork
@@ -641,7 +706,11 @@ module Ductwork
       if too_many
         with_claim_fence do
           latest_step.update!(status: :completed, completed_at: Time.current)
-          halt_branch_and_resolve_run!(transition, advancement, "max_fanout_exceeded")
+          halt_branch_and_resolve_run_without_rescue!(
+            transition,
+            advancement,
+            "max_fanout_exceeded"
+          )
         end
       else
         with_claim_fence do
@@ -686,7 +755,11 @@ module Ductwork
       if max_depth != -1 && return_value.count > max_depth
         with_claim_fence do
           latest_step.update!(status: :completed, completed_at: Time.current)
-          halt_branch_and_resolve_run!(transition, advancement, "max_fanout_exceeded")
+          halt_branch_and_resolve_run_without_rescue!(
+            transition,
+            advancement,
+            "max_fanout_exceeded"
+          )
         end
       elsif return_value.none?
         complete_branch_and_pipeline(transition, advancement)
@@ -698,7 +771,7 @@ module Ductwork
     def bulk_create_steps_and_jobs(edge:, return_value:, transition:, advancement:) # rubocop:todo Metrics
       node = edge[:to].sole
       next_klass = run.parsed_definition.dig(:edges, node, :klass)
-      now = Time.current
+      now = Ductwork::DatabaseClock.now
 
       with_claim_fence do # rubocop:todo Metrics/BlockLength
         latest_step.update!(status: :completed, completed_at: Time.current)