chrono_forge 0.11.0 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c2b3299dbce8cb34289b708af6e0d22d19d764de09978ae257cb87fd37322b5b
-  data.tar.gz: f8ca2ae0af221d262d8109b06bd4c319a679d14ea0b440404ef96997134f8b17
+  metadata.gz: fd4d0c02357d2d185a6d82034d21a6e09c034c0ae04a9b2cbd5cfcb5bb29b8f0
+  data.tar.gz: 981e5d7208c8a80dacb9a13a439e28a896037980364b9a810f976daa93597d65
 SHA512:
-  metadata.gz: c0e81af8116529cbcb1b13fc2f23c952c3622b13f615281f7f97c82ce41cda244316bbb77a99666c4709e8211747d931bdf31d7b02cad011119db8b49b72714d
-  data.tar.gz: 427469cfe3d72bd2891d156227be9a32a1879cfa1dc8f52c8c54f87bf99feeabc1b8d9dfca46fb37e58bae6cb84f560f77fca94888dcfcbd32fededfc70bd1f1
+  metadata.gz: 7a4ccac2097e6c808103327201f00c1e79341bbd433f1564c5a2ffc306c3740509e195825e613781d62d27844e230ce12b3a048d116a2d44ba3b61ea11f16f7c
+  data.tar.gz: e974b5cfd5aaf41045b84947993cb1052a19503bab85e2309b8b5ea13563340fac5136800c30d1a63ee34408c47262ed67922dc21166a6f2361d3742d021be18

data/CHANGELOG.md

@@ -1,3 +1,13 @@
+## [0.12.0] - 2026-07-09
+
+### Documentation
+
+- Document the stalled-workflow reaper and fix broken README anchors
+
+### Features
+
+- Reap workflows stranded in running by hard-killed workers
+
 ## [0.11.0] - 2026-07-04
 
 ### Bug Fixes

data/README.md

@@ -90,7 +90,7 @@ A few deliberate choices behind that table:
 - **ActiveJob Integration**: Compatible with all ActiveJob backends, though database-backed processors (like Solid Queue) provide the most reliable experience for long-running workflows
 - **Retention & Cleanup**: A schedulable job to prune finished workflows and the unbounded logs that periodic tasks accumulate (see [Cleanup & Retention](#-cleanup--retention))
 
-## 🖥️ Dashboard
+## 🖥 Dashboard
 
 ChronoForge has a free, mountable dashboard for visibility and recovery: workflow list, step replay timeline, a per-run **definition graph** (the durable steps a workflow will run, statically parsed from `perform`, with live run status overlaid), context inspector, periodic-task health, wait-state age, and retry/unlock actions. It ships as a separate gem, `chrono_forge-dashboard`, so the core stays lean.
 
@@ -344,7 +344,15 @@ end
 - **Idempotent**: Same operation won't be executed twice during replays
 - **Automatic Retries**: Failed executions retry per a unified `RetryPolicy` (exponential backoff with jitter; the step default caps at 30s over 3 attempts)
 - **Error Tracking**: All failures are logged with detailed error information
-- **Configurable**: Pass a `retry_policy:` per call, or set a class-wide default with the `retry_policy` DSL (see [Retry Policies](#retry-policies))
+- **Configurable**: Pass a `retry_policy:` per call, or set a class-wide default with the `retry_policy` DSL (see [Retry Policies](#-retry-policies))
+
+> **Write your side effects to be idempotent.** A step short-circuits on replay only
+> once its log row reaches `completed`. If a worker is hard-killed (SIGKILL/OOM/eviction)
+> *after* a step's side effect commits but *before* its log is marked `completed`, the
+> step re-runs when the workflow is resumed (including by the [reaper](#recovering-stranded-workflows-hard-killed-workers)).
+> Make external side effects safe to repeat — use a natural/unique key with
+> `create_or_find_by`, an upsert, or a rescue on the uniqueness violation — rather than
+> assuming exactly-once execution.
 
 #### 🔁 Retry Policies
 
@@ -918,6 +926,7 @@ stateDiagram-v2
 - **Identifiers**: Has locked_at and locked_by values set
 - **Behavior**: Protected against concurrent execution
 - **Duration**: Should be brief unless performing long operations
+- **Note**: A workflow whose worker was hard-killed mid-pass stays stuck here with a stale lock. See [Recovering stranded workflows](#recovering-stranded-workflows-hard-killed-workers).
 
 #### Completed
 - **Description**: The workflow has successfully executed all steps
@@ -966,25 +975,60 @@ ChronoForge::Workflow.failed.find_each(&:retry_later)
 The class-level form (`MyWorkflow.retry_now(key)` / `retry_later(key)`) still
 works if you have the class and key rather than the record.
 
-#### Monitoring Running Workflows
+#### Recovering stranded workflows (hard-killed workers)
+
+When a worker is **hard-killed** mid-pass — SIGKILL from a deploy/rollout, an OOM kill,
+a node eviction, or a SolidQueue heartbeat prune — Ruby's `ensure` block does not run.
+The executor releases the workflow's lock and publishes the resume continuation in that
+`ensure`, so a hard kill leaves the workflow stuck in `running` with a stale lock **and**
+nothing scheduled to wake it. It is fully resumable (a resuming pass steals the stale
+lock and replays completed steps as no-ops), but nothing re-enqueues it on its own:
+`retry_now`/`retry_later` refuse a `running` workflow, and a dashboard "force unlock"
+clears the lock but enqueues no job.
 
-Long-running workflows might indicate issues:
+`ChronoForge::Workflow.reap_stalled` reconciles these. It finds every workflow in
+`running` whose lock is older than `reap_stale_after` (top-level workflows **and** branch
+children) and re-enqueues it, returning the number reaped:
 
 ```ruby
-# Find workflows running for too long
-long_running = ChronoForge::Workflow.where(state: :running)
-                                   .where('locked_at < ?', 30.minutes.ago)
+ChronoForge::Workflow.reap_stalled
+# => 3   (also logs "ChronoForge reaped 3 stalled workflow(s)")
 
-long_running.each do |workflow|
-  # Log potential issues for investigation
-  Rails.logger.warn "Workflow #{workflow.key} has been running for >30 minutes"
-  
-  # Optionally force unlock if you suspect deadlock
-  # CAUTION: Only do this if you're certain the job is stuck
-  # workflow.update!(locked_at: nil, locked_by: nil, state: :idle)
+# Override the threshold for a one-off sweep:
+ChronoForge::Workflow.reap_stalled(stale_after: 15.minutes)
+```
+
+It is **not** run automatically — schedule it from your own scheduler, the same way you
+schedule `ChronoForge::Cleanup`. For example, a SolidQueue recurring task:
+
+```yaml
+# config/recurring.yml
+reap_stalled_workflows:
+  command: "ChronoForge::Workflow.reap_stalled"
+  schedule: "every 5 minutes"
+```
+
+Re-enqueue is safe under concurrency: overlapping sweeps (or a re-enqueue landing while
+the old stale lock still shows) at worst enqueue a duplicate, which loses the
+lock-acquisition race and no-ops. Because reaping replays the interrupted pass, steps
+with external side effects must be idempotent — see the note under
+[Durable Execution](#-durable-execution).
+
+`reap_stale_after` defaults to **3× `max_duration`** (30 minutes out of the box). Both are
+configurable, and because the reap threshold derives from `max_duration` it always stays
+safely above the lock-steal threshold — raise one and the other follows:
+
+```ruby
+ChronoForge.configure do |c|
+  c.max_duration    = 10.minutes  # how long one pass may hold its lock before it's stealable
+  c.reap_stale_after = 45.minutes # optional: pin the reap threshold explicitly (else 3x max_duration)
 end
 ```
 
+> **Not covered:** a workflow parked on a branch merge whose `BranchMergeJob` poller was
+> itself hard-killed sits `idle` (not `running`), so the reaper does not sweep it. That is
+> a distinct failure mode.
+
 ## 🧹 Cleanup & Retention
 
 ChronoForge keeps every workflow and execution-log row indefinitely so that

data/docs/fanout-scale-test.md

@@ -64,8 +64,7 @@ degradation as the tables grew. The path has a **stable steady-state ceiling**.
 
 ## Part 2 — Benchmark: baseline vs commit-consolidation
 
-**Baseline** is the current released engine (v0.10). **Consolidation** is the
-unreleased patch targeted for **v0.12** that cuts per-child write cost without
+**Baseline** is (v0.10). **Consolidation** is the patch released in **v0.11** that cuts per-child write cost without
 changing behaviour. It (a) folds `started_at` into the lock-acquire
 transaction, (b) collapses `complete_workflow!`'s three separate commits — marker
 INSERT + `workflow.completed!` UPDATE + marker UPDATE — into one transaction, and

data/docs/superpowers/specs/2026-07-09-chrono-forge-reaper-design.md

@@ -0,0 +1,175 @@
+# Design: Stalled-Workflow Reaper for ChronoForge
+
+**Date:** 2026-07-09
+**Gem:** `chrono_forge` (with `chrono_forge-dashboard`)
+**Status:** Approved for planning
+
+## Problem
+
+A workflow that is mid-pass when its worker is **hard-killed** (SIGKILL: deploy/rollout,
+OOM, node eviction, SolidQueue heartbeat prune) is stranded permanently in `state: running`
+with a stale lock, and nothing ever resumes it.
+
+The mechanism, confirmed against the code:
+
+- `Executor#perform` releases the lock and publishes the resume continuation in its `ensure`
+  block (`lib/chrono_forge/executor.rb:178-197`): `release_lock` first, then
+  `flush_continuation!`.
+- `enqueue_continuation` (`executor.rb:369-371`) only records `@continuation` in memory;
+  `flush_continuation!` (`executor.rb:376-382`) is what actually enqueues the ActiveJob.
+- On SIGKILL, Ruby's `ensure` does not run. So (1) `release_lock` never runs — the row stays
+  `state: running` with a past `locked_at`; and (2) `flush_continuation!` never runs — no job
+  is scheduled to resume the workflow.
+
+The workflow is fully *resumable* — `Workflow#executable?` is `idle? || running?`
+(`workflow.rb:48-50`) and `LockStrategy.acquire_lock` steals any lock older than
+`max_duration` (10 min) (`lock_strategy.rb:17-21`) — but nothing re-enqueues it. Existing
+recovery mechanisms do not cover this:
+
+- **`BranchMergeJob` rekick** (`branch_merge_job.rb:232-273`) only re-enqueues children that
+  are `state: idle, started_at: nil` (never-started dropped dispatches). A child hard-killed
+  **mid-pass** is `state: running` with `started_at` set — explicitly excluded (see the
+  comment at `branch_merge_job.rb:226-231`). So branch children have the identical gap.
+- **Workflow-level retry** rides on the same process that must survive to its `ensure`; a
+  killed process never reaches it.
+- **`retry_now`/`retry_later`** require `retryable?` (`stalled? || failed?`,
+  `workflow.rb:53-55`); a stranded workflow is `running`, so they refuse it.
+- **Dashboard "Force unlock"** clears the lock but enqueues nothing.
+
+Real-world impact (reported): 735 workflows stranded over ~8.5 months in one production app,
+still accruing ~8/day.
+
+## Solution Overview
+
+Add a reconciliation entry point, `ChronoForge::Workflow.reap_stalled`, that finds workflows
+stuck in `running` past a safety threshold and re-enqueues them. Re-enqueue is safe:
+`acquire_lock` steals the stale lock and completed durable steps replay as no-ops.
+
+### Decisions
+
+| Decision | Choice | Rationale |
+|---|---|---|
+| Shipping | **Class method only** | No Railtie/rake task/self-scheduling job. Host apps call it from their own cron/recurring job. Smallest surface; no new gem infrastructure. |
+| Scope | **All stranded `running` workflows** (top-level AND branch children) | Strictly more correct — rekick misses mid-pass-killed children. Re-enqueue of a child directly is the proven-safe pattern rekick already uses. |
+| Threshold | **Config + per-call override** | `ChronoForge.config.reap_stale_after` (default `30.minutes`), overridable via `stale_after:` arg. Parity with the existing `branch_merge_queue` config. |
+| Deliverable | **Code + tests + docs** | Implementation, tests (incl. simulated hard-kill strand), and docs covering the reaper + step-idempotency expectations. |
+
+## Component
+
+`ChronoForge::Workflow.reap_stalled` — a class method on the AR model. It only queries and
+re-enqueues; it holds no lock and performs no replay itself (the re-enqueued job does the
+replay, under the normal lock protocol).
+
+```ruby
+# ChronoForge::Workflow
+def self.reap_stalled(stale_after: ChronoForge.config.reap_stale_after)
+  reaped = 0
+  where(state: states[:running])
+    .where("locked_at < ?", stale_after.ago)
+    .find_each do |wf|
+      wf.job_klass.perform_later(wf.key, **wf.kwargs.symbolize_keys)
+      reaped += 1
+    rescue => e
+      Rails.logger.error { "ChronoForge reap failed for workflow(#{wf.key}): #{e.class}: #{e.message}" }
+    end
+  Rails.logger.info { "ChronoForge reaped #{reaped} stalled workflow(s)" } if reaped.positive?
+  reaped
+end
+```
+
+Design points:
+
+- **Target = `running` + stale `locked_at`.** The strand happens precisely because the
+  `ensure` never ran, so `release_lock` never flipped the row off `running`. A `NULL locked_at`
+  cannot match `locked_at < ?` and is naturally excluded (an anomalous running-without-lock row
+  is not our target).
+- **Sweeps children too.** No `parent_execution_log_id` filter. A re-enqueued child runs,
+  replays, completes, and updates its `BranchProbe` state so the parent's merge poller observes
+  completion.
+- **Re-enqueue via the public guarded `perform_later`.** `wf.kwargs` is a JSON column (string
+  keys); `.symbolize_keys` for `**`, matching `setup_workflow!` and `rekick_dropped_jobs`. The
+  guard rejects `RESERVED_KWARGS`, which stored user kwargs never contain. Fresh `attempt: 0`
+  — a full workflow-retry budget, exactly like a normal resume.
+- **Per-row rescue** so one bad row (e.g. cross-version kwarg drift tripping the enqueue guard)
+  never aborts the sweep — mirrors `rekick_dropped_jobs` (`branch_merge_job.rb:265-270`).
+- **Returns the reaped count** for cron visibility; logs via `Rails.logger` block syntax.
+
+## Configuration
+
+`max_duration` (previously a hardcoded `10.minutes` in the executor) becomes a config value,
+and `reap_stale_after` **derives from it** so the "reap threshold must exceed the lock-steal
+threshold" invariant is automatic rather than documented-and-hoped.
+
+Add to `ChronoForge::Configuration`:
+
+```ruby
+# How long a single pass may hold its lock before it's stealable (feeds
+# LockStrategy.acquire_lock via the executor's #max_duration). Default 10 minutes.
+attr_accessor :max_duration
+
+# Age past which a :running workflow is considered stranded and re-enqueued by
+# Workflow.reap_stalled. Defaults to 3x max_duration (30 min out of the box) so it always
+# clears the lock-steal threshold; an explicit value overrides the derived default.
+def reap_stale_after
+  @reap_stale_after || max_duration * 3
+end
+attr_writer :reap_stale_after
+
+# in #initialize:
+@max_duration = 10.minutes
+@reap_stale_after = nil
+```
+
+The executor's `#max_duration` now reads `ChronoForge.config.max_duration` instead of
+returning a literal. Deriving `reap_stale_after` from `max_duration` means raising one raises
+the other, so the reaper can never be configured below the lock-steal threshold by accident.
+
+## Idempotency & Safety (documented behavior)
+
+- **Concurrency-safe.** Overlapping cron runs, or a re-enqueue landing while the old stale
+  lock still shows, at worst enqueue a duplicate job — the second loses the `acquire_lock`
+  race and no-ops via `ConcurrentExecutionError`. Mild, harmless churn. Recommend a cron
+  interval comfortably longer than a typical resume pass.
+- **Step idempotency.** Reaping replays the interrupted pass. A `durably_execute` step only
+  short-circuits when `completed`, so a step whose side effect committed but whose log never
+  reached `completed` will re-run on replay. Steps with external side effects must be
+  idempotent (natural/unique key + `create_or_find_by`/rescue). Documented alongside
+  `wait`/`durably_execute`.
+
+## Non-Goals (conscious boundaries)
+
+- **Dropped `BranchMergeJob` poller → parent stranded `idle`.** A parent parked on a merge
+  whose poller was hard-killed sits `idle`, not `running`, so this reaper won't catch it. A
+  distinct failure mode; the code already hints (`branch_merge_job.rb:181-184`) that a
+  "`next_poll_at` far in the past with work still pending" detector is the right fix. Out of
+  scope here.
+- **Micro-window between `release_lock` and `flush_continuation!`.** A process dying between
+  those two statements yields `idle` + no continuation. Theoretically possible, not seen in the
+  reported evidence (all strands were `running`), not covered.
+
+## Testing
+
+- **Query selection (unit):** selects only `running` + stale-lock rows; excludes
+  fresh-lock `running`, `idle`, `completed`, `failed`, `stalled`, and `NULL locked_at`.
+- **Strand-and-recover (behavioral):** create a workflow, force it into `running` with an old
+  `locked_at` and no scheduled resume job (simulating the hard-kill), run `reap_stalled`,
+  assert a job is enqueued and that draining the queue resumes and completes the workflow.
+  Cover both a top-level workflow and a branch child.
+- **Per-row rescue:** a row that raises on re-enqueue does not abort the sweep; the returned
+  count reflects only successful re-enqueues.
+- **Config:** default `reap_stale_after` honored; per-call `stale_after:` override honored.
+
+## Documentation
+
+- Document `Workflow.reap_stalled` and the recommended cron wiring (host-owned; no shipped
+  task), including the `reap_stale_after > max_duration` constraint.
+- Document the step-idempotency expectation alongside `wait`/`durably_execute`.
+
+## Alternatives Considered
+
+- **Railtie + `chrono_forge:reap` rake task** (angarium `angarium:reap` parity) — rejected;
+  heavier, would add the gem's first Railtie.
+- **Self-scheduling reaper ActiveJob** — rejected; adds an always-on job and scheduling
+  assumptions the host may not want.
+- **Top-level-only sweep** (`parent_execution_log_id IS NULL`) — rejected; leaves
+  mid-pass-killed branch children stranded, which rekick does not recover.

data/lib/chrono_forge/configuration.rb

@@ -18,8 +18,34 @@ module ChronoForge
     # runs promptly throughout the drain.
     attr_accessor :branch_merge_queue
 
+    # How long a single workflow pass may hold its lock before another job is
+    # allowed to steal it (LockStrategy.acquire_lock treats a lock older than this
+    # as stale). It bounds the assumed maximum duration of one execution pass.
+    # Defaults to 10 minutes.
+    attr_accessor :max_duration
+
     def initialize
       @branch_merge_queue = :default
+      @max_duration = 10.minutes
+      @reap_stale_after = nil
     end
+
+    # Age past which a workflow still in :running is treated as stranded and
+    # re-enqueued by ChronoForge::Workflow.reap_stalled. A workflow reaches this
+    # state when its worker is hard-killed (SIGKILL/OOM/eviction) mid-pass, before
+    # the executor's `ensure` block could release the lock and publish the resume
+    # continuation — so it stays locked in :running with nothing scheduled to wake it.
+    #
+    # Defaults to 3x max_duration (30 min out of the box), so it always comfortably
+    # exceeds the lock-steal threshold: acquire_lock only steals locks older than
+    # max_duration, so a shorter reap threshold would just enqueue resume jobs that
+    # immediately no-op via ConcurrentExecutionError. Deriving from max_duration keeps
+    # that invariant automatic — raise max_duration and the reaper backs off with it.
+    # An explicit value (set via the writer) overrides the derived default.
+    def reap_stale_after
+      @reap_stale_after || max_duration * 3
+    end
+
+    attr_writer :reap_stale_after
   end
 end

data/lib/chrono_forge/executor.rb

@@ -394,7 +394,7 @@ module ChronoForge
     end
 
     def max_duration
-      10.minutes
+      ChronoForge.config.max_duration
     end
   end
 end

data/lib/chrono_forge/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ChronoForge
-  VERSION = "0.11.0"
+  VERSION = "0.12.0"
 end

data/lib/chrono_forge/workflow.rb

@@ -45,6 +45,49 @@ module ChronoForge
       stalled
     ]
 
+    # Reconcile workflows stranded in :running by a hard-killed worker. When a
+    # worker is SIGKILLed (deploy/rollout, OOM, node eviction, SolidQueue heartbeat
+    # prune) mid-pass, the executor's `ensure` block never runs, so the lock is
+    # never released (the row stays :running with a stale locked_at) and the resume
+    # continuation is never published — nothing is left to wake the workflow. No
+    # other mechanism recovers this: workflow-level retry rides on the same process
+    # that must reach its `ensure`; retry_now/retry_later require stalled?/failed?;
+    # and BranchMergeJob rekick only re-drives never-started idle children.
+    #
+    # This sweeps every workflow in :running whose lock is older than `stale_after`
+    # (top-level AND branch children) and re-enqueues it. Re-enqueue is safe:
+    # acquire_lock steals the stale lock and completed durable steps replay as
+    # no-ops. Overlapping sweeps (or a re-enqueue landing while the old stale lock
+    # still shows) at worst enqueue a duplicate, which loses the acquire_lock race
+    # and no-ops via ConcurrentExecutionError.
+    #
+    # Intended to be run periodically by the host app (e.g. a SolidQueue recurring
+    # task or cron). Returns the number of workflows re-enqueued.
+    #
+    # NOTE: replaying an interrupted pass re-runs any durably_execute step whose
+    # side effect committed but whose log never reached :completed. Steps with
+    # external side effects must be idempotent (natural/unique key +
+    # create_or_find_by/rescue).
+    def self.reap_stalled(stale_after: ChronoForge.config.reap_stale_after)
+      reaped = 0
+      where(state: states[:running])
+        .where("locked_at < ?", stale_after.ago)
+        .find_each do |workflow|
+          # Guarded per row: one bad workflow (e.g. a since-deleted job class that
+          # no longer constantizes, or cross-version kwarg drift tripping the
+          # enqueue guard) must never abort the sweep and strand every healthy
+          # sibling. Mirrors BranchMergeJob#rekick_dropped_jobs.
+          workflow.job_klass.perform_later(workflow.key, **workflow.kwargs.symbolize_keys)
+          reaped += 1
+        rescue => e
+          Rails.logger.error do
+            "ChronoForge reap failed for workflow(#{workflow.key}): #{e.class}: #{e.message}"
+          end
+        end
+      Rails.logger.info { "ChronoForge reaped #{reaped} stalled workflow(s)" } if reaped.positive?
+      reaped
+    end
+
     def executable?
       idle? || running?
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: chrono_forge
 version: !ruby/object:Gem::Version
-  version: 0.11.0
+  version: 0.12.0
 platform: ruby
 authors:
 - Stefan Froelich
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2026-07-04 00:00:00.000000000 Z
+date: 2026-07-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -234,6 +234,7 @@ files:
 - docs/superpowers/specs/2026-06-26-dashboard-branch-view-design.md
 - docs/superpowers/specs/2026-06-26-deferral-continuation-race-and-catchup-design.md
 - docs/superpowers/specs/2026-07-01-workflow-definition-dag-design.md
+- docs/superpowers/specs/2026-07-09-chrono-forge-reaper-design.md
 - examples/continue_if_webhook_example.rb
 - gemfiles/rails_7.1.gemfile
 - gemfiles/rails_7.1.gemfile.lock