chrono_forge 0.10.0 → 0.12.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +44 -1
  3. data/README.md +246 -119
  4. data/Rakefile +4 -0
  5. data/cliff.toml +62 -0
  6. data/docs/design/per-child-commit-overhead.md +213 -0
  7. data/docs/fanout-scale-test.md +246 -0
  8. data/docs/superpowers/plans/2026-06-30-poller-rekick-and-eta-cadence.md +205 -0
  9. data/docs/superpowers/plans/2026-06-30-poller-rekick-and-eta-cadence.md.tasks.json +33 -0
  10. data/docs/superpowers/plans/2026-07-01-workflow-definition-dag.md +1373 -0
  11. data/docs/superpowers/plans/2026-07-01-workflow-definition-dag.md.tasks.json +68 -0
  12. data/docs/superpowers/specs/2026-07-01-workflow-definition-dag-design.md +203 -0
  13. data/docs/superpowers/specs/2026-07-09-chrono-forge-reaper-design.md +175 -0
  14. data/lib/chrono_forge/branch_merge_job.rb +158 -21
  15. data/lib/chrono_forge/branch_probe.rb +44 -0
  16. data/lib/chrono_forge/configuration.rb +51 -0
  17. data/lib/chrono_forge/definition.rb +37 -0
  18. data/lib/chrono_forge/definition_analyzer.rb +501 -0
  19. data/lib/chrono_forge/executor/context.rb +23 -0
  20. data/lib/chrono_forge/executor/lock_strategy.rb +10 -3
  21. data/lib/chrono_forge/executor/methods/continue_if.rb +15 -6
  22. data/lib/chrono_forge/executor/methods/durably_execute.rb +15 -7
  23. data/lib/chrono_forge/executor/methods/durably_repeat.rb +30 -14
  24. data/lib/chrono_forge/executor/methods/merge_branches.rb +5 -4
  25. data/lib/chrono_forge/executor/methods/workflow_states.rb +35 -47
  26. data/lib/chrono_forge/executor.rb +35 -10
  27. data/lib/chrono_forge/version.rb +1 -1
  28. data/lib/chrono_forge/workflow.rb +43 -0
  29. data/lib/chrono_forge.rb +8 -0
  30. data/lib/tasks/release.rake +212 -0
  31. metadata +29 -2
@@ -0,0 +1,68 @@
1
+ {
2
+ "planPath": "docs/superpowers/plans/2026-07-01-workflow-definition-dag.md",
3
+ "tasks": [
4
+ {
5
+ "id": 1,
6
+ "subject": "Task 1: Definition value objects + Prism dep",
7
+ "status": "completed",
8
+ "description": "Graph data model (Definition/Node/Edge) + prism runtime dependency, round-trippable to_h.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition.rb\", \"chrono_forge.gemspec\", \"test/definition_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_test.rb\", \"acceptanceCriteria\": [\"Definition holds nodes/edges/warnings; to_h JSON-safe\", \"Node#dynamic?\", \"prism declared dependency\"], \"requiresUserVerification\": false}\n```"
9
+ },
10
+ {
11
+ "id": 2,
12
+ "subject": "Task 2: Analyzer — linear steps",
13
+ "status": "completed",
14
+ "blockedBy": [1],
15
+ "description": "DefinitionAnalyzer.call resolves perform via Prism; a node per straight-line durable call with sequential edges from start.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"node per durable call in source order with exact step_name\", \"seq edges start->n1->n2\", \"non-durable Ruby ignored\", \"no DB/exec\"], \"requiresUserVerification\": false}\n```"
16
+ },
17
+ {
18
+ "id": 3,
19
+ "subject": "Task 3: Analyzer — conditionals & guards",
20
+ "status": "completed",
21
+ "blockedBy": [2],
22
+ "description": "if/unless/case around durable calls -> :conditional edges with guard labels; rejoin skip/body; continue_if false path -> :terminal.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"guarded conditional edge with source guard\", \"rejoin skip+body\", \"continue_if terminal edge\"], \"requiresUserVerification\": false}\n```"
23
+ },
24
+ {
25
+ "id": 4,
26
+ "subject": "Task 4: Analyzer — branch fan-out + merge join",
27
+ "status": "completed",
28
+ "blockedBy": [3],
29
+ "description": "branch block -> :branch fan-out node + child-group node via :fanout edge; merge_branches -> :join edge from branch.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"branch$name node + child-group with pattern\", \"branch->child :fanout\", \"branch->merge :join\"], \"requiresUserVerification\": false}\n```"
30
+ },
31
+ {
32
+ "id": 5,
33
+ "subject": "Task 5: Analyzer — repeat, helper tracing, loop warnings",
34
+ "status": "completed",
35
+ "blockedBy": [4],
36
+ "description": "durably_repeat -> :repeat node; trace durable calls in same-class helpers (recursion-guarded); durable call inside a loop -> warning, no crash.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"durably_repeat$tick single repeat node\", \"same-class helper traced in position\", \"loop-with-durable warns\", \"no infinite recursion\"], \"requiresUserVerification\": false}\n```"
37
+ },
38
+ {
39
+ "id": 6,
40
+ "subject": "Task 6: Dashboard — DefinitionOverlay",
41
+ "status": "completed",
42
+ "blockedBy": [1],
43
+ "description": "Annotate nodes with runtime status from execution_logs; fan-out/repeat aggregates via BranchProbe/rep logs; append unmapped nodes.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/app/presenters/chrono_forge/dashboard/definition_overlay.rb\", \"chrono_forge-dashboard/test/definition_overlay_test.rb\"], \"verifyCommand\": \"cd chrono_forge-dashboard && bundle exec rake test TEST=test/definition_overlay_test.rb\", \"acceptanceCriteria\": [\"exact-name status\", \"branch/merge counts\", \"repeat repetitions\", \"unmapped logs appended\"], \"requiresUserVerification\": false}\n```"
44
+ },
45
+ {
46
+ "id": 7,
47
+ "subject": "Task 7: Dashboard — MermaidRenderer",
48
+ "status": "completed",
49
+ "blockedBy": [6],
50
+ "description": "Statused nodes + edges -> Mermaid flowchart TD string with shapes by kind, :::status classes, guard edge labels, classDef lines.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/app/presenters/chrono_forge/dashboard/mermaid_renderer.rb\", \"chrono_forge-dashboard/test/mermaid_renderer_test.rb\"], \"verifyCommand\": \"cd chrono_forge-dashboard && bundle exec rake test TEST=test/mermaid_renderer_test.rb\", \"acceptanceCriteria\": [\"flowchart TD header\", \"node line per node with :::status\", \"edge lines with guard labels\", \"classDef per used status\"], \"requiresUserVerification\": false}\n```"
51
+ },
52
+ {
53
+ "id": 8,
54
+ "subject": "Task 8: Dashboard — definition page (route/controller/view/Mermaid/link)",
55
+ "status": "completed",
56
+ "blockedBy": [5, 6, 7],
57
+ "description": "GET workflows/:id/definition page — analyze class, overlay run, render Mermaid client-side (vendored), warnings panel, graceful degradation, link from detail page.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/config/routes.rb\", \"chrono_forge-dashboard/app/controllers/chrono_forge/dashboard/definitions_controller.rb\", \"chrono_forge-dashboard/app/views/chrono_forge/dashboard/definitions/show.html.erb\", \"chrono_forge-dashboard/app/assets/chrono_forge/dashboard/mermaid.min.js\", \"chrono_forge-dashboard/app/views/chrono_forge/dashboard/workflows/show.html.erb\", \"chrono_forge-dashboard/test/definitions_controller_test.rb\"], \"verifyCommand\": \"cd chrono_forge-dashboard && bundle exec rake test TEST=test/definitions_controller_test.rb\", \"acceptanceCriteria\": [\"200 with flowchart TD for analyzable wf\", \"unknown class degrades to warning not 500\", \"detail page links to page\"], \"requiresUserVerification\": false}\n```"
58
+ },
59
+ {
60
+ "id": 9,
61
+ "subject": "Task 9: Full suite + docs",
62
+ "status": "completed",
63
+ "blockedBy": [8],
64
+ "description": "Both packages green, lint clean, document the definition page.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/README.md\"], \"verifyCommand\": \"bundle exec rake test && cd chrono_forge-dashboard && bundle exec rake test\", \"acceptanceCriteria\": [\"core suite green\", \"dashboard suite green\", \"lint clean on new files\"], \"requiresUserVerification\": false}\n```"
65
+ }
66
+ ],
67
+ "lastUpdated": "2026-07-01T00:00:00Z"
68
+ }
@@ -0,0 +1,203 @@
1
+ # Workflow Definition DAG — static "future timeline" for ChronoForge
2
+
3
+ **Status:** Design approved (pending written-spec review)
4
+ **Date:** 2026-07-01
5
+ **Reference:** the `durable_flow` gem's `DefinitionAnalyzer` (Prism-based static analyzer + definition DAG overlaid with runtime status).
6
+
7
+ ## Problem
8
+
9
+ The dashboard today shows only the **historical** timeline of a workflow — the
10
+ `execution_logs` that have already run. There is no forward view: an operator
11
+ can't see the steps a workflow *will* run, where the current run sits in the
12
+ overall shape, or which branches/loops are still ahead.
13
+
14
+ ChronoForge workflows are plain Ruby: a `perform` method that the engine
15
+ **replays** every resume, with each durable step identified by a string name
16
+ (`durably_execute$name`, `wait_until$cond`, `branch$name`, `merge$a,b`,
17
+ `durably_repeat$name$<ts>`). Because the structure is expressed in source, we can
18
+ recover a *projection* of the step sequence by statically parsing `perform` with
19
+ Prism — without executing anything — and then paint the run's actual status onto
20
+ that static map.
21
+
22
+ ## Goal
23
+
24
+ A **new per-run dashboard page** that renders a workflow's **conditional DAG**
25
+ (the static definition graph) with the current run's `execution_logs` **overlaid**
26
+ as node status. The existing workflow detail page is unchanged; it gains a link
27
+ to this page.
28
+
29
+ Non-goals for v1 are listed under [Scope](#scope-v1).
30
+
31
+ ## Key decisions (locked during brainstorming)
32
+
33
+ 1. **Primary consumer:** dashboard overlay — run status painted on the static map
34
+ (mirrors durable_flow's run → definition-DAG view).
35
+ 2. **Map shape:** a **conditional DAG** — guarded edges for `if`/`continue_if`,
36
+ fan-out groups for branches, joins for merges.
37
+ 3. **Fidelity:** **conservative + trace same-class helper methods**. Resolve step
38
+ names statically where possible; anything unresolvable (computed `name:`,
39
+ data-dependent loop count, a durable call behind an unknown/external call)
40
+ becomes an explicit **`dynamic` node with a warning**. No unrolling, no
41
+ cross-class tracing.
42
+ 4. **Rendering:** **Mermaid.js** (client-side, vendored). The analyzer's graph
43
+ model is rendering-agnostic; a renderer emits Mermaid flowchart text with
44
+ status encoded as node classes.
45
+ 5. **Static vs runtime:** static Prism analysis is the source of the *shape*
46
+ (only it can show not-yet-run steps and untaken branches); the run log is the
47
+ *overlay*, never the source of the graph.
48
+ 6. **Placement:** a **new route/page**, not an inline addition to the detail page.
49
+
50
+ ## Architecture
51
+
52
+ ```
53
+ workflow_class
54
+ │ DefinitionAnalyzer.call (core gem; Prism; memoized by class + source digest)
55
+
56
+ Definition (Node[], Edge[], warnings) (plain, JSON-serializable value objects)
57
+ │ DefinitionOverlay(execution_logs) (dashboard; read-only queries; per-run; never cached)
58
+
59
+ statused Definition
60
+ │ MermaidRenderer (dashboard; statused graph → flowchart text)
61
+
62
+ new DAG page → vendored Mermaid JS renders client-side (inside data-poll-region)
63
+ ```
64
+
65
+ ### Core gem — `lib/chrono_forge/` (rendering-agnostic, no dashboard/DB dependency)
66
+
67
+ - **`ChronoForge::DefinitionAnalyzer`** — `.call(workflow_class) → Definition`.
68
+ - Resolves `workflow_class.instance_method(:perform).source_location`, reads the
69
+ file, `Prism.parse`, locates the `perform` def node, and walks its body with a
70
+ visitor.
71
+ - **Traces durable calls in same-class helper methods** to a fixed point within
72
+ the class (a call to a method defined on the same class whose body contains
73
+ durable DSL calls is expanded inline; recursion is guarded).
74
+ - Emits nodes, edges, and warnings. **Only reads source text — never touches the
75
+ DB, never executes workflow code.**
76
+ - **`ChronoForge::Definition`** (+ `Node`, `Edge`) — plain value objects,
77
+ JSON-serializable so a `Definition` can be cached.
78
+ - `Node`: `id`, `kind` ∈ `{:execute, :wait, :wait_until, :continue_if, :branch,
79
+ :merge, :repeat, :dynamic}`, `label`, and **either** an exact `step_name`
80
+ **or** a `step_name_pattern` (fan-out/repeat/dynamic), plus optional `guard`
81
+ (condition source label) and `warnings`.
82
+ - `Edge`: `from`, `to`, optional `guard` label, and a `kind` (`:seq`,
83
+ `:conditional`, `:fanout`, `:join`, `:terminal`).
84
+
85
+ ### Dashboard package — `chrono_forge-dashboard/`
86
+
87
+ - **`DefinitionOverlay`** — takes a `Definition` + a workflow's `execution_logs`
88
+ (and, for `:branch`/`:merge` nodes, child-workflow state counts via the existing
89
+ `BranchProbe`) and annotates each node with a runtime `status`. Read-only.
90
+ - **`MermaidRenderer`** — `statused Definition → flowchart text`; status encoded
91
+ as `classDef` + `class` assignments.
92
+ - **New controller action + view** — `GET workflows/:id/definition`, plus a
93
+ "Definition graph" link from the existing detail page.
94
+ - **Vendored Mermaid JS** — the dashboard's first client script, initialized
95
+ inside the existing `data-poll-region` so the DAG re-renders on the normal
96
+ page refresh.
97
+
98
+ ## Node → step-name binding
99
+
100
+ Each node knows the step-name it *would* produce, so the overlay is a lookup, not
101
+ guesswork:
102
+
103
+ | DSL call | Node kind | Binds to |
104
+ |---|---|---|
105
+ | `durably_execute :m` / `name: "x"` | `:execute` | exact `durably_execute$x` (or `$m`) |
106
+ | `durably_execute :m, name: <expr>` | `:dynamic` | prefix `durably_execute$`, by ordinal |
107
+ | `wait <duration>, "n"` | `:wait` | exact `wait$n` (name is the 2nd positional) |
108
+ | `wait_until :cond` | `:wait_until` | exact `wait_until$cond` |
109
+ | `continue_if :cond` | `:continue_if` | exact `continue_if$cond` |
110
+ | `branch :name { spawn/spawn_each }` | `:branch` (fan-out) | `branch$name` + child-workflow aggregate |
111
+ | `merge_branches :a, :b` | `:merge` (join) | `merge$a,b` (names sorted) |
112
+ | `durably_repeat :name` | `:repeat` (loop) | `durably_repeat$name` coord + `$<ts>` reps |
113
+
114
+ **Fan-out (`branch`/`spawn_each`) and `durably_repeat` collapse to a single node
115
+ with aggregate status** — not one node per child/iteration.
116
+
117
+ ## Overlay status vocabulary (→ Mermaid classes)
118
+
119
+ - `done` — matching log is `completed`.
120
+ - `active` — log is `started`/`running`, not completed.
121
+ - `pending` — reached but not done (a coordination log exists, work outstanding).
122
+ - `not_reached` — no log yet.
123
+ - `failed` / `stalled` — from the log state.
124
+ - `conditional` — statically guarded; may be skipped.
125
+ - `dynamic` — unresolved name; bound by prefix + ordinal.
126
+ - `unmapped` — **a runtime log with no matching static node**; appended so
127
+ analyzer gaps are surfaced, not hidden.
128
+
129
+ Aggregates:
130
+ - `:repeat` → "N done, current active, `till` met?" from the coordination log +
131
+ its `$<ts>` repetition logs.
132
+ - `:branch`/`:merge` → child-workflow state counts (running/idle/completed/failed)
133
+ via `BranchProbe`.
134
+
135
+ ## Edges & conditionals
136
+
137
+ - Sequential DSL calls → `:seq` edges.
138
+ - `if`/`unless`/`case`/`&&`/`||`/early-return around a step → `:conditional` edge
139
+ labeled with the condition source; steps only reachable under a guard render
140
+ `conditional`.
141
+ - `continue_if` → a gate node; its false path is a `:terminal` edge (workflow
142
+ halts).
143
+ - `branch` block → fans out (`:fanout`) to its spawn/`spawn_each` child-group;
144
+ `merge_branches` is the `:join` those edges reconnect into.
145
+ - `each`/`times`/`while` containing durable calls → one node + a "dynamic loop
146
+ count" **warning** (conservative — no unrolling).
147
+
148
+ ## Error handling
149
+
150
+ The analyzer must never break the dashboard:
151
+
152
+ - Source unavailable (`source_location` nil, C-defined, `eval`'d, unreadable
153
+ file) → return a `Definition` carrying a single `unavailable` warning; the page
154
+ renders "can't be statically analyzed" gracefully. Never raises.
155
+ - Any Prism parse issue degrades the same way (Prism is error-tolerant).
156
+ - Missing/unloadable `job_class`, or a partially-resolved analysis → render what
157
+ was found plus a warnings panel.
158
+ - **Analyzer is pure/read-only over source text**; the overlay does read-only
159
+ queries only.
160
+
161
+ ## Caching
162
+
163
+ - Memoize `Definition` by `job_class` + source-file digest — auto-invalidates on
164
+ dev code reload, stable in prod.
165
+ - The **overlay is never cached** — it is per-run and changes every poll.
166
+
167
+ ## Testing
168
+
169
+ - **Analyzer unit tests (no DB):** a fixture set of workflow classes — linear,
170
+ conditional/`continue_if`, `branch`+`spawn_each`, `durably_repeat`, dynamic
171
+ `name:`, helper-traced, unanalyzable loop — asserting node kinds, edges, guards,
172
+ and warnings. Deterministic and fast.
173
+ - **Overlay tests (dashboard harness):** seed `execution_logs` + child workflows;
174
+ assert per-node status, fan-out aggregates, repeat counts, and the `unmapped`
175
+ path.
176
+ - **`MermaidRenderer`:** golden-text tests (statused `Definition` → expected
177
+ flowchart string).
178
+
179
+ ## Scope (v1)
180
+
181
+ **In:** all seven primitives as nodes + conditional edges + fan-out/repeat
182
+ aggregation + the overlay + the new per-run DAG page + Mermaid rendering;
183
+ same-class helper tracing.
184
+
185
+ **Out (deferred):**
186
+ - Cross-class helper tracing.
187
+ - Recursively expanding a spawned child *workflow class* into its own graph
188
+ (v1 shows it as one fan-out node; "drill into child" is a future feature).
189
+ - Per-node ETA/timing beyond status + counts (that's the separate progress/ETA
190
+ feature).
191
+ - A class-level (no-overlay) definition view (trivial later addition).
192
+
193
+ ## Open questions / risks
194
+
195
+ - **Helper-tracing fixed point:** need a clear rule for what counts as "a durable
196
+ call inside a same-class method" vs. ordinary work, and recursion/mutual-call
197
+ guards. The analyzer stays conservative — when in doubt, emit a `dynamic` node +
198
+ warning rather than a confident-but-wrong expansion.
199
+ - **Ordinal binding for dynamic siblings** is best-effort; if two dynamic
200
+ `durably_execute` calls interleave at runtime out of source order, the overlay
201
+ may mis-bind. Acceptable for v1 (surfaced as `dynamic`).
202
+ - **Mermaid as first client dependency** — keep it vendored and isolated so the
203
+ rest of the dashboard stays server-rendered.
@@ -0,0 +1,175 @@
1
+ # Design: Stalled-Workflow Reaper for ChronoForge
2
+
3
+ **Date:** 2026-07-09
4
+ **Gem:** `chrono_forge` (with `chrono_forge-dashboard`)
5
+ **Status:** Approved for planning
6
+
7
+ ## Problem
8
+
9
+ A workflow that is mid-pass when its worker is **hard-killed** (SIGKILL: deploy/rollout,
10
+ OOM, node eviction, SolidQueue heartbeat prune) is stranded permanently in `state: running`
11
+ with a stale lock, and nothing ever resumes it.
12
+
13
+ The mechanism, confirmed against the code:
14
+
15
+ - `Executor#perform` releases the lock and publishes the resume continuation in its `ensure`
16
+ block (`lib/chrono_forge/executor.rb:178-197`): `release_lock` first, then
17
+ `flush_continuation!`.
18
+ - `enqueue_continuation` (`executor.rb:369-371`) only records `@continuation` in memory;
19
+ `flush_continuation!` (`executor.rb:376-382`) is what actually enqueues the ActiveJob.
20
+ - On SIGKILL, Ruby's `ensure` does not run. So (1) `release_lock` never runs — the row stays
21
+ `state: running` with a past `locked_at`; and (2) `flush_continuation!` never runs — no job
22
+ is scheduled to resume the workflow.
23
+
24
+ The workflow is fully *resumable* — `Workflow#executable?` is `idle? || running?`
25
+ (`workflow.rb:48-50`) and `LockStrategy.acquire_lock` steals any lock older than
26
+ `max_duration` (10 min) (`lock_strategy.rb:17-21`) — but nothing re-enqueues it. Existing
27
+ recovery mechanisms do not cover this:
28
+
29
+ - **`BranchMergeJob` rekick** (`branch_merge_job.rb:232-273`) only re-enqueues children that
30
+ are `state: idle, started_at: nil` (never-started dropped dispatches). A child hard-killed
31
+ **mid-pass** is `state: running` with `started_at` set — explicitly excluded (see the
32
+ comment at `branch_merge_job.rb:226-231`). So branch children have the identical gap.
33
+ - **Workflow-level retry** rides on the same process that must survive to its `ensure`; a
34
+ killed process never reaches it.
35
+ - **`retry_now`/`retry_later`** require `retryable?` (`stalled? || failed?`,
36
+ `workflow.rb:53-55`); a stranded workflow is `running`, so they refuse it.
37
+ - **Dashboard "Force unlock"** clears the lock but enqueues nothing.
38
+
39
+ Real-world impact (reported): 735 workflows stranded over ~8.5 months in one production app,
40
+ still accruing ~8/day.
41
+
42
+ ## Solution Overview
43
+
44
+ Add a reconciliation entry point, `ChronoForge::Workflow.reap_stalled`, that finds workflows
45
+ stuck in `running` past a safety threshold and re-enqueues them. Re-enqueue is safe:
46
+ `acquire_lock` steals the stale lock and completed durable steps replay as no-ops.
47
+
48
+ ### Decisions
49
+
50
+ | Decision | Choice | Rationale |
51
+ |---|---|---|
52
+ | Shipping | **Class method only** | No Railtie/rake task/self-scheduling job. Host apps call it from their own cron/recurring job. Smallest surface; no new gem infrastructure. |
53
+ | Scope | **All stranded `running` workflows** (top-level AND branch children) | Strictly more correct — rekick misses mid-pass-killed children. Re-enqueue of a child directly is the proven-safe pattern rekick already uses. |
54
+ | Threshold | **Config + per-call override** | `ChronoForge.config.reap_stale_after` (default `30.minutes`), overridable via `stale_after:` arg. Parity with the existing `branch_merge_queue` config. |
55
+ | Deliverable | **Code + tests + docs** | Implementation, tests (incl. simulated hard-kill strand), and docs covering the reaper + step-idempotency expectations. |
56
+
57
+ ## Component
58
+
59
+ `ChronoForge::Workflow.reap_stalled` — a class method on the AR model. It only queries and
60
+ re-enqueues; it holds no lock and performs no replay itself (the re-enqueued job does the
61
+ replay, under the normal lock protocol).
62
+
63
+ ```ruby
64
+ # ChronoForge::Workflow
65
+ def self.reap_stalled(stale_after: ChronoForge.config.reap_stale_after)
66
+ reaped = 0
67
+ where(state: states[:running])
68
+ .where("locked_at < ?", stale_after.ago)
69
+ .find_each do |wf|
70
+ wf.job_klass.perform_later(wf.key, **wf.kwargs.symbolize_keys)
71
+ reaped += 1
72
+ rescue => e
73
+ Rails.logger.error { "ChronoForge reap failed for workflow(#{wf.key}): #{e.class}: #{e.message}" }
74
+ end
75
+ Rails.logger.info { "ChronoForge reaped #{reaped} stalled workflow(s)" } if reaped.positive?
76
+ reaped
77
+ end
78
+ ```
79
+
80
+ Design points:
81
+
82
+ - **Target = `running` + stale `locked_at`.** The strand happens precisely because the
83
+ `ensure` never ran, so `release_lock` never flipped the row off `running`. A `NULL locked_at`
84
+ cannot match `locked_at < ?` and is naturally excluded (an anomalous running-without-lock row
85
+ is not our target).
86
+ - **Sweeps children too.** No `parent_execution_log_id` filter. A re-enqueued child runs,
87
+ replays, completes, and updates its `BranchProbe` state so the parent's merge poller observes
88
+ completion.
89
+ - **Re-enqueue via the public guarded `perform_later`.** `wf.kwargs` is a JSON column (string
90
+ keys); `.symbolize_keys` for `**`, matching `setup_workflow!` and `rekick_dropped_jobs`. The
91
+ guard rejects `RESERVED_KWARGS`, which stored user kwargs never contain. Fresh `attempt: 0`
92
+ — a full workflow-retry budget, exactly like a normal resume.
93
+ - **Per-row rescue** so one bad row (e.g. cross-version kwarg drift tripping the enqueue guard)
94
+ never aborts the sweep — mirrors `rekick_dropped_jobs` (`branch_merge_job.rb:265-270`).
95
+ - **Returns the reaped count** for cron visibility; logs via `Rails.logger` block syntax.
96
+
97
+ ## Configuration
98
+
99
+ `max_duration` (previously a hardcoded `10.minutes` in the executor) becomes a config value,
100
+ and `reap_stale_after` **derives from it** so the "reap threshold must exceed the lock-steal
101
+ threshold" invariant is automatic rather than documented-and-hoped.
102
+
103
+ Add to `ChronoForge::Configuration`:
104
+
105
+ ```ruby
106
+ # How long a single pass may hold its lock before it's stealable (feeds
107
+ # LockStrategy.acquire_lock via the executor's #max_duration). Default 10 minutes.
108
+ attr_accessor :max_duration
109
+
110
+ # Age past which a :running workflow is considered stranded and re-enqueued by
111
+ # Workflow.reap_stalled. Defaults to 3x max_duration (30 min out of the box) so it always
112
+ # clears the lock-steal threshold; an explicit value overrides the derived default.
113
+ def reap_stale_after
114
+ @reap_stale_after || max_duration * 3
115
+ end
116
+ attr_writer :reap_stale_after
117
+
118
+ # in #initialize:
119
+ @max_duration = 10.minutes
120
+ @reap_stale_after = nil
121
+ ```
122
+
123
+ The executor's `#max_duration` now reads `ChronoForge.config.max_duration` instead of
124
+ returning a literal. Deriving `reap_stale_after` from `max_duration` means raising one raises
125
+ the other, so the reaper can never be configured below the lock-steal threshold by accident.
126
+
127
+ ## Idempotency & Safety (documented behavior)
128
+
129
+ - **Concurrency-safe.** Overlapping cron runs, or a re-enqueue landing while the old stale
130
+ lock still shows, at worst enqueue a duplicate job — the second loses the `acquire_lock`
131
+ race and no-ops via `ConcurrentExecutionError`. Mild, harmless churn. Recommend a cron
132
+ interval comfortably longer than a typical resume pass.
133
+ - **Step idempotency.** Reaping replays the interrupted pass. A `durably_execute` step only
134
+ short-circuits when `completed`, so a step whose side effect committed but whose log never
135
+ reached `completed` will re-run on replay. Steps with external side effects must be
136
+ idempotent (natural/unique key + `create_or_find_by`/rescue). Documented alongside
137
+ `wait`/`durably_execute`.
138
+
139
+ ## Non-Goals (conscious boundaries)
140
+
141
+ - **Dropped `BranchMergeJob` poller → parent stranded `idle`.** A parent parked on a merge
142
+ whose poller was hard-killed sits `idle`, not `running`, so this reaper won't catch it. A
143
+ distinct failure mode; the code already hints (`branch_merge_job.rb:181-184`) that a
144
+ "`next_poll_at` far in the past with work still pending" detector is the right fix. Out of
145
+ scope here.
146
+ - **Micro-window between `release_lock` and `flush_continuation!`.** A process dying between
147
+ those two statements yields `idle` + no continuation. Theoretically possible, not seen in the
148
+ reported evidence (all strands were `running`), not covered.
149
+
150
+ ## Testing
151
+
152
+ - **Query selection (unit):** selects only `running` + stale-lock rows; excludes
153
+ fresh-lock `running`, `idle`, `completed`, `failed`, `stalled`, and `NULL locked_at`.
154
+ - **Strand-and-recover (behavioral):** create a workflow, force it into `running` with an old
155
+ `locked_at` and no scheduled resume job (simulating the hard-kill), run `reap_stalled`,
156
+ assert a job is enqueued and that draining the queue resumes and completes the workflow.
157
+ Cover both a top-level workflow and a branch child.
158
+ - **Per-row rescue:** a row that raises on re-enqueue does not abort the sweep; the returned
159
+ count reflects only successful re-enqueues.
160
+ - **Config:** default `reap_stale_after` honored; per-call `stale_after:` override honored.
161
+
162
+ ## Documentation
163
+
164
+ - Document `Workflow.reap_stalled` and the recommended cron wiring (host-owned; no shipped
165
+ task), including the `reap_stale_after > max_duration` constraint.
166
+ - Document the step-idempotency expectation alongside `wait`/`durably_execute`.
167
+
168
+ ## Alternatives Considered
169
+
170
+ - **Railtie + `chrono_forge:reap` rake task** (angarium `angarium:reap` parity) — rejected;
171
+ heavier, would add the gem's first Railtie.
172
+ - **Self-scheduling reaper ActiveJob** — rejected; adds an always-on job and scheduling
173
+ assumptions the host may not want.
174
+ - **Top-level-only sweep** (`parent_execution_log_id IS NULL`) — rejected; leaves
175
+ mid-pass-killed branch children stranded, which rekick does not recover.