chrono_forge 0.10.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +44 -1
- data/README.md +246 -119
- data/Rakefile +4 -0
- data/cliff.toml +62 -0
- data/docs/design/per-child-commit-overhead.md +213 -0
- data/docs/fanout-scale-test.md +246 -0
- data/docs/superpowers/plans/2026-06-30-poller-rekick-and-eta-cadence.md +205 -0
- data/docs/superpowers/plans/2026-06-30-poller-rekick-and-eta-cadence.md.tasks.json +33 -0
- data/docs/superpowers/plans/2026-07-01-workflow-definition-dag.md +1373 -0
- data/docs/superpowers/plans/2026-07-01-workflow-definition-dag.md.tasks.json +68 -0
- data/docs/superpowers/specs/2026-07-01-workflow-definition-dag-design.md +203 -0
- data/docs/superpowers/specs/2026-07-09-chrono-forge-reaper-design.md +175 -0
- data/lib/chrono_forge/branch_merge_job.rb +158 -21
- data/lib/chrono_forge/branch_probe.rb +44 -0
- data/lib/chrono_forge/configuration.rb +51 -0
- data/lib/chrono_forge/definition.rb +37 -0
- data/lib/chrono_forge/definition_analyzer.rb +501 -0
- data/lib/chrono_forge/executor/context.rb +23 -0
- data/lib/chrono_forge/executor/lock_strategy.rb +10 -3
- data/lib/chrono_forge/executor/methods/continue_if.rb +15 -6
- data/lib/chrono_forge/executor/methods/durably_execute.rb +15 -7
- data/lib/chrono_forge/executor/methods/durably_repeat.rb +30 -14
- data/lib/chrono_forge/executor/methods/merge_branches.rb +5 -4
- data/lib/chrono_forge/executor/methods/workflow_states.rb +35 -47
- data/lib/chrono_forge/executor.rb +35 -10
- data/lib/chrono_forge/version.rb +1 -1
- data/lib/chrono_forge/workflow.rb +43 -0
- data/lib/chrono_forge.rb +8 -0
- data/lib/tasks/release.rake +212 -0
- metadata +29 -2
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
{
|
|
2
|
+
"planPath": "docs/superpowers/plans/2026-07-01-workflow-definition-dag.md",
|
|
3
|
+
"tasks": [
|
|
4
|
+
{
|
|
5
|
+
"id": 1,
|
|
6
|
+
"subject": "Task 1: Definition value objects + Prism dep",
|
|
7
|
+
"status": "completed",
|
|
8
|
+
"description": "Graph data model (Definition/Node/Edge) + prism runtime dependency, round-trippable to_h.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition.rb\", \"chrono_forge.gemspec\", \"test/definition_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_test.rb\", \"acceptanceCriteria\": [\"Definition holds nodes/edges/warnings; to_h JSON-safe\", \"Node#dynamic?\", \"prism declared dependency\"], \"requiresUserVerification\": false}\n```"
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"id": 2,
|
|
12
|
+
"subject": "Task 2: Analyzer — linear steps",
|
|
13
|
+
"status": "completed",
|
|
14
|
+
"blockedBy": [1],
|
|
15
|
+
"description": "DefinitionAnalyzer.call resolves perform via Prism; a node per straight-line durable call with sequential edges from start.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"node per durable call in source order with exact step_name\", \"seq edges start->n1->n2\", \"non-durable Ruby ignored\", \"no DB/exec\"], \"requiresUserVerification\": false}\n```"
|
|
16
|
+
},
|
|
17
|
+
{
|
|
18
|
+
"id": 3,
|
|
19
|
+
"subject": "Task 3: Analyzer — conditionals & guards",
|
|
20
|
+
"status": "completed",
|
|
21
|
+
"blockedBy": [2],
|
|
22
|
+
"description": "if/unless/case around durable calls -> :conditional edges with guard labels; rejoin skip/body; continue_if false path -> :terminal.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"guarded conditional edge with source guard\", \"rejoin skip+body\", \"continue_if terminal edge\"], \"requiresUserVerification\": false}\n```"
|
|
23
|
+
},
|
|
24
|
+
{
|
|
25
|
+
"id": 4,
|
|
26
|
+
"subject": "Task 4: Analyzer — branch fan-out + merge join",
|
|
27
|
+
"status": "completed",
|
|
28
|
+
"blockedBy": [3],
|
|
29
|
+
"description": "branch block -> :branch fan-out node + child-group node via :fanout edge; merge_branches -> :join edge from branch.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"branch$name node + child-group with pattern\", \"branch->child :fanout\", \"branch->merge :join\"], \"requiresUserVerification\": false}\n```"
|
|
30
|
+
},
|
|
31
|
+
{
|
|
32
|
+
"id": 5,
|
|
33
|
+
"subject": "Task 5: Analyzer — repeat, helper tracing, loop warnings",
|
|
34
|
+
"status": "completed",
|
|
35
|
+
"blockedBy": [4],
|
|
36
|
+
"description": "durably_repeat -> :repeat node; trace durable calls in same-class helpers (recursion-guarded); durable call inside a loop -> warning, no crash.\n\n```json:metadata\n{\"files\": [\"lib/chrono_forge/definition_analyzer.rb\", \"test/support/definition_fixtures.rb\", \"test/definition_analyzer_test.rb\"], \"verifyCommand\": \"bundle exec ruby -I test test/definition_analyzer_test.rb\", \"acceptanceCriteria\": [\"durably_repeat$tick single repeat node\", \"same-class helper traced in position\", \"loop-with-durable warns\", \"no infinite recursion\"], \"requiresUserVerification\": false}\n```"
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"id": 6,
|
|
40
|
+
"subject": "Task 6: Dashboard — DefinitionOverlay",
|
|
41
|
+
"status": "completed",
|
|
42
|
+
"blockedBy": [1],
|
|
43
|
+
"description": "Annotate nodes with runtime status from execution_logs; fan-out/repeat aggregates via BranchProbe/rep logs; append unmapped nodes.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/app/presenters/chrono_forge/dashboard/definition_overlay.rb\", \"chrono_forge-dashboard/test/definition_overlay_test.rb\"], \"verifyCommand\": \"cd chrono_forge-dashboard && bundle exec rake test TEST=test/definition_overlay_test.rb\", \"acceptanceCriteria\": [\"exact-name status\", \"branch/merge counts\", \"repeat repetitions\", \"unmapped logs appended\"], \"requiresUserVerification\": false}\n```"
|
|
44
|
+
},
|
|
45
|
+
{
|
|
46
|
+
"id": 7,
|
|
47
|
+
"subject": "Task 7: Dashboard — MermaidRenderer",
|
|
48
|
+
"status": "completed",
|
|
49
|
+
"blockedBy": [6],
|
|
50
|
+
"description": "Statused nodes + edges -> Mermaid flowchart TD string with shapes by kind, :::status classes, guard edge labels, classDef lines.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/app/presenters/chrono_forge/dashboard/mermaid_renderer.rb\", \"chrono_forge-dashboard/test/mermaid_renderer_test.rb\"], \"verifyCommand\": \"cd chrono_forge-dashboard && bundle exec rake test TEST=test/mermaid_renderer_test.rb\", \"acceptanceCriteria\": [\"flowchart TD header\", \"node line per node with :::status\", \"edge lines with guard labels\", \"classDef per used status\"], \"requiresUserVerification\": false}\n```"
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"id": 8,
|
|
54
|
+
"subject": "Task 8: Dashboard — definition page (route/controller/view/Mermaid/link)",
|
|
55
|
+
"status": "completed",
|
|
56
|
+
"blockedBy": [5, 6, 7],
|
|
57
|
+
"description": "GET workflows/:id/definition page — analyze class, overlay run, render Mermaid client-side (vendored), warnings panel, graceful degradation, link from detail page.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/config/routes.rb\", \"chrono_forge-dashboard/app/controllers/chrono_forge/dashboard/definitions_controller.rb\", \"chrono_forge-dashboard/app/views/chrono_forge/dashboard/definitions/show.html.erb\", \"chrono_forge-dashboard/app/assets/chrono_forge/dashboard/mermaid.min.js\", \"chrono_forge-dashboard/app/views/chrono_forge/dashboard/workflows/show.html.erb\", \"chrono_forge-dashboard/test/definitions_controller_test.rb\"], \"verifyCommand\": \"cd chrono_forge-dashboard && bundle exec rake test TEST=test/definitions_controller_test.rb\", \"acceptanceCriteria\": [\"200 with flowchart TD for analyzable wf\", \"unknown class degrades to warning not 500\", \"detail page links to page\"], \"requiresUserVerification\": false}\n```"
|
|
58
|
+
},
|
|
59
|
+
{
|
|
60
|
+
"id": 9,
|
|
61
|
+
"subject": "Task 9: Full suite + docs",
|
|
62
|
+
"status": "completed",
|
|
63
|
+
"blockedBy": [8],
|
|
64
|
+
"description": "Both packages green, lint clean, document the definition page.\n\n```json:metadata\n{\"files\": [\"chrono_forge-dashboard/README.md\"], \"verifyCommand\": \"bundle exec rake test && cd chrono_forge-dashboard && bundle exec rake test\", \"acceptanceCriteria\": [\"core suite green\", \"dashboard suite green\", \"lint clean on new files\"], \"requiresUserVerification\": false}\n```"
|
|
65
|
+
}
|
|
66
|
+
],
|
|
67
|
+
"lastUpdated": "2026-07-01T00:00:00Z"
|
|
68
|
+
}
|
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# Workflow Definition DAG — static "future timeline" for ChronoForge
|
|
2
|
+
|
|
3
|
+
**Status:** Design approved (pending written-spec review)
|
|
4
|
+
**Date:** 2026-07-01
|
|
5
|
+
**Reference:** the `durable_flow` gem's `DefinitionAnalyzer` (Prism-based static analyzer + definition DAG overlaid with runtime status).
|
|
6
|
+
|
|
7
|
+
## Problem
|
|
8
|
+
|
|
9
|
+
The dashboard today shows only the **historical** timeline of a workflow — the
|
|
10
|
+
`execution_logs` that have already run. There is no forward view: an operator
|
|
11
|
+
can't see the steps a workflow *will* run, where the current run sits in the
|
|
12
|
+
overall shape, or which branches/loops are still ahead.
|
|
13
|
+
|
|
14
|
+
ChronoForge workflows are plain Ruby: a `perform` method that the engine
|
|
15
|
+
**replays** every resume, with each durable step identified by a string name
|
|
16
|
+
(`durably_execute$name`, `wait_until$cond`, `branch$name`, `merge$a,b`,
|
|
17
|
+
`durably_repeat$name$<ts>`). Because the structure is expressed in source, we can
|
|
18
|
+
recover a *projection* of the step sequence by statically parsing `perform` with
|
|
19
|
+
Prism — without executing anything — and then paint the run's actual status onto
|
|
20
|
+
that static map.
|
|
21
|
+
|
|
22
|
+
## Goal
|
|
23
|
+
|
|
24
|
+
A **new per-run dashboard page** that renders a workflow's **conditional DAG**
|
|
25
|
+
(the static definition graph) with the current run's `execution_logs` **overlaid**
|
|
26
|
+
as node status. The existing workflow detail page is unchanged; it gains a link
|
|
27
|
+
to this page.
|
|
28
|
+
|
|
29
|
+
Non-goals for v1 are listed under [Scope](#scope-v1).
|
|
30
|
+
|
|
31
|
+
## Key decisions (locked during brainstorming)
|
|
32
|
+
|
|
33
|
+
1. **Primary consumer:** dashboard overlay — run status painted on the static map
|
|
34
|
+
(mirrors durable_flow's run → definition-DAG view).
|
|
35
|
+
2. **Map shape:** a **conditional DAG** — guarded edges for `if`/`continue_if`,
|
|
36
|
+
fan-out groups for branches, joins for merges.
|
|
37
|
+
3. **Fidelity:** **conservative + trace same-class helper methods**. Resolve step
|
|
38
|
+
names statically where possible; anything unresolvable (computed `name:`,
|
|
39
|
+
data-dependent loop count, a durable call behind an unknown/external call)
|
|
40
|
+
becomes an explicit **`dynamic` node with a warning**. No unrolling, no
|
|
41
|
+
cross-class tracing.
|
|
42
|
+
4. **Rendering:** **Mermaid.js** (client-side, vendored). The analyzer's graph
|
|
43
|
+
model is rendering-agnostic; a renderer emits Mermaid flowchart text with
|
|
44
|
+
status encoded as node classes.
|
|
45
|
+
5. **Static vs runtime:** static Prism analysis is the source of the *shape*
|
|
46
|
+
(only it can show not-yet-run steps and untaken branches); the run log is the
|
|
47
|
+
*overlay*, never the source of the graph.
|
|
48
|
+
6. **Placement:** a **new route/page**, not an inline addition to the detail page.
|
|
49
|
+
|
|
50
|
+
## Architecture
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
workflow_class
|
|
54
|
+
│ DefinitionAnalyzer.call (core gem; Prism; memoized by class + source digest)
|
|
55
|
+
▼
|
|
56
|
+
Definition (Node[], Edge[], warnings) (plain, JSON-serializable value objects)
|
|
57
|
+
│ DefinitionOverlay(execution_logs) (dashboard; read-only queries; per-run; never cached)
|
|
58
|
+
▼
|
|
59
|
+
statused Definition
|
|
60
|
+
│ MermaidRenderer (dashboard; statused graph → flowchart text)
|
|
61
|
+
▼
|
|
62
|
+
new DAG page → vendored Mermaid JS renders client-side (inside data-poll-region)
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
### Core gem — `lib/chrono_forge/` (rendering-agnostic, no dashboard/DB dependency)
|
|
66
|
+
|
|
67
|
+
- **`ChronoForge::DefinitionAnalyzer`** — `.call(workflow_class) → Definition`.
|
|
68
|
+
- Resolves `workflow_class.instance_method(:perform).source_location`, reads the
|
|
69
|
+
file, `Prism.parse`, locates the `perform` def node, and walks its body with a
|
|
70
|
+
visitor.
|
|
71
|
+
- **Traces durable calls in same-class helper methods** to a fixed point within
|
|
72
|
+
the class (a call to a method defined on the same class whose body contains
|
|
73
|
+
durable DSL calls is expanded inline; recursion is guarded).
|
|
74
|
+
- Emits nodes, edges, and warnings. **Only reads source text — never touches the
|
|
75
|
+
DB, never executes workflow code.**
|
|
76
|
+
- **`ChronoForge::Definition`** (+ `Node`, `Edge`) — plain value objects,
|
|
77
|
+
JSON-serializable so a `Definition` can be cached.
|
|
78
|
+
- `Node`: `id`, `kind` ∈ `{:execute, :wait, :wait_until, :continue_if, :branch,
|
|
79
|
+
:merge, :repeat, :dynamic}`, `label`, and **either** an exact `step_name`
|
|
80
|
+
**or** a `step_name_pattern` (fan-out/repeat/dynamic), plus optional `guard`
|
|
81
|
+
(condition source label) and `warnings`.
|
|
82
|
+
- `Edge`: `from`, `to`, optional `guard` label, and a `kind` (`:seq`,
|
|
83
|
+
`:conditional`, `:fanout`, `:join`, `:terminal`).
|
|
84
|
+
|
|
85
|
+
### Dashboard package — `chrono_forge-dashboard/`
|
|
86
|
+
|
|
87
|
+
- **`DefinitionOverlay`** — takes a `Definition` + a workflow's `execution_logs`
|
|
88
|
+
(and, for `:branch`/`:merge` nodes, child-workflow state counts via the existing
|
|
89
|
+
`BranchProbe`) and annotates each node with a runtime `status`. Read-only.
|
|
90
|
+
- **`MermaidRenderer`** — `statused Definition → flowchart text`; status encoded
|
|
91
|
+
as `classDef` + `class` assignments.
|
|
92
|
+
- **New controller action + view** — `GET workflows/:id/definition`, plus a
|
|
93
|
+
"Definition graph" link from the existing detail page.
|
|
94
|
+
- **Vendored Mermaid JS** — the dashboard's first client script, initialized
|
|
95
|
+
inside the existing `data-poll-region` so the DAG re-renders on the normal
|
|
96
|
+
page refresh.
|
|
97
|
+
|
|
98
|
+
## Node → step-name binding
|
|
99
|
+
|
|
100
|
+
Each node knows the step-name it *would* produce, so the overlay is a lookup, not
|
|
101
|
+
guesswork:
|
|
102
|
+
|
|
103
|
+
| DSL call | Node kind | Binds to |
|
|
104
|
+
|---|---|---|
|
|
105
|
+
| `durably_execute :m` / `name: "x"` | `:execute` | exact `durably_execute$x` (or `$m`) |
|
|
106
|
+
| `durably_execute :m, name: <expr>` | `:dynamic` | prefix `durably_execute$`, by ordinal |
|
|
107
|
+
| `wait <duration>, "n"` | `:wait` | exact `wait$n` (name is the 2nd positional) |
|
|
108
|
+
| `wait_until :cond` | `:wait_until` | exact `wait_until$cond` |
|
|
109
|
+
| `continue_if :cond` | `:continue_if` | exact `continue_if$cond` |
|
|
110
|
+
| `branch :name { spawn/spawn_each }` | `:branch` (fan-out) | `branch$name` + child-workflow aggregate |
|
|
111
|
+
| `merge_branches :a, :b` | `:merge` (join) | `merge$a,b` (names sorted) |
|
|
112
|
+
| `durably_repeat :name` | `:repeat` (loop) | `durably_repeat$name` coord + `$<ts>` reps |
|
|
113
|
+
|
|
114
|
+
**Fan-out (`branch`/`spawn_each`) and `durably_repeat` collapse to a single node
|
|
115
|
+
with aggregate status** — not one node per child/iteration.
|
|
116
|
+
|
|
117
|
+
## Overlay status vocabulary (→ Mermaid classes)
|
|
118
|
+
|
|
119
|
+
- `done` — matching log is `completed`.
|
|
120
|
+
- `active` — log is `started`/`running`, not completed.
|
|
121
|
+
- `pending` — reached but not done (a coordination log exists, work outstanding).
|
|
122
|
+
- `not_reached` — no log yet.
|
|
123
|
+
- `failed` / `stalled` — from the log state.
|
|
124
|
+
- `conditional` — statically guarded; may be skipped.
|
|
125
|
+
- `dynamic` — unresolved name; bound by prefix + ordinal.
|
|
126
|
+
- `unmapped` — **a runtime log with no matching static node**; appended so
|
|
127
|
+
analyzer gaps are surfaced, not hidden.
|
|
128
|
+
|
|
129
|
+
Aggregates:
|
|
130
|
+
- `:repeat` → "N done, current active, `till` met?" from the coordination log +
|
|
131
|
+
its `$<ts>` repetition logs.
|
|
132
|
+
- `:branch`/`:merge` → child-workflow state counts (running/idle/completed/failed)
|
|
133
|
+
via `BranchProbe`.
|
|
134
|
+
|
|
135
|
+
## Edges & conditionals
|
|
136
|
+
|
|
137
|
+
- Sequential DSL calls → `:seq` edges.
|
|
138
|
+
- `if`/`unless`/`case`/`&&`/`||`/early-return around a step → `:conditional` edge
|
|
139
|
+
labeled with the condition source; steps only reachable under a guard render
|
|
140
|
+
`conditional`.
|
|
141
|
+
- `continue_if` → a gate node; its false path is a `:terminal` edge (workflow
|
|
142
|
+
halts).
|
|
143
|
+
- `branch` block → fans out (`:fanout`) to its spawn/`spawn_each` child-group;
|
|
144
|
+
`merge_branches` is the `:join` those edges reconnect into.
|
|
145
|
+
- `each`/`times`/`while` containing durable calls → one node + a "dynamic loop
|
|
146
|
+
count" **warning** (conservative — no unrolling).
|
|
147
|
+
|
|
148
|
+
## Error handling
|
|
149
|
+
|
|
150
|
+
The analyzer must never break the dashboard:
|
|
151
|
+
|
|
152
|
+
- Source unavailable (`source_location` nil, C-defined, `eval`'d, unreadable
|
|
153
|
+
file) → return a `Definition` carrying a single `unavailable` warning; the page
|
|
154
|
+
renders "can't be statically analyzed" gracefully. Never raises.
|
|
155
|
+
- Any Prism parse issue degrades the same way (Prism is error-tolerant).
|
|
156
|
+
- Missing/unloadable `job_class`, or a partially-resolved analysis → render what
|
|
157
|
+
was found plus a warnings panel.
|
|
158
|
+
- **Analyzer is pure/read-only over source text**; the overlay does read-only
|
|
159
|
+
queries only.
|
|
160
|
+
|
|
161
|
+
## Caching
|
|
162
|
+
|
|
163
|
+
- Memoize `Definition` by `job_class` + source-file digest — auto-invalidates on
|
|
164
|
+
dev code reload, stable in prod.
|
|
165
|
+
- The **overlay is never cached** — it is per-run and changes every poll.
|
|
166
|
+
|
|
167
|
+
## Testing
|
|
168
|
+
|
|
169
|
+
- **Analyzer unit tests (no DB):** a fixture set of workflow classes — linear,
|
|
170
|
+
conditional/`continue_if`, `branch`+`spawn_each`, `durably_repeat`, dynamic
|
|
171
|
+
`name:`, helper-traced, unanalyzable loop — asserting node kinds, edges, guards,
|
|
172
|
+
and warnings. Deterministic and fast.
|
|
173
|
+
- **Overlay tests (dashboard harness):** seed `execution_logs` + child workflows;
|
|
174
|
+
assert per-node status, fan-out aggregates, repeat counts, and the `unmapped`
|
|
175
|
+
path.
|
|
176
|
+
- **`MermaidRenderer`:** golden-text tests (statused `Definition` → expected
|
|
177
|
+
flowchart string).
|
|
178
|
+
|
|
179
|
+
## Scope (v1)
|
|
180
|
+
|
|
181
|
+
**In:** all seven primitives as nodes + conditional edges + fan-out/repeat
|
|
182
|
+
aggregation + the overlay + the new per-run DAG page + Mermaid rendering;
|
|
183
|
+
same-class helper tracing.
|
|
184
|
+
|
|
185
|
+
**Out (deferred):**
|
|
186
|
+
- Cross-class helper tracing.
|
|
187
|
+
- Recursively expanding a spawned child *workflow class* into its own graph
|
|
188
|
+
(v1 shows it as one fan-out node; "drill into child" is a future feature).
|
|
189
|
+
- Per-node ETA/timing beyond status + counts (that's the separate progress/ETA
|
|
190
|
+
feature).
|
|
191
|
+
- A class-level (no-overlay) definition view (trivial later addition).
|
|
192
|
+
|
|
193
|
+
## Open questions / risks
|
|
194
|
+
|
|
195
|
+
- **Helper-tracing fixed point:** need a clear rule for what counts as "a durable
|
|
196
|
+
call inside a same-class method" vs. ordinary work, and recursion/mutual-call
|
|
197
|
+
guards. The analyzer stays conservative — when in doubt, emit a `dynamic` node +
|
|
198
|
+
warning rather than a confident-but-wrong expansion.
|
|
199
|
+
- **Ordinal binding for dynamic siblings** is best-effort; if two dynamic
|
|
200
|
+
`durably_execute` calls interleave at runtime out of source order, the overlay
|
|
201
|
+
may mis-bind. Acceptable for v1 (surfaced as `dynamic`).
|
|
202
|
+
- **Mermaid as first client dependency** — keep it vendored and isolated so the
|
|
203
|
+
rest of the dashboard stays server-rendered.
|
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# Design: Stalled-Workflow Reaper for ChronoForge
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-07-09
|
|
4
|
+
**Gem:** `chrono_forge` (with `chrono_forge-dashboard`)
|
|
5
|
+
**Status:** Approved for planning
|
|
6
|
+
|
|
7
|
+
## Problem
|
|
8
|
+
|
|
9
|
+
A workflow that is mid-pass when its worker is **hard-killed** (SIGKILL: deploy/rollout,
|
|
10
|
+
OOM, node eviction, SolidQueue heartbeat prune) is stranded permanently in `state: running`
|
|
11
|
+
with a stale lock, and nothing ever resumes it.
|
|
12
|
+
|
|
13
|
+
The mechanism, confirmed against the code:
|
|
14
|
+
|
|
15
|
+
- `Executor#perform` releases the lock and publishes the resume continuation in its `ensure`
|
|
16
|
+
block (`lib/chrono_forge/executor.rb:178-197`): `release_lock` first, then
|
|
17
|
+
`flush_continuation!`.
|
|
18
|
+
- `enqueue_continuation` (`executor.rb:369-371`) only records `@continuation` in memory;
|
|
19
|
+
`flush_continuation!` (`executor.rb:376-382`) is what actually enqueues the ActiveJob.
|
|
20
|
+
- On SIGKILL, Ruby's `ensure` does not run. So (1) `release_lock` never runs — the row stays
|
|
21
|
+
`state: running` with a past `locked_at`; and (2) `flush_continuation!` never runs — no job
|
|
22
|
+
is scheduled to resume the workflow.
|
|
23
|
+
|
|
24
|
+
The workflow is fully *resumable* — `Workflow#executable?` is `idle? || running?`
|
|
25
|
+
(`workflow.rb:48-50`) and `LockStrategy.acquire_lock` steals any lock older than
|
|
26
|
+
`max_duration` (10 min) (`lock_strategy.rb:17-21`) — but nothing re-enqueues it. Existing
|
|
27
|
+
recovery mechanisms do not cover this:
|
|
28
|
+
|
|
29
|
+
- **`BranchMergeJob` rekick** (`branch_merge_job.rb:232-273`) only re-enqueues children that
|
|
30
|
+
are `state: idle, started_at: nil` (never-started dropped dispatches). A child hard-killed
|
|
31
|
+
**mid-pass** is `state: running` with `started_at` set — explicitly excluded (see the
|
|
32
|
+
comment at `branch_merge_job.rb:226-231`). So branch children have the identical gap.
|
|
33
|
+
- **Workflow-level retry** rides on the same process that must survive to its `ensure`; a
|
|
34
|
+
killed process never reaches it.
|
|
35
|
+
- **`retry_now`/`retry_later`** require `retryable?` (`stalled? || failed?`,
|
|
36
|
+
`workflow.rb:53-55`); a stranded workflow is `running`, so they refuse it.
|
|
37
|
+
- **Dashboard "Force unlock"** clears the lock but enqueues nothing.
|
|
38
|
+
|
|
39
|
+
Real-world impact (reported): 735 workflows stranded over ~8.5 months in one production app,
|
|
40
|
+
still accruing ~8/day.
|
|
41
|
+
|
|
42
|
+
## Solution Overview
|
|
43
|
+
|
|
44
|
+
Add a reconciliation entry point, `ChronoForge::Workflow.reap_stalled`, that finds workflows
|
|
45
|
+
stuck in `running` past a safety threshold and re-enqueues them. Re-enqueue is safe:
|
|
46
|
+
`acquire_lock` steals the stale lock and completed durable steps replay as no-ops.
|
|
47
|
+
|
|
48
|
+
### Decisions
|
|
49
|
+
|
|
50
|
+
| Decision | Choice | Rationale |
|
|
51
|
+
|---|---|---|
|
|
52
|
+
| Shipping | **Class method only** | No Railtie/rake task/self-scheduling job. Host apps call it from their own cron/recurring job. Smallest surface; no new gem infrastructure. |
|
|
53
|
+
| Scope | **All stranded `running` workflows** (top-level AND branch children) | Strictly more correct — rekick misses mid-pass-killed children. Re-enqueue of a child directly is the proven-safe pattern rekick already uses. |
|
|
54
|
+
| Threshold | **Config + per-call override** | `ChronoForge.config.reap_stale_after` (default `30.minutes`), overridable via `stale_after:` arg. Parity with the existing `branch_merge_queue` config. |
|
|
55
|
+
| Deliverable | **Code + tests + docs** | Implementation, tests (incl. simulated hard-kill strand), and docs covering the reaper + step-idempotency expectations. |
|
|
56
|
+
|
|
57
|
+
## Component
|
|
58
|
+
|
|
59
|
+
`ChronoForge::Workflow.reap_stalled` — a class method on the AR model. It only queries and
|
|
60
|
+
re-enqueues; it holds no lock and performs no replay itself (the re-enqueued job does the
|
|
61
|
+
replay, under the normal lock protocol).
|
|
62
|
+
|
|
63
|
+
```ruby
|
|
64
|
+
# ChronoForge::Workflow
|
|
65
|
+
def self.reap_stalled(stale_after: ChronoForge.config.reap_stale_after)
|
|
66
|
+
reaped = 0
|
|
67
|
+
where(state: states[:running])
|
|
68
|
+
.where("locked_at < ?", stale_after.ago)
|
|
69
|
+
.find_each do |wf|
|
|
70
|
+
wf.job_klass.perform_later(wf.key, **wf.kwargs.symbolize_keys)
|
|
71
|
+
reaped += 1
|
|
72
|
+
rescue => e
|
|
73
|
+
Rails.logger.error { "ChronoForge reap failed for workflow(#{wf.key}): #{e.class}: #{e.message}" }
|
|
74
|
+
end
|
|
75
|
+
Rails.logger.info { "ChronoForge reaped #{reaped} stalled workflow(s)" } if reaped.positive?
|
|
76
|
+
reaped
|
|
77
|
+
end
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Design points:
|
|
81
|
+
|
|
82
|
+
- **Target = `running` + stale `locked_at`.** The strand happens precisely because the
|
|
83
|
+
`ensure` never ran, so `release_lock` never flipped the row off `running`. A `NULL locked_at`
|
|
84
|
+
cannot match `locked_at < ?` and is naturally excluded (an anomalous running-without-lock row
|
|
85
|
+
is not our target).
|
|
86
|
+
- **Sweeps children too.** No `parent_execution_log_id` filter. A re-enqueued child runs,
|
|
87
|
+
replays, completes, and updates its `BranchProbe` state so the parent's merge poller observes
|
|
88
|
+
completion.
|
|
89
|
+
- **Re-enqueue via the public guarded `perform_later`.** `wf.kwargs` is a JSON column (string
|
|
90
|
+
keys); `.symbolize_keys` for `**`, matching `setup_workflow!` and `rekick_dropped_jobs`. The
|
|
91
|
+
guard rejects `RESERVED_KWARGS`, which stored user kwargs never contain. Fresh `attempt: 0`
|
|
92
|
+
— a full workflow-retry budget, exactly like a normal resume.
|
|
93
|
+
- **Per-row rescue** so one bad row (e.g. cross-version kwarg drift tripping the enqueue guard)
|
|
94
|
+
never aborts the sweep — mirrors `rekick_dropped_jobs` (`branch_merge_job.rb:265-270`).
|
|
95
|
+
- **Returns the reaped count** for cron visibility; logs via `Rails.logger` block syntax.
|
|
96
|
+
|
|
97
|
+
## Configuration
|
|
98
|
+
|
|
99
|
+
`max_duration` (previously a hardcoded `10.minutes` in the executor) becomes a config value,
|
|
100
|
+
and `reap_stale_after` **derives from it** so the "reap threshold must exceed the lock-steal
|
|
101
|
+
threshold" invariant is automatic rather than documented-and-hoped.
|
|
102
|
+
|
|
103
|
+
Add to `ChronoForge::Configuration`:
|
|
104
|
+
|
|
105
|
+
```ruby
|
|
106
|
+
# How long a single pass may hold its lock before it's stealable (feeds
|
|
107
|
+
# LockStrategy.acquire_lock via the executor's #max_duration). Default 10 minutes.
|
|
108
|
+
attr_accessor :max_duration
|
|
109
|
+
|
|
110
|
+
# Age past which a :running workflow is considered stranded and re-enqueued by
|
|
111
|
+
# Workflow.reap_stalled. Defaults to 3x max_duration (30 min out of the box) so it always
|
|
112
|
+
# clears the lock-steal threshold; an explicit value overrides the derived default.
|
|
113
|
+
def reap_stale_after
|
|
114
|
+
@reap_stale_after || max_duration * 3
|
|
115
|
+
end
|
|
116
|
+
attr_writer :reap_stale_after
|
|
117
|
+
|
|
118
|
+
# in #initialize:
|
|
119
|
+
@max_duration = 10.minutes
|
|
120
|
+
@reap_stale_after = nil
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
The executor's `#max_duration` now reads `ChronoForge.config.max_duration` instead of
|
|
124
|
+
returning a literal. Deriving `reap_stale_after` from `max_duration` means raising one raises
|
|
125
|
+
the other, so the reaper can never be configured below the lock-steal threshold by accident.
|
|
126
|
+
|
|
127
|
+
## Idempotency & Safety (documented behavior)
|
|
128
|
+
|
|
129
|
+
- **Concurrency-safe.** Overlapping cron runs, or a re-enqueue landing while the old stale
|
|
130
|
+
lock still shows, at worst enqueue a duplicate job — the second loses the `acquire_lock`
|
|
131
|
+
race and no-ops via `ConcurrentExecutionError`. Mild, harmless churn. Recommend a cron
|
|
132
|
+
interval comfortably longer than a typical resume pass.
|
|
133
|
+
- **Step idempotency.** Reaping replays the interrupted pass. A `durably_execute` step only
|
|
134
|
+
short-circuits when `completed`, so a step whose side effect committed but whose log never
|
|
135
|
+
reached `completed` will re-run on replay. Steps with external side effects must be
|
|
136
|
+
idempotent (natural/unique key + `create_or_find_by`/rescue). Documented alongside
|
|
137
|
+
`wait`/`durably_execute`.
|
|
138
|
+
|
|
139
|
+
## Non-Goals (conscious boundaries)
|
|
140
|
+
|
|
141
|
+
- **Dropped `BranchMergeJob` poller → parent stranded `idle`.** A parent parked on a merge
|
|
142
|
+
whose poller was hard-killed sits `idle`, not `running`, so this reaper won't catch it. A
|
|
143
|
+
distinct failure mode; the code already hints (`branch_merge_job.rb:181-184`) that a
|
|
144
|
+
"`next_poll_at` far in the past with work still pending" detector is the right fix. Out of
|
|
145
|
+
scope here.
|
|
146
|
+
- **Micro-window between `release_lock` and `flush_continuation!`.** A process dying between
|
|
147
|
+
those two statements yields `idle` + no continuation. Theoretically possible, not seen in the
|
|
148
|
+
reported evidence (all strands were `running`), not covered.
|
|
149
|
+
|
|
150
|
+
## Testing
|
|
151
|
+
|
|
152
|
+
- **Query selection (unit):** selects only `running` + stale-lock rows; excludes
|
|
153
|
+
fresh-lock `running`, `idle`, `completed`, `failed`, `stalled`, and `NULL locked_at`.
|
|
154
|
+
- **Strand-and-recover (behavioral):** create a workflow, force it into `running` with an old
|
|
155
|
+
`locked_at` and no scheduled resume job (simulating the hard-kill), run `reap_stalled`,
|
|
156
|
+
assert a job is enqueued and that draining the queue resumes and completes the workflow.
|
|
157
|
+
Cover both a top-level workflow and a branch child.
|
|
158
|
+
- **Per-row rescue:** a row that raises on re-enqueue does not abort the sweep; the returned
|
|
159
|
+
count reflects only successful re-enqueues.
|
|
160
|
+
- **Config:** default `reap_stale_after` honored; per-call `stale_after:` override honored.
|
|
161
|
+
|
|
162
|
+
## Documentation
|
|
163
|
+
|
|
164
|
+
- Document `Workflow.reap_stalled` and the recommended cron wiring (host-owned; no shipped
|
|
165
|
+
task), including the `reap_stale_after > max_duration` constraint.
|
|
166
|
+
- Document the step-idempotency expectation alongside `wait`/`durably_execute`.
|
|
167
|
+
|
|
168
|
+
## Alternatives Considered
|
|
169
|
+
|
|
170
|
+
- **Railtie + `chrono_forge:reap` rake task** (angarium `angarium:reap` parity) — rejected;
|
|
171
|
+
heavier, would add the gem's first Railtie.
|
|
172
|
+
- **Self-scheduling reaper ActiveJob** — rejected; adds an always-on job and scheduling
|
|
173
|
+
assumptions the host may not want.
|
|
174
|
+
- **Top-level-only sweep** (`parent_execution_log_id IS NULL`) — rejected; leaves
|
|
175
|
+
mid-pass-killed branch children stranded, which rekick does not recover.
|