@ssheleg/agent-stack 0.9.0 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +91 -0
- package/README.md +13 -2
- package/package.json +1 -1
- package/plugins/agent-stack/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-stack/skills/agent-evals/SKILL.md +44 -3
- package/plugins/agent-stack/skills/agent-harness/SKILL.md +29 -3
- package/plugins/agent-stack/skills/agent-harness/scripts/audit_agent.py +100 -32
- package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +100 -101
- package/plugins/agent-stack/skills/agent-orchestrator/references/graph-engineering.md +331 -0
- package/plugins/agent-stack/skills/agent-orchestrator/references/llm-proxy-billing.md +1 -1
- package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md +38 -14
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,96 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.10.1] — 2026-08-15
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- **`graph-engineering.md`'s stamp names all three of its sources, and the two numbers that
|
|
8
|
+
come from the third say so.** The file pins an article and Claude Code's changelog, and
|
|
9
|
+
its execution section also states a concurrency cap and a lifetime agent cap that come
|
|
10
|
+
from **neither** — they are the Workflow tool contract the host presents at runtime, a
|
|
11
|
+
source with its own lifetime and no public changelog entry to check them against. A
|
|
12
|
+
revision stamp that names two sources for a file carrying three is the exact defect the
|
|
13
|
+
stamp exists to prevent, one level up. Both numbers now carry the instruction to read them
|
|
14
|
+
back from the running host.
|
|
15
|
+
|
|
16
|
+
## [0.10.0] — 2026-08-15
|
|
17
|
+
|
|
18
|
+
### Added
|
|
19
|
+
|
|
20
|
+
- **`agent-orchestrator/references/graph-engineering.md` — deciding the shape of the work
|
|
21
|
+
before doing it.** The pack could wire a loop, prove it behaved, connect it and tell it
|
|
22
|
+
what to do, and had nothing to say about the question that comes before all four: *does
|
|
23
|
+
this actually have to happen in a line?* Node and edge, the fake-edge test, the diamond,
|
|
24
|
+
the two ways a diamond fails silently, the checker node, static versus dynamic, and the
|
|
25
|
+
cost table that says when a graph is not worth building.
|
|
26
|
+
|
|
27
|
+
Sourced from *Graph Engineering with Claude*
|
|
28
|
+
(`https://x.com/Mahaximus_/status/2082442856417956173`, published 2026-07-29), and the
|
|
29
|
+
link is kept so the original can be re-read rather than remembered through the summary.
|
|
30
|
+
Four sections are **this pack's** and are marked as such in the file: what the host
|
|
31
|
+
actually runs, the barrier distinction, what a checker costs, and auditability as a hard
|
|
32
|
+
rule rather than a preference.
|
|
33
|
+
|
|
34
|
+
- **What Claude Code actually executes, with version evidence.** The source's one
|
|
35
|
+
operational claim aged out six weeks after publication: the `workflow` keyword it names
|
|
36
|
+
was renamed to `ultracode` in **v2.1.160**, and the YAML it shows is a way of describing
|
|
37
|
+
a graph in a prompt rather than a syntax the host parses — the execution contract is a
|
|
38
|
+
script whose primitives are `agent()`, `parallel()` (a barrier) and `pipeline()` (none).
|
|
39
|
+
Every claim in that section carries the `v2.1.x` changelog entry that establishes it,
|
|
40
|
+
which is why the correction reads as a dated fact rather than as an error by the author.
|
|
41
|
+
|
|
42
|
+
- **`agent-harness` — *Static or dynamic, the second question*.** The workflow-versus-agent
|
|
43
|
+
table decided one thing and left the other open. Six rows, and one of them is hard: a run
|
|
44
|
+
that has to be auditable is static, because a graph that picks its own next nodes produces
|
|
45
|
+
a shape nobody drew, and then *"here is the design"* and *"here is what happened"* stop
|
|
46
|
+
being the same document.
|
|
47
|
+
|
|
48
|
+
- **`agent-evals` §5a — the checker node as an evaluator that runs inside the graph.** Its
|
|
49
|
+
five catches split three code checks and two judge calls, so §5's *cheap checks first*
|
|
50
|
+
applies to a position in the graph rather than to a suite. And the part that makes it eval
|
|
51
|
+
work: **a checker that has never rejected anything is a finding, not a reassurance** — its
|
|
52
|
+
verdicts are scores with a source, and its rejection rate belongs on the same dashboard as
|
|
53
|
+
its pass rate.
|
|
54
|
+
|
|
55
|
+
- **A sixth scanner detector, `unguarded-fanout`.** `asyncio.gather` with no
|
|
56
|
+
`return_exceptions=True`, or `Promise.all` with no `allSettled` and no per-branch
|
|
57
|
+
`.catch`: the first sibling to fail cancels the batch, the others' completed work is
|
|
58
|
+
discarded, and the node consuming the results cannot tell a failed branch from an empty
|
|
59
|
+
one. Conservative like the rest — the capturing form anywhere in the file silences it.
|
|
60
|
+
|
|
61
|
+
### Changed
|
|
62
|
+
|
|
63
|
+
- **The self-test proves silence as well as noise.** `PLANTS` and `CLEAN` replaced the
|
|
64
|
+
single dict, because one detector now reads two languages and needed a plant in each, and
|
|
65
|
+
because a detector that fires on the defect *and* on its fix has no discriminating power.
|
|
66
|
+
A fan-out that **does** capture its branches is now a fixture the pass must stay silent
|
|
67
|
+
on. `self-test: 6/6` → **`9/9`** (seven plants, two clean fixtures), counted by running it.
|
|
68
|
+
|
|
69
|
+
- **`agent-orchestrator` §5 no longer contradicts its own data model.** `PlanStage` declared
|
|
70
|
+
`depends_on` and the executor beside it walked `plan.stages` in list order — a plan that
|
|
71
|
+
went to the trouble of saying it need not be serialised, serialised. It now executes in
|
|
72
|
+
dependency layers (Kahn), gates a layer of more than one on the checker before anything
|
|
73
|
+
consumes it, and `ExecutionPlan` grows the `layers()` that makes the declaration mean
|
|
74
|
+
something. A cycle fails the plan rather than deadlocking the run.
|
|
75
|
+
|
|
76
|
+
### Removed
|
|
77
|
+
|
|
78
|
+
- **The hardcoded context-window table.** `references/patterns.md` carried nine vendor model
|
|
79
|
+
ids with their windows and a `DEFAULT_CONTEXT_WINDOW` of 16 000. Every number was correct
|
|
80
|
+
when written and none survived: ids were renamed, long-context variants shipped under the
|
|
81
|
+
same family name, and a system reading that table would size its budget for a window an
|
|
82
|
+
order of magnitude smaller than the one it was given. Replaced by the resolution order —
|
|
83
|
+
configuration, then the provider, then a conservative floor **with a loud log line** — and
|
|
84
|
+
the check is mechanical: no vendor model id remains anywhere in the shipped skill text.
|
|
85
|
+
Two illustrative ones in `SKILL.md` and `llm-proxy-billing.md` went with it, because a
|
|
86
|
+
class fixed in one place and left in two others is not fixed.
|
|
87
|
+
|
|
88
|
+
- **Two duplicated homes.** §8's learning tables and §10's prompt-assembly snippet restated
|
|
89
|
+
what `references/patterns.md` and `agent-harness/references/system-prompt.md` already own.
|
|
90
|
+
Both now state the decision and point at the home. `SKILL.md` is **502 lines / 5670
|
|
91
|
+
tokens** against a 4750 working budget — the number is counted, it is over, and the next
|
|
92
|
+
addition to this body should split rather than absorb.
|
|
93
|
+
|
|
3
94
|
## [0.9.0] — 2026-08-15
|
|
4
95
|
|
|
5
96
|
### Added
|
package/README.md
CHANGED
|
@@ -11,7 +11,7 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
|
|
|
11
11
|
|
|
12
12
|
Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it
|
|
13
13
|
behaves, `agent-interop` for everything it talks to outside its own process,
|
|
14
|
-
`agent-harness` for what it is **told** — and
|
|
14
|
+
`agent-harness` for what it is **told** — and nineteen references they load on demand,
|
|
15
15
|
plus one scanner.
|
|
16
16
|
|
|
17
17
|
**The orchestrator** (`SKILL.md`) — what the agent reads first:
|
|
@@ -31,6 +31,10 @@ plus one scanner.
|
|
|
31
31
|
decay, plus conflict resolution when a new learning contradicts an old one
|
|
32
32
|
- context budget allocation by priority
|
|
33
33
|
- self-learning feedback loops
|
|
34
|
+
- **the shape of the work, decided before the work**: an edge that carries no data is
|
|
35
|
+
no edge, a plan that declares `depends_on` is executed in dependency layers rather
|
|
36
|
+
than in list order, and a parallel layer gets a checker before the node that
|
|
37
|
+
consumes it
|
|
34
38
|
|
|
35
39
|
**The evals skill** (`agent-evals/SKILL.md`) — how you know any of it works. An
|
|
36
40
|
agent's behaviour is not in its source, so the artifact under test is the
|
|
@@ -74,9 +78,16 @@ eight extension seams where a permission gate or a context rewrite can actually
|
|
|
74
78
|
|
|
75
79
|
It runs in both directions: **building a harness and auditing somebody else's are one
|
|
76
80
|
checklist read forwards and backwards.** `scripts/audit_agent.py` is the mechanical half —
|
|
77
|
-
|
|
81
|
+
six conservative detectors, and it always prints what it *cannot* see plus a denominator,
|
|
78
82
|
so its silence is never read as a pass.
|
|
79
83
|
|
|
84
|
+
**`references/graph-engineering.md`** — deciding the shape of the work before
|
|
85
|
+
doing it: node and edge, the fake-edge test, the diamond and the two ways it
|
|
86
|
+
fails silently, the checker node and what it costs, static versus dynamic with
|
|
87
|
+
auditability as the hard rule, when a graph is not worth building, and what a
|
|
88
|
+
host actually executes when it fans out — with the version evidence, because the
|
|
89
|
+
keyword the source named was renamed six weeks after it was published.
|
|
90
|
+
|
|
80
91
|
**`references/context-engineering.md`** — what the loop gives up when the window
|
|
81
92
|
runs out: the five-rung compaction ladder and why to re-measure between rungs,
|
|
82
93
|
the tool-pair boundary invariant, typed carryover blocks copied across the
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ssheleg/agent-stack",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.1",
|
|
4
4
|
"description": "Production patterns for AI agent orchestrators \u2014 tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay \u2014 plus the wallet side of reselling LLM access. This package is the installer CLI.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"agent-stack": "bin/agent-stack.js"
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "agent-stack",
|
|
3
3
|
"displayName": "Agent Stack",
|
|
4
4
|
"description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
|
|
5
|
-
"version": "0.
|
|
5
|
+
"version": "0.10.1",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "ssheleg",
|
|
8
8
|
"url": "https://x.com/sshlg93"
|
|
@@ -7,9 +7,10 @@ description: >-
|
|
|
7
7
|
evals. Covers the three observability primitives (run, trace, thread) crossed with three
|
|
8
8
|
eval granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
|
|
9
9
|
timing axis, pass-fail rubrics over scalar scores, cheap code checks before model
|
|
10
|
-
judges,
|
|
11
|
-
|
|
12
|
-
"
|
|
10
|
+
judges, the checker node as an evaluator running inside the graph, simulated users with
|
|
11
|
+
adversarial personas, annotation queues, and what to instrument so any of it is
|
|
12
|
+
possible. Triggers - "agent eval", "eval suite", "LLM judge", "regression fixture",
|
|
13
|
+
"trajectory eval", "checker node", "is the agent getting better", "эвалы агента",
|
|
13
14
|
"оценка агента", "LLM-судья", "регрессионный набор", "как проверить агента". Not for
|
|
14
15
|
unit tests of ordinary code, or for benchmarking a model.
|
|
15
16
|
license: MIT
|
|
@@ -159,6 +160,44 @@ domain expert and accept that this tier stays human.
|
|
|
159
160
|
|
|
160
161
|
---
|
|
161
162
|
|
|
163
|
+
## 5a. The checker node — an evaluator that runs inside the graph
|
|
164
|
+
|
|
165
|
+
Everything above evaluates a run *afterwards*. One evaluator runs **during** it, and it is
|
|
166
|
+
the one most systems are missing: a **checker node** sitting between a parallel layer and
|
|
167
|
+
the node that consumes it. Its only job is *usable / not usable*, and the convergence
|
|
168
|
+
depends on **it** rather than on the branches — otherwise the gate has a bypass.
|
|
169
|
+
|
|
170
|
+
It matters here because it is the same machinery as §5, positioned differently:
|
|
171
|
+
|
|
172
|
+
| It catches | Decided by |
|
|
173
|
+
|---|---|
|
|
174
|
+
| an empty or null output | a code check |
|
|
175
|
+
| a confidence signal below the downstream bar | a code check |
|
|
176
|
+
| a shape that will break the consumer's parsing | a code check |
|
|
177
|
+
| two outputs that cannot both be true | a judge |
|
|
178
|
+
| an output answering a different question than the one asked | a judge |
|
|
179
|
+
|
|
180
|
+
Three of five are free. Run them first — §5's *cheap checks first*, applied to a position
|
|
181
|
+
in the graph rather than to a suite.
|
|
182
|
+
|
|
183
|
+
**A checker is a node, so it can be wrong, and its failure mode is silent approval.** A
|
|
184
|
+
model checker that has never been shown a bad input passes everything, and a graph with a
|
|
185
|
+
checker that always says yes is **worse** than one with none: the missing checkpoint has
|
|
186
|
+
been replaced by a false one that everything downstream now trusts. Three consequences,
|
|
187
|
+
and they are eval work rather than orchestration work:
|
|
188
|
+
|
|
189
|
+
- **Watch it refuse a planted bad output** before trusting it, exactly as §5 requires of
|
|
190
|
+
any judge before it scores unattended.
|
|
191
|
+
- **Record every verdict as a score bound to the run**, with `source: code_check` or
|
|
192
|
+
`llm_judge` (§7). A checker whose verdicts are not stored cannot be asked afterwards how
|
|
193
|
+
often it fired, which means it cannot be calibrated.
|
|
194
|
+
- **A checker that has never rejected anything is a finding, not a reassurance.** Put the
|
|
195
|
+
rejection rate on the same dashboard as the pass rate; a rate of zero is either a
|
|
196
|
+
perfect upstream or a broken gate, and only the stored verdicts can tell you which.
|
|
197
|
+
|
|
198
|
+
Where the checker sits in the shape, and why the convergence needs one at all:
|
|
199
|
+
`agent-orchestrator/references/graph-engineering.md`.
|
|
200
|
+
|
|
162
201
|
## 6. The corpus grows from production
|
|
163
202
|
|
|
164
203
|
Never author the suite up front. Every production failure and every thumbs-down becomes a
|
|
@@ -215,6 +254,8 @@ None of the above runs without these, and they are the part people skip:
|
|
|
215
254
|
- [ ] Pass/fail rubrics with enumerated failure conditions, written with behaviour owners
|
|
216
255
|
- [ ] Code checks before model judges
|
|
217
256
|
- [ ] Judge calibrated against human labels before it is trusted
|
|
257
|
+
- [ ] Every checker node watched refusing a planted output, its verdicts stored as scores,
|
|
258
|
+
and its rejection rate on the dashboard — a checker at zero rejections is a finding
|
|
218
259
|
- [ ] Domain-expert review for output a general judge cannot grade
|
|
219
260
|
- [ ] Every production failure minimised into a permanent fixture
|
|
220
261
|
- [ ] Annotation queue with filters, and the two reviewer roles kept separate
|
|
@@ -8,9 +8,10 @@ description: >-
|
|
|
8
8
|
instead of a score, plus a scanner. Carries Pi as a worked kernel implementation — SDK, RPC
|
|
9
9
|
and extension seams — for embedding or extending a harness. Triggers - "system prompt",
|
|
10
10
|
"tool description", "agent picks the wrong tool", "agent loops forever", "prompt
|
|
11
|
-
engineering", "ReAct", "workflow or agent", "
|
|
12
|
-
SDK", "Pi harness", "системный промпт", "агент не вызывает тул",
|
|
13
|
-
агента". Not for the loop's plumbing, its evals, or its
|
|
11
|
+
engineering", "ReAct", "workflow or agent", "static or dynamic", "audit this agent",
|
|
12
|
+
"embed an agent", "agent SDK", "Pi harness", "системный промпт", "агент не вызывает тул",
|
|
13
|
+
"аудит агента", "встроить агента". Not for the loop's plumbing, its evals, or its
|
|
14
|
+
protocols — those are siblings.
|
|
14
15
|
---
|
|
15
16
|
|
|
16
17
|
# Agent harness — what the agent is told, and how to audit what someone else told theirs
|
|
@@ -81,6 +82,30 @@ that only where a fixed path genuinely cannot be written.
|
|
|
81
82
|
from a known set. Orchestration invents the set per request. If you can enumerate the
|
|
82
83
|
branches, you wanted routing and it is cheaper.
|
|
83
84
|
|
|
85
|
+
### Static or dynamic — the second question, and it is not the same one
|
|
86
|
+
|
|
87
|
+
Having chosen a workflow, one thing is still open: **is its shape known before it runs?**
|
|
88
|
+
A **static** graph has every node and edge decided up front. A **dynamic** one grows — a
|
|
89
|
+
node finishes, reads what it found, and decides what comes next.
|
|
90
|
+
|
|
91
|
+
| Reach for | When |
|
|
92
|
+
|---|---|
|
|
93
|
+
| **static** | the task repeats and the structure is the same every time |
|
|
94
|
+
| **static** | predictability and speed matter more than flexibility |
|
|
95
|
+
| **static** | **always first** — go dynamic only after the static version hits a wall you can name |
|
|
96
|
+
| dynamic | the scope of the work depends on what is discovered along the way |
|
|
97
|
+
| dynamic | a node must choose its successors from its own output |
|
|
98
|
+
| **never dynamic** | **the run has to be auditable** — see below |
|
|
99
|
+
|
|
100
|
+
**The audit rule is hard, not a preference.** A dynamic graph's executed shape is not the
|
|
101
|
+
shape anybody drew, so *"here is the design"* and *"here is what happened"* stop being the
|
|
102
|
+
same document and every claim about the run becomes unfalsifiable from outside. Most
|
|
103
|
+
workflows that feel like they need a dynamic graph need a better static one.
|
|
104
|
+
|
|
105
|
+
The rest of the model — the fake-edge test, the diamond, the checker node before a
|
|
106
|
+
convergence, and what a host actually executes when it fans out — is
|
|
107
|
+
`agent-orchestrator/references/graph-engineering.md`.
|
|
108
|
+
|
|
84
109
|
---
|
|
85
110
|
|
|
86
111
|
## References
|
|
@@ -151,6 +176,7 @@ prompt.
|
|
|
151
176
|
## Checklist — a harness worth shipping
|
|
152
177
|
|
|
153
178
|
- [ ] Workflow-versus-agent decided deliberately, and the simpler option was actually tried
|
|
179
|
+
- [ ] Static-versus-dynamic decided too, and a run that must be auditable is static
|
|
154
180
|
- [ ] System prompt at the **right altitude** — heuristics, not hardcoded branches, not vague hope
|
|
155
181
|
- [ ] Every status, category and enum the agent must produce is **enumerated in the prompt**
|
|
156
182
|
- [ ] Today's date, and any other volatile context, injected rather than assumed
|
|
@@ -158,14 +158,50 @@ def check_hardcoded_model(rel, text, lines):
|
|
|
158
158
|
"tenant, system default — is the shape that bills correctly")
|
|
159
159
|
|
|
160
160
|
|
|
161
|
+
def check_unguarded_fanout(rel, text, lines):
|
|
162
|
+
"""A fan-out whose siblings' failures are not captured.
|
|
163
|
+
|
|
164
|
+
`asyncio.gather` without `return_exceptions=True` cancels the whole batch on the first
|
|
165
|
+
exception: every other branch's completed work is discarded, and the node that consumes
|
|
166
|
+
the results cannot tell a branch that FAILED from one that returned nothing. `Promise.all`
|
|
167
|
+
has the identical shape. That is the failure a checker node between a parallel layer and
|
|
168
|
+
its convergence exists to stop, and it is invisible in a green test run because the happy
|
|
169
|
+
path never exercises it.
|
|
170
|
+
|
|
171
|
+
Conservative twice over, like every detector here: the file must already look
|
|
172
|
+
agent-related, and the capturing form must be absent from the WHOLE file — one
|
|
173
|
+
`return_exceptions` or `allSettled` anywhere is taken as evidence the author knows the
|
|
174
|
+
distinction, and this pass says nothing.
|
|
175
|
+
"""
|
|
176
|
+
if re.search(r"return_exceptions|allSettled", text):
|
|
177
|
+
return
|
|
178
|
+
for i, l in enumerate(lines, 1):
|
|
179
|
+
if re.search(r"\basyncio\.gather\s*\(", l):
|
|
180
|
+
add("unguarded-fanout", rel, i,
|
|
181
|
+
"`asyncio.gather` with no `return_exceptions=True` — the first sibling to "
|
|
182
|
+
"raise cancels the batch and throws away what the others already produced",
|
|
183
|
+
"Capture each branch's outcome, then gate the convergence on a checker that "
|
|
184
|
+
"can tell a failed branch from an empty one")
|
|
185
|
+
if re.search(r"\bPromise\.all\s*\(", l) and ".catch" not in l:
|
|
186
|
+
add("unguarded-fanout", rel, i,
|
|
187
|
+
"`Promise.all` with no `allSettled` and no per-branch `.catch` — one "
|
|
188
|
+
"rejection discards every other branch's completed result",
|
|
189
|
+
"Use `Promise.allSettled` or catch per branch, then gate the convergence on "
|
|
190
|
+
"a checker that can see which branch failed")
|
|
191
|
+
|
|
192
|
+
|
|
161
193
|
CHECKS = [check_unbounded_loop, check_tool_without_description, check_swallowed_error,
|
|
162
|
-
check_no_timeout, check_hardcoded_model]
|
|
194
|
+
check_no_timeout, check_hardcoded_model, check_unguarded_fanout]
|
|
163
195
|
|
|
164
196
|
# What no static pass can reach. Printed every run, never suppressed.
|
|
165
197
|
BLIND = [
|
|
166
198
|
"whether the SYSTEM PROMPT is at the right altitude — or whether it is in this repo at all",
|
|
167
199
|
"whether two tool descriptions actually distinguish themselves to a model",
|
|
168
200
|
"whether the workflow/agent choice was made deliberately or defaulted to an agent",
|
|
201
|
+
"whether a fan-out has a CHECKER between it and the node that consumes it — this pass "
|
|
202
|
+
"sees an unguarded gather, never a missing gate",
|
|
203
|
+
"whether a declared dependency graph is actually executed in dependency order, or in "
|
|
204
|
+
"the order the stages happen to be listed in",
|
|
169
205
|
"whether retries and fallbacks MULTIPLY (three providers x three retries is nine calls)",
|
|
170
206
|
"whether compaction preserves decisions and open questions, or keeps the discussion",
|
|
171
207
|
"whether tool output is treated as untrusted input",
|
|
@@ -217,54 +253,86 @@ def report_text(root, seen, considered):
|
|
|
217
253
|
return "\n".join(out)
|
|
218
254
|
|
|
219
255
|
|
|
256
|
+
PY_HEADER = ("import requests\n"
|
|
257
|
+
"system_prompt = 'x'\n"
|
|
258
|
+
"tools = [{'name': 't', 'description': 'does a thing'}]\n"
|
|
259
|
+
"messages = []\n")
|
|
260
|
+
JS_HEADER = ("const system_prompt = 'x';\n"
|
|
261
|
+
"const tools = [{name: 't', description: 'does a thing'}];\n"
|
|
262
|
+
"const messages = [];\n")
|
|
263
|
+
|
|
264
|
+
# (label, the check that MUST fire, filename, body)
|
|
265
|
+
PLANTS = [
|
|
266
|
+
("unbounded-loop", "unbounded-loop", "agent.py",
|
|
267
|
+
PY_HEADER + "while True:\n pass\n"),
|
|
268
|
+
("tool-no-description", "tool-no-description", "agent.py",
|
|
269
|
+
PY_HEADER + "T = [{'name': 'a', 'description': ''}]\n"),
|
|
270
|
+
("swallowed-error", "swallowed-error", "agent.py",
|
|
271
|
+
PY_HEADER + "try:\n x = 1\nexcept Exception:\n pass\n"),
|
|
272
|
+
("no-timeout", "no-timeout", "agent.py",
|
|
273
|
+
PY_HEADER + "r = requests.get('https://example.com')\n"),
|
|
274
|
+
("hardcoded-model", "hardcoded-model", "agent.py",
|
|
275
|
+
PY_HEADER + "a = 'claude-opus-4'\nb = 'claude-opus-4'\n"),
|
|
276
|
+
("unguarded-fanout (asyncio)", "unguarded-fanout", "agent.py",
|
|
277
|
+
PY_HEADER + "out = await asyncio.gather(*(run(t) for t in tasks))\n"),
|
|
278
|
+
("unguarded-fanout (promise)", "unguarded-fanout", "agent.js",
|
|
279
|
+
JS_HEADER + "const out = await Promise.all(tasks.map(t => run(t)));\n"),
|
|
280
|
+
]
|
|
281
|
+
|
|
282
|
+
# A detector that fires on the defect AND on its fix has no discriminating power. Each
|
|
283
|
+
# clean fixture is the half of the evidence that says which one this is.
|
|
284
|
+
CLEAN = [
|
|
285
|
+
("the ordinary correct file", "agent.py",
|
|
286
|
+
PY_HEADER + "for _ in range(10):\n pass\n"
|
|
287
|
+
"r = requests.get('https://example.com', timeout=5)\n"),
|
|
288
|
+
("a fan-out that DOES capture its branches", "agent.py",
|
|
289
|
+
PY_HEADER + "for _ in range(10):\n pass\n"
|
|
290
|
+
"out = await asyncio.gather(*(run(t) for t in tasks), return_exceptions=True)\n"),
|
|
291
|
+
]
|
|
292
|
+
|
|
293
|
+
|
|
220
294
|
def self_test():
|
|
221
295
|
"""Plant each defect and require the matching check to fire.
|
|
222
296
|
|
|
223
297
|
A detector nobody has watched fire is not evidence that it works, and every plant
|
|
224
298
|
asserts it changed something so a reworded fixture fails HERE rather than reporting a
|
|
225
299
|
healthy checker as broken.
|
|
300
|
+
|
|
301
|
+
Two things the shape of this function is deliberate about. A detector that reads two
|
|
302
|
+
languages gets a plant in **each** — one passing shape is not evidence about the
|
|
303
|
+
other. And every run also asserts silence on the CORRECT shape of the same defect,
|
|
304
|
+
which is what separates a detector from a keyword search.
|
|
226
305
|
"""
|
|
227
306
|
import tempfile
|
|
228
|
-
header = ("import requests\n"
|
|
229
|
-
"system_prompt = 'x'\n"
|
|
230
|
-
"tools = [{'name': 't', 'description': 'does a thing'}]\n"
|
|
231
|
-
"messages = []\n")
|
|
232
|
-
cases = {
|
|
233
|
-
"unbounded-loop": header + "while True:\n pass\n",
|
|
234
|
-
"tool-no-description": header + "T = [{'name': 'a', 'description': ''}]\n",
|
|
235
|
-
"swallowed-error": header + "try:\n x = 1\nexcept Exception:\n pass\n",
|
|
236
|
-
"no-timeout": header + "r = requests.get('https://example.com')\n",
|
|
237
|
-
"hardcoded-model": header + "a = 'claude-opus-4'\nb = 'claude-opus-4'\n",
|
|
238
|
-
}
|
|
239
307
|
failures = 0
|
|
240
|
-
for kind, body in
|
|
308
|
+
for label, kind, fname, body in PLANTS:
|
|
241
309
|
FINDINGS.clear()
|
|
242
310
|
with tempfile.TemporaryDirectory() as d:
|
|
243
|
-
|
|
244
|
-
with open(p, "w", encoding="utf-8") as fh:
|
|
311
|
+
with open(os.path.join(d, fname), "w", encoding="utf-8") as fh:
|
|
245
312
|
fh.write(body)
|
|
246
|
-
assert agentish(body), f"PLANT DID NOT LAND: fixture for {
|
|
313
|
+
assert agentish(body), f"PLANT DID NOT LAND: fixture for {label} is not agent-related"
|
|
247
314
|
scan(d)
|
|
248
315
|
got = {f["check"] for f in FINDINGS}
|
|
249
316
|
if kind in got:
|
|
250
|
-
print(f" OK {
|
|
317
|
+
print(f" OK {label}: detected")
|
|
251
318
|
else:
|
|
252
|
-
print(f" FAIL {
|
|
319
|
+
print(f" FAIL {label}: NOT detected (found {sorted(got) or 'nothing'})")
|
|
253
320
|
failures += 1
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
321
|
+
for label, fname, body in CLEAN:
|
|
322
|
+
FINDINGS.clear()
|
|
323
|
+
with tempfile.TemporaryDirectory() as d:
|
|
324
|
+
with open(os.path.join(d, fname), "w", encoding="utf-8") as fh:
|
|
325
|
+
fh.write(body)
|
|
326
|
+
assert agentish(body), f"CLEAN FIXTURE NOT READ: {label} is not agent-related"
|
|
327
|
+
scan(d)
|
|
328
|
+
if FINDINGS:
|
|
329
|
+
print(f" FAIL {label}: produced {len(FINDINGS)} finding(s): "
|
|
330
|
+
f"{[f['check'] for f in FINDINGS]}")
|
|
331
|
+
failures += 1
|
|
332
|
+
else:
|
|
333
|
+
print(f" OK {label}: silent")
|
|
334
|
+
total = len(PLANTS) + len(CLEAN)
|
|
335
|
+
print(f"\nself-test: {total - failures}/{total} passed")
|
|
268
336
|
return 1 if failures else 0
|
|
269
337
|
|
|
270
338
|
|
|
@@ -5,13 +5,14 @@ description: >-
|
|
|
5
5
|
use, an AI pipeline — or when metering and billing the LLM access it burns. Covers tool-
|
|
6
6
|
calling loops, multi-stage pipelines with human checkpoints, provider routing with fallback
|
|
7
7
|
and retry, four-layer memory with confidence decay, context budgets, sub-agent coordination
|
|
8
|
-
and error hierarchies;
|
|
9
|
-
|
|
8
|
+
and error hierarchies; the work as a graph — parallel layers, fake edges, a checker before
|
|
9
|
+
a convergence; for resale: tiered wallets, the single markup boundary, two-phase commit
|
|
10
|
+
across a database and a provider API, spend-delta polling, budget and loop guardrails,
|
|
10
11
|
per-tenant key lifecycle. Triggers - "agent", "orchestrator", "tool calling", "sub-agent",
|
|
11
12
|
"LLM router", "fallback chain", "human in the loop", "memory layer", "LLM billing", "token
|
|
12
|
-
wallet", "агент", "оркестратор", "суб-агент", "роутер моделей", "человек в
|
|
13
|
-
памяти", "биллинг LLM", "
|
|
14
|
-
wording.
|
|
13
|
+
wallet", "checker node", "агент", "оркестратор", "суб-агент", "роутер моделей", "человек в
|
|
14
|
+
цикле", "слой памяти", "биллинг LLM", "граф задач". Not for a single LLM call in a script,
|
|
15
|
+
or for prompt wording.
|
|
15
16
|
---
|
|
16
17
|
|
|
17
18
|
# Agent Orchestrator — Production Best Practices
|
|
@@ -59,7 +60,7 @@ class AgentContext:
|
|
|
59
60
|
connection_config: ... | None # external resource config
|
|
60
61
|
user_id: str | None
|
|
61
62
|
preferred_provider: str | None # e.g. "openrouter"
|
|
62
|
-
model: str | None # e.g. "
|
|
63
|
+
model: str | None # e.g. "<provider>/<model-id>"
|
|
63
64
|
extra: dict[str, Any] # pipeline_action, flags, overrides
|
|
64
65
|
```
|
|
65
66
|
|
|
@@ -143,40 +144,28 @@ else:
|
|
|
143
144
|
|
|
144
145
|
## 3. Meta-Tools (Orchestrator-Level)
|
|
145
146
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
name="query_database",
|
|
151
|
-
description="Query the connected database. Handles SQL generation, validation, execution.",
|
|
152
|
-
parameters=[ToolParameter(name="question", type="string", description="Data question")]
|
|
153
|
-
)
|
|
154
|
-
ASK_USER_TOOL = Tool(
|
|
155
|
-
name="ask_user",
|
|
156
|
-
description="Ask the user a structured clarification question.",
|
|
157
|
-
parameters=[
|
|
158
|
-
ToolParameter(name="question", type="string", ...),
|
|
159
|
-
ToolParameter(name="question_type", type="string",
|
|
160
|
-
enum=["yes_no", "multiple_choice", "free_text"]),
|
|
161
|
-
ToolParameter(name="options", type="string", required=False),
|
|
162
|
-
]
|
|
163
|
-
)
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
**Assemble tools dynamically** based on available capabilities:
|
|
147
|
+
The orchestrator's tools **delegate to sub-agents** rather than execute:
|
|
148
|
+
`query_database` takes a question in natural language and the SQL agent behind it owns
|
|
149
|
+
generation, validation and execution. One parameter, one responsibility, and the caller
|
|
150
|
+
never learns the sub-agent exists.
|
|
167
151
|
|
|
168
152
|
```python
|
|
169
153
|
def get_tools(*, has_db=False, has_kb=False, has_mcp=False) -> list[Tool]:
|
|
154
|
+
"""Assembled per request from the same capability flags that build the prompt (§10)."""
|
|
170
155
|
tools = []
|
|
171
|
-
if has_db:
|
|
172
|
-
|
|
173
|
-
if
|
|
174
|
-
tools.append(SEARCH_CODEBASE)
|
|
175
|
-
if has_mcp:
|
|
176
|
-
tools.append(QUERY_MCP)
|
|
156
|
+
if has_db: tools.extend([QUERY_DB, PROCESS_DATA, MANAGE_RULES, ASK_USER])
|
|
157
|
+
if has_kb: tools.append(SEARCH_CODEBASE)
|
|
158
|
+
if has_mcp: tools.append(QUERY_MCP)
|
|
177
159
|
return tools
|
|
178
160
|
```
|
|
179
161
|
|
|
162
|
+
Two rules that are this layer's and not the prompt's: **a capability the request does not
|
|
163
|
+
have contributes no tool**, and every enum a tool accepts is closed at the schema
|
|
164
|
+
(`ask_user`'s `question_type` is `yes_no | multiple_choice | free_text`, never free
|
|
165
|
+
prose). How to *describe* a tool so the model picks the right one — the sentence naming
|
|
166
|
+
when to use it, and the neighbour it is confused with — is
|
|
167
|
+
`agent-harness/references/tools.md`.
|
|
168
|
+
|
|
180
169
|
---
|
|
181
170
|
|
|
182
171
|
## 4. Sub-Agent Retry and Validation
|
|
@@ -240,8 +229,8 @@ async def detect_complexity_adaptive(question, llm, history) -> bool:
|
|
|
240
229
|
### Pipeline Components
|
|
241
230
|
|
|
242
231
|
```
|
|
243
|
-
QueryPlanner → (single LLM call) → ExecutionPlan (
|
|
244
|
-
StageExecutor → runs stages
|
|
232
|
+
QueryPlanner → (single LLM call) → ExecutionPlan (stages + their depends_on)
|
|
233
|
+
StageExecutor → runs stages in DEPENDENCY LAYERS with validation + retry (§13)
|
|
245
234
|
StageValidator → checks data shape, row bounds, cross-stage consistency
|
|
246
235
|
StageContext → in-memory state (plan, results per stage, user feedback)
|
|
247
236
|
PipelineRun → DB-persisted state for resume/retry across requests
|
|
@@ -250,21 +239,27 @@ PipelineRun → DB-persisted state for resume/retry across requests
|
|
|
250
239
|
### Checkpoint Pattern (Human-in-the-Loop)
|
|
251
240
|
|
|
252
241
|
```python
|
|
253
|
-
for
|
|
254
|
-
|
|
255
|
-
validation = validator.validate(stage, result, stage_ctx)
|
|
242
|
+
for layer in plan.layers(): # Kahn over depends_on — never list order
|
|
243
|
+
results = await run_layer(layer, context) # execute_with_retries per stage, together
|
|
256
244
|
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
if
|
|
260
|
-
|
|
261
|
-
|
|
245
|
+
for i, (stage, result) in enumerate(zip(layer, results)):
|
|
246
|
+
validation = validator.validate(stage, result, stage_ctx)
|
|
247
|
+
if not validation.passed:
|
|
248
|
+
results[i] = await retry_failed_validation(stage, context, validation)
|
|
249
|
+
if results[i] is None:
|
|
250
|
+
return StageFailedResult(stage, validation) # ask user
|
|
262
251
|
|
|
263
|
-
|
|
252
|
+
if len(layer) > 1: # cheap per-stage checks ran first; this
|
|
253
|
+
verdict = checker.check(results) # one is the cross-item gate (§13)
|
|
254
|
+
if not verdict.passed:
|
|
255
|
+
return StageFailedResult(layer, verdict) # nothing converges on a flagged output
|
|
264
256
|
|
|
265
|
-
|
|
257
|
+
for stage, result in zip(layer, results):
|
|
258
|
+
stage_ctx.set_result(stage.id, result)
|
|
259
|
+
|
|
260
|
+
if any(s.checkpoint for s in layer):
|
|
266
261
|
persist_to_db(pipeline_run_id, stage_ctx)
|
|
267
|
-
return CheckpointResult(
|
|
262
|
+
return CheckpointResult(layer, results) # pause for user review
|
|
268
263
|
# User responds: "continue" | "modify" | "retry"
|
|
269
264
|
```
|
|
270
265
|
|
|
@@ -329,37 +324,22 @@ a floor.
|
|
|
329
324
|
mode cross a compaction boundary as copied typed blocks, not prose (§12).
|
|
330
325
|
## 8. Self-Learning Feedback Loops
|
|
331
326
|
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
After every SQL execution cycle, heuristic extractors analyze the attempt sequence:
|
|
335
|
-
|
|
336
|
-
| Extractor | Detects | Creates |
|
|
337
|
-
|-----------|---------|---------|
|
|
338
|
-
| Table preference | Wrong table A fixed to B | "Use `B` instead of `A`" |
|
|
339
|
-
| Column correction | column_not_found → suggested col | "Use `full_name` not `user_name`" |
|
|
340
|
-
| Format discovery | Division by 100/1000 added | "Amounts in cents, divide by 100" |
|
|
341
|
-
| Schema gotcha | `deleted_at IS NULL` added | "Soft-delete: filter active records" |
|
|
342
|
-
| Performance hint | Timeout fixed by LIMIT/date filter | "Always add LIMIT to this table" |
|
|
343
|
-
|
|
344
|
-
LLM-based deep analysis (3+ attempts, 1hr cooldown) for cross-query patterns.
|
|
345
|
-
|
|
346
|
-
### Cycle 2: User Feedback
|
|
327
|
+
Three cycles feed layers 3 and 4, and they differ by what supplies the signal:
|
|
347
328
|
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
# rejected → learning + note + flag stale benchmark
|
|
354
|
-
# Categorize rejection: currency/format → data_format, filter → schema_gotcha,
|
|
355
|
-
# table → table_preference, join → schema_gotcha
|
|
356
|
-
```
|
|
329
|
+
| Cycle | Signal | Produces |
|
|
330
|
+
|---|---|---|
|
|
331
|
+
| **Validation** | the attempt sequence of a call that failed and was then fixed | a learning, extracted by heuristic — the wrong table, a renamed column, a unit divisor, a soft-delete filter, a missing `LIMIT`. Deep LLM analysis only past 3 attempts, on a cooldown |
|
|
332
|
+
| **User feedback** | a thumbs-down, or a data verdict of confirmed / approximate / rejected | a benchmark, a session note with the deviation, or a learning plus a flag on the now-stale benchmark |
|
|
333
|
+
| **Lifecycle** | time, and contradiction | decay, conflict resolution by negation flip, and promotion of a pattern seen on two independent resources |
|
|
357
334
|
|
|
358
|
-
|
|
335
|
+
The extractors, the exact confidence arithmetic and the promotion query live in
|
|
336
|
+
`references/patterns.md` — **Learning Extraction Heuristics**, **Confidence Management**
|
|
337
|
+
and **Cross-Resource Learning Transfer** — and not here, because a decay rate is a
|
|
338
|
+
constant to tune and a constant with two homes is one that will disagree with itself.
|
|
359
339
|
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
340
|
+
**The rule the whole section exists for:** a learning is written from a **contrast** — the
|
|
341
|
+
attempt that failed beside the attempt that worked — never from a single successful run.
|
|
342
|
+
A system that learns from its successes learns its own habits.
|
|
363
343
|
|
|
364
344
|
---
|
|
365
345
|
|
|
@@ -383,39 +363,26 @@ class WorkflowTracker:
|
|
|
383
363
|
# orchestrator:sql_agent, orchestrator:llm_retry, orchestrator:warning
|
|
384
364
|
```
|
|
385
365
|
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
for i in range(0, len(text), chunk_size):
|
|
391
|
-
await tracker.emit(wf_id, "token", "streaming", text[i:i+chunk_size])
|
|
392
|
-
```
|
|
366
|
+
The final answer streams in chunks as `token` events on the same bus — a typing effect is
|
|
367
|
+
a chunked emit, not a second mechanism. What makes the feed reliable rather than decorative
|
|
368
|
+
is in `references/runtime.md`: a monotonic id per event so a reconnecting client can resume,
|
|
369
|
+
and the feed being a **view over the durable trace** rather than the record itself.
|
|
393
370
|
|
|
394
371
|
---
|
|
395
372
|
|
|
396
373
|
## 10. Dynamic System Prompts
|
|
397
374
|
|
|
398
|
-
|
|
375
|
+
**Assemble the prompt from the capabilities that are actually present**, in the same pass
|
|
376
|
+
that assembles the tools (§3): one section naming each live capability, the resource map
|
|
377
|
+
if there is one, the current learnings, then the guidelines. A prompt that describes a
|
|
378
|
+
tool the agent was not given is how a model spends a turn calling something that is not
|
|
379
|
+
there.
|
|
399
380
|
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
if has_connection:
|
|
406
|
-
sections.append("- query_database: ... SQL agent handles everything")
|
|
407
|
-
sections.append("- process_data: ... enrich/aggregate/filter")
|
|
408
|
-
sections.append("- manage_rules: ... CRUD project rules")
|
|
409
|
-
if has_kb:
|
|
410
|
-
sections.append("- search_codebase: ... RAG over indexed code")
|
|
411
|
-
|
|
412
|
-
if table_map:
|
|
413
|
-
sections.append(f"DATABASE TABLES: {table_map}")
|
|
414
|
-
if recent_learnings:
|
|
415
|
-
sections.append(recent_learnings) # "AGENT LEARNINGS: ..."
|
|
416
|
-
sections.append("GUIDELINES: ...") # routing rules, verification protocol
|
|
417
|
-
return "\n".join(sections)
|
|
418
|
-
```
|
|
381
|
+
What belongs in that text, at what altitude, and how to enumerate the vocabulary so the
|
|
382
|
+
agent stops inventing status values is the **`agent-harness`** skill's
|
|
383
|
+
`agent-harness/references/system-prompt.md` — one home, and it is not this one. What is *this* skill's
|
|
384
|
+
is the wiring: the prompt is rebuilt per request from the same capability flags the tool
|
|
385
|
+
list is built from, so the two can never disagree.
|
|
419
386
|
|
|
420
387
|
**Data Verification Protocol** (inject when DB connected):
|
|
421
388
|
- First-time metrics: ask user "Do these numbers match expectations?"
|
|
@@ -461,6 +428,33 @@ transcript.
|
|
|
461
428
|
|
|
462
429
|
---
|
|
463
430
|
|
|
431
|
+
## 13. The Work as a Graph
|
|
432
|
+
|
|
433
|
+
Before the loop, the pipeline or the sub-agents: **decide the shape.** A node is one unit
|
|
434
|
+
of work; an edge is a dependency, and an edge carries data. The full model, the source it
|
|
435
|
+
comes from, and what this host actually executes are in
|
|
436
|
+
[`references/graph-engineering.md`](references/graph-engineering.md).
|
|
437
|
+
|
|
438
|
+
Four rules, and these are the ones that change code:
|
|
439
|
+
|
|
440
|
+
- **Label every edge with what crosses it. No payload, no edge.** Run the fake-edge test
|
|
441
|
+
over any chain you inherited: write the steps as boxes, ask of each arrow whether data
|
|
442
|
+
from A actually enters B, and delete the arrows that only encode the order somebody
|
|
443
|
+
typed. Two or three per workflow is the normal yield.
|
|
444
|
+
- **`depends_on` is a claim, so execute by layer.** §5's executor walked `plan.stages` in
|
|
445
|
+
list order beside a model that declared its dependencies — which serialises a plan that
|
|
446
|
+
went to the trouble of saying it need not be. Kahn the graph; a cycle fails the plan
|
|
447
|
+
rather than deadlocking the run.
|
|
448
|
+
- **A parallel layer needs a checker before its convergence.** Three branches run, one
|
|
449
|
+
returns a hallucination, and the synthesis node cannot tell: it combines all three and
|
|
450
|
+
answers confidently. The checker decides *usable / not usable* and nothing else, and
|
|
451
|
+
the convergence depends on **the checker**, never directly on a branch.
|
|
452
|
+
- **Static unless you can name what forces dynamic.** A graph that picks its own next
|
|
453
|
+
nodes cannot be audited afterwards, because the shape that ran is not the shape anyone
|
|
454
|
+
drew. Where a run has to be explainable, that settles it.
|
|
455
|
+
|
|
456
|
+
---
|
|
457
|
+
|
|
464
458
|
## Checklist — Building a New Orchestrator
|
|
465
459
|
|
|
466
460
|
- [ ] Shared `AgentContext` dataclass with all sub-agents
|
|
@@ -485,6 +479,10 @@ transcript.
|
|
|
485
479
|
- [ ] `ask_user` clarification mechanism
|
|
486
480
|
- [ ] Graceful degradation (partial answers on context overflow or max iterations)
|
|
487
481
|
- [ ] Compaction ladder, tool-pair-safe boundaries, typed carryover, output offload
|
|
482
|
+
- [ ] Every declared dependency names the data it carries — the fake-edge test run once
|
|
483
|
+
- [ ] Plans executed in dependency layers, not in the order the stages were listed
|
|
484
|
+
- [ ] A checker between every parallel layer and the node that consumes it, and that
|
|
485
|
+
checker watched refusing a planted bad input at least once
|
|
488
486
|
|
|
489
487
|
---
|
|
490
488
|
|
|
@@ -496,6 +494,7 @@ stays an index and the two cannot drift apart.
|
|
|
496
494
|
|
|
497
495
|
| File | Read it when |
|
|
498
496
|
|---|---|
|
|
497
|
+
| [`references/graph-engineering.md`](references/graph-engineering.md) | you are deciding the **shape of the work** — the fake-edge test, the diamond, the checker node, static versus dynamic, and what the host actually runs |
|
|
499
498
|
| [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms** under the body |
|
|
500
499
|
| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** |
|
|
501
500
|
| [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** |
|
|
@@ -0,0 +1,331 @@
|
|
|
1
|
+
# Graph engineering — deciding the shape of the work before doing it
|
|
2
|
+
|
|
3
|
+
**Load this when** a job has more than two steps and you are about to write them in a
|
|
4
|
+
line: choosing between a chain and a graph, finding the dependencies that are not real,
|
|
5
|
+
placing a check between a parallel layer and the node that consumes it, or deciding
|
|
6
|
+
whether the structure may be discovered while it runs.
|
|
7
|
+
|
|
8
|
+
**Spec pinned:** *Graph Engineering with Claude*, `https://x.com/Mahaximus_/status/2082442856417956173`
|
|
9
|
+
(published 2026-07-29); Claude Code `CHANGELOG.md` v2.1.154 – v2.1.229, and the Workflow
|
|
10
|
+
tool contract as presented in-session at CLI **2.1.223** · read 2026-08-15
|
|
11
|
+
|
|
12
|
+
`SKILL.md` §2 owns the tool-calling loop and §5 the multi-stage pipeline. Both assume
|
|
13
|
+
the shape is already decided. **This file is how it gets decided**, and it is upstream of
|
|
14
|
+
every constant in the rest of the pack: a threshold tuned inside the wrong shape is a
|
|
15
|
+
well-measured answer to the wrong question.
|
|
16
|
+
|
|
17
|
+
## Contents
|
|
18
|
+
|
|
19
|
+
- [The source, and what this file adds](#the-source-and-what-this-file-adds)
|
|
20
|
+
- [1. Node and edge](#1-node-and-edge)
|
|
21
|
+
- [2. Your loop is already a graph, and most of its edges are fake](#2-your-loop-is-already-a-graph-and-most-of-its-edges-are-fake)
|
|
22
|
+
- [3. The fake-edge test](#3-the-fake-edge-test)
|
|
23
|
+
- [4. The diamond](#4-the-diamond)
|
|
24
|
+
- [5. Two ways a diamond fails silently](#5-two-ways-a-diamond-fails-silently)
|
|
25
|
+
- [6. The checker node](#6-the-checker-node)
|
|
26
|
+
- [7. Static or dynamic](#7-static-or-dynamic)
|
|
27
|
+
- [8. When not to build a graph at all](#8-when-not-to-build-a-graph-at-all)
|
|
28
|
+
- [9. What Claude Code actually executes](#9-what-claude-code-actually-executes)
|
|
29
|
+
- [10. Barrier or no barrier](#10-barrier-or-no-barrier)
|
|
30
|
+
- [11. Project defaults, written once](#11-project-defaults-written-once)
|
|
31
|
+
- [12. The source's four diagrams, and what each one is for](#12-the-sources-four-diagrams-and-what-each-one-is-for)
|
|
32
|
+
- [Where this file disagrees with its source](#where-this-file-disagrees-with-its-source)
|
|
33
|
+
|
|
34
|
+
## The source, and what this file adds
|
|
35
|
+
|
|
36
|
+
The model below — node, edge, the fake-edge test, the diamond, the checker node, static
|
|
37
|
+
versus dynamic — is taken from the article pinned above. It is the clearest short
|
|
38
|
+
statement of the idea available, and the link is kept so the original can be re-read
|
|
39
|
+
rather than remembered through this summary.
|
|
40
|
+
|
|
41
|
+
Four things are **this pack's**, not the source's, and each is marked where it appears:
|
|
42
|
+
|
|
43
|
+
| Added here | Why the source could not carry it |
|
|
44
|
+
|---|---|
|
|
45
|
+
| §9 — what the host actually runs, with version evidence | the article's one operational claim aged out six weeks after publication (see §9) |
|
|
46
|
+
| §10 — the barrier distinction | the article's diamond has a barrier at every convergence; most convergences do not need one |
|
|
47
|
+
| §6 — what a checker costs, and when it is a rubber stamp | a check nobody measures is a node that always says yes |
|
|
48
|
+
| §7 — the auditability rule as a **hard** rule, not a preference | this pack's own doctrine is that a green nobody watched fail is not evidence |
|
|
49
|
+
|
|
50
|
+
## 1. Node and edge
|
|
51
|
+
|
|
52
|
+
**A node is one unit of work.** One input, one output, one job. Not *"research the topic,
|
|
53
|
+
summarise it, and check the sources"* — that is three nodes wearing one name. The
|
|
54
|
+
smaller and more defined the job, the more useful the node, because a node is also the
|
|
55
|
+
unit you retry, cache, review and replace.
|
|
56
|
+
|
|
57
|
+
**An edge is a dependency, and it carries data.** It exists when the second node
|
|
58
|
+
genuinely consumes what the first produced. Not when the second merely *happens after*
|
|
59
|
+
the first.
|
|
60
|
+
|
|
61
|
+
That distinction is the whole discipline. Write it on the edge and it stops being
|
|
62
|
+
abstract: `research --findings--> write --draft--> verify`. **An edge you cannot label
|
|
63
|
+
with what crosses it is not an edge.**
|
|
64
|
+
|
|
65
|
+
## 2. Your loop is already a graph, and most of its edges are fake
|
|
66
|
+
|
|
67
|
+
A prompt that says *"research this, then summarise, then draft"* is a graph — a single
|
|
68
|
+
unbranching chain in which every step waits for its predecessor. It is correct. It is
|
|
69
|
+
also the slowest possible arrangement of that work and the most brittle: one bad step
|
|
70
|
+
takes the whole chain, and nothing runs while any step is running.
|
|
71
|
+
|
|
72
|
+
The first move is therefore not to learn a new structure. It is to look at the one you
|
|
73
|
+
already have and ask which of its waits are real.
|
|
74
|
+
|
|
75
|
+
## 3. The fake-edge test
|
|
76
|
+
|
|
77
|
+
Five minutes, no tooling, and it is the highest-yield thing in this file.
|
|
78
|
+
|
|
79
|
+
1. Write every step as a box.
|
|
80
|
+
2. Draw an arrow between each pair of consecutive steps.
|
|
81
|
+
3. For each arrow ask: **does data from A actually enter B?** — not *"does B come after
|
|
82
|
+
A"*.
|
|
83
|
+
4. Yes → keep it, and **write the payload on the arrow**.
|
|
84
|
+
5. No → delete it. That wait was free to give away and you were paying for it.
|
|
85
|
+
6. Everything with no incoming arrow starts immediately.
|
|
86
|
+
7. Everything with no outgoing arrow is a final output.
|
|
87
|
+
|
|
88
|
+
The tell that the test is being done honestly is step 4: if the payload cell is empty,
|
|
89
|
+
the edge is fake, and the person drawing it now has to say so out loud rather than
|
|
90
|
+
leaving the arrow in place because it looked orderly.
|
|
91
|
+
|
|
92
|
+
**Expect two or three fake edges in any workflow you have not run this against.** The
|
|
93
|
+
classic is *"review file A, then review file B"*: it reads as a sequence, and the review
|
|
94
|
+
of B never once looks at what A returned.
|
|
95
|
+
|
|
96
|
+
## 4. The diamond
|
|
97
|
+
|
|
98
|
+
One node fans out into several independent nodes; those all feed one node that combines
|
|
99
|
+
them. Drawn out, it is a diamond, and it is the shape that makes graphs worth the setup.
|
|
100
|
+
|
|
101
|
+
```
|
|
102
|
+
┌──────────┐
|
|
103
|
+
│ SPLIT │
|
|
104
|
+
└────┬─────┘
|
|
105
|
+
┌─────────────┼─────────────┐
|
|
106
|
+
▼ ▼ ▼ ← parallel layer
|
|
107
|
+
┌─────────┐ ┌─────────┐ ┌─────────┐
|
|
108
|
+
│ source1 │ │ source2 │ │ source3 │
|
|
109
|
+
└────┬────┘ └────┬────┘ └────┬────┘
|
|
110
|
+
└─────────────┼─────────────┘
|
|
111
|
+
▼
|
|
112
|
+
┌──────────┐
|
|
113
|
+
│SYNTHESIZE│ ← convergence
|
|
114
|
+
└──────────┘
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
The convergence waits for the slowest branch, not for the sum of all of them.
|
|
118
|
+
|
|
119
|
+
**Two rules, and both have to hold:**
|
|
120
|
+
|
|
121
|
+
1. **The parallel nodes are genuinely independent** — no fake edge dressed as a real one,
|
|
122
|
+
and no shared mutable state. Two "independent" workers writing one file are one node
|
|
123
|
+
with a race in it.
|
|
124
|
+
2. **The convergence genuinely needs all of them.** If it needs only the first to answer,
|
|
125
|
+
the rest are paid-for waste and you wanted a race, not a diamond.
|
|
126
|
+
|
|
127
|
+
Once you look for it, the shape is everywhere there is a *gather-then-combine*: research,
|
|
128
|
+
multi-file review, market analysis, a fan of checks over one artifact.
|
|
129
|
+
|
|
130
|
+
## 5. Two ways a diamond fails silently
|
|
131
|
+
|
|
132
|
+
Both are failures **of the convergence**, which is why sequential chains do not have
|
|
133
|
+
them.
|
|
134
|
+
|
|
135
|
+
**A bad node goes undetected.** Three branches run, one returns a hallucination, an empty
|
|
136
|
+
result or a misread file, and that output arrives at the synthesis node beside two good
|
|
137
|
+
ones. The synthesis node does not know one of its inputs is wrong. It produces a
|
|
138
|
+
confident answer built partly on garbage. Parallelism bought the speed by deleting the
|
|
139
|
+
checkpoints where a human would have noticed.
|
|
140
|
+
|
|
141
|
+
**The error cascades and dilutes.** In a chain, a bad step produces a visibly bad output.
|
|
142
|
+
At a convergence, the bad output is *mixed* with good ones, so the damage is spread thin
|
|
143
|
+
and the trace back to its source is gone. By the time anything looks wrong, three nodes
|
|
144
|
+
have averaged it into plausibility.
|
|
145
|
+
|
|
146
|
+
Both are the same defect: **the convergence trusts its inputs because they arrived.**
|
|
147
|
+
|
|
148
|
+
## 6. The checker node
|
|
149
|
+
|
|
150
|
+
A node between the parallel layer and the convergence whose only job is to decide whether
|
|
151
|
+
each output may proceed. It synthesises nothing and writes nothing. It answers *is this
|
|
152
|
+
usable* and then passes, flags, retries or drops.
|
|
153
|
+
|
|
154
|
+
Five things it must catch — the list is the contract, and a checker that cannot say which
|
|
155
|
+
of the five it is asserting is not a checker:
|
|
156
|
+
|
|
157
|
+
1. **Empty or null** — the node returned nothing usable.
|
|
158
|
+
2. **Mutually contradictory** — two outputs that cannot both be true.
|
|
159
|
+
3. **Off-topic** — an output that answers a different question than the one asked.
|
|
160
|
+
4. **Under-confident** — a confidence signal below the bar for the downstream decision.
|
|
161
|
+
5. **Malformed** — a shape that will break the convergence node's parsing.
|
|
162
|
+
|
|
163
|
+
Three of the five are code checks (1, 4, 5) and cost nothing; only 2 and 3 need a model.
|
|
164
|
+
Run them in that order — this is `agent-evals` §5's *cheap checks first*, applied to a
|
|
165
|
+
position in the graph rather than to a test suite.
|
|
166
|
+
|
|
167
|
+
**What a checker costs, and how it turns into a rubber stamp — this pack's addition.** A
|
|
168
|
+
checker is a node, so it has the failure mode of every node: it can be wrong. A model
|
|
169
|
+
checker that has never been shown a bad input will pass everything, and a graph with a
|
|
170
|
+
checker that always says yes is strictly worse than one with no checker, because the
|
|
171
|
+
absent checkpoint has been replaced by a false one. So:
|
|
172
|
+
|
|
173
|
+
- **Give it a planted bad input at least once and watch it refuse.** Same rule as any
|
|
174
|
+
other guard in this family.
|
|
175
|
+
- **Record its verdicts as scores with a source** (`agent-evals` §7), or you can never
|
|
176
|
+
ask afterwards how often it fired.
|
|
177
|
+
- **A checker that has never rejected anything is a finding**, not a reassurance.
|
|
178
|
+
|
|
179
|
+
**Wire the convergence to the checker, not to the layer.** The synthesis node depends on
|
|
180
|
+
the checker; the checker depends on the branches. If synthesis also takes a direct edge
|
|
181
|
+
from a branch, the gate has a bypass and the shape is decoration.
|
|
182
|
+
|
|
183
|
+
## 7. Static or dynamic
|
|
184
|
+
|
|
185
|
+
A **static** graph has its nodes and edges decided before it runs. A **dynamic** graph
|
|
186
|
+
grows: a node finishes, looks at what it found, and decides what should come next.
|
|
187
|
+
|
|
188
|
+
| Reach for | When |
|
|
189
|
+
|---|---|
|
|
190
|
+
| **static** | the task repeats and the structure is the same each time |
|
|
191
|
+
| **static** | predictability and speed matter more than flexibility |
|
|
192
|
+
| **static** | **always first** — switch only after the static version hits a wall you can name |
|
|
193
|
+
| dynamic | the scope of the work depends on what is discovered along the way |
|
|
194
|
+
| dynamic | a node must choose its successors from its own output |
|
|
195
|
+
| **never dynamic** | **you will need to audit exactly what ran and why** |
|
|
196
|
+
|
|
197
|
+
The last row is a hard rule in this pack, not a preference. A dynamic graph's executed
|
|
198
|
+
shape is not the shape anybody drew, so *"here is the graph"* and *"here is what
|
|
199
|
+
happened"* stop being the same document — and every claim about the run becomes
|
|
200
|
+
unfalsifiable from the outside. That is the same failure `agent-evals` names when a
|
|
201
|
+
system has no durable trace.
|
|
202
|
+
|
|
203
|
+
**Most workflows that feel like they need a dynamic graph need a better static one.**
|
|
204
|
+
Dynamic is more powerful and much harder to control; it is the second reach, never the
|
|
205
|
+
first.
|
|
206
|
+
|
|
207
|
+
## 8. When not to build a graph at all
|
|
208
|
+
|
|
209
|
+
The honest cost table. A graph is not free, and for a one-off it usually loses:
|
|
210
|
+
|
|
211
|
+
| | Chain | Graph |
|
|
212
|
+
|---|---|---|
|
|
213
|
+
| Time to build | low | higher — the dependencies have to be worked out |
|
|
214
|
+
| Time to run | the sum of the steps | the longest path |
|
|
215
|
+
| Debugging | easy — one line to walk | harder — concurrent state, diluted errors |
|
|
216
|
+
| Mid-run failure | poor, but visible immediately | good **only if** there is a checker |
|
|
217
|
+
| A one-off task | right answer | overkill |
|
|
218
|
+
| Something you run weekly | works | better, and the setup amortises |
|
|
219
|
+
| Growth in task size | does not scale | scales |
|
|
220
|
+
|
|
221
|
+
**Build the graph when the work repeats, or when a mid-run error is expensive enough that
|
|
222
|
+
the checker pays for itself.** Otherwise write the chain and move on — this is
|
|
223
|
+
`agent-harness`'s *start at the simplest thing that works* applied to shape.
|
|
224
|
+
|
|
225
|
+
## 9. What Claude Code actually executes
|
|
226
|
+
|
|
227
|
+
**This section is the pack's, not the source's, and it exists because the source's one
|
|
228
|
+
operational claim has since changed.** The article tells the reader that Claude Code has
|
|
229
|
+
a `workflow` keyword which parses a YAML block of `nodes:` and `depends_on:` and
|
|
230
|
+
parallelises it. Two corrections, both from the vendor's own changelog:
|
|
231
|
+
|
|
232
|
+
| Version | Entry (quoted from `anthropics/claude-code` `CHANGELOG.md`) |
|
|
233
|
+
|---|---|
|
|
234
|
+
| v2.1.154 | "Introducing dynamic workflows: ask Claude to create a workflow and it orchestrates work across tens to hundreds of agents in the background" |
|
|
235
|
+
| v2.1.160 | "Renamed the dynamic-workflow trigger keyword from `workflow` to `ultracode`. The word 'workflow' no longer triggers a run; asking for one in your own words still works" |
|
|
236
|
+
| v2.1.178 | the keyword "trigger[s] only on explicit phrases like 'run a workflow' or 'workflow:', not on any mention of the word" |
|
|
237
|
+
| v2.1.219 | dynamic workflows "default to a medium size guideline (aim for fewer than 15 agents)"; settable via `workflowSizeGuideline` |
|
|
238
|
+
| v2.1.229 | fan-outs "stagger same-prefix sibling agents so subsequent agents read the cached prompt prefix instead of re-paying it" |
|
|
239
|
+
|
|
240
|
+
So the keyword named in the article stopped being the keyword in v2.1.160, and the
|
|
241
|
+
opt-in today is `ultracode` or an explicit phrase.
|
|
242
|
+
|
|
243
|
+
**And the YAML is not what runs.** The host does not parse `nodes:`/`depends_on:`. It
|
|
244
|
+
authors and executes a **script** whose primitives are the real contract:
|
|
245
|
+
|
|
246
|
+
| Primitive | Is | Note |
|
|
247
|
+
|---|---|---|
|
|
248
|
+
| `agent(prompt, opts)` | one subagent | `opts.schema` forces a validated object back, so downstream stages get data, not prose to parse |
|
|
249
|
+
| `parallel(thunks)` | concurrent, **with a barrier** | awaits all; a thrower resolves to `null` rather than rejecting the call |
|
|
250
|
+
| `pipeline(items, ...stages)` | each item through all stages, **no barrier** | item A can be in stage 3 while B is still in stage 1 |
|
|
251
|
+
| `phase(title)` | a progress grouping | display, not semantics |
|
|
252
|
+
| `isolation: "worktree"` | a private checkout per agent | the only safe way to fan out writers |
|
|
253
|
+
|
|
254
|
+
Concurrency is capped at `min(16, cores − 2)` per run, and a run's total agents at 1000.
|
|
255
|
+
Passing 100 items is fine — they queue. **Those two numbers and the primitive table come
|
|
256
|
+
from the tool contract the host presents to the agent, not from the changelog** — a
|
|
257
|
+
different source with a different lifetime, which is why the stamp above names both. Read
|
|
258
|
+
them back from the running host before relying on them.
|
|
259
|
+
|
|
260
|
+
**Why this matters for the model above:** the article's diamond is `parallel()`, and the
|
|
261
|
+
next section is the distinction it does not draw.
|
|
262
|
+
|
|
263
|
+
## 10. Barrier or no barrier
|
|
264
|
+
|
|
265
|
+
**This section is the pack's.** A convergence node is a barrier: nothing downstream of it
|
|
266
|
+
starts until every branch has finished. The article treats that as the definition of a
|
|
267
|
+
diamond. It is actually a *choice*, and the wrong default.
|
|
268
|
+
|
|
269
|
+
A barrier is correct only when the downstream stage needs **cross-item** context:
|
|
270
|
+
|
|
271
|
+
- deduplicating or merging across the whole result set before expensive work;
|
|
272
|
+
- an early exit that depends on the total ("zero findings → skip verification");
|
|
273
|
+
- a stage whose prompt genuinely compares one item against the others — **which is
|
|
274
|
+
exactly what a checker node does**, and is why the checker is a legitimate barrier.
|
|
275
|
+
|
|
276
|
+
A barrier is **not** justified by:
|
|
277
|
+
|
|
278
|
+
- *"I need to flatten or filter the results first"* — do that inside a stage;
|
|
279
|
+
- *"the stages are conceptually separate"* — separate is not the same as synchronised;
|
|
280
|
+
- *"it reads more cleanly"* — the cost is real. With five branches where the slowest takes
|
|
281
|
+
three times the fastest, a barrier idles the four fast ones for two thirds of the wait.
|
|
282
|
+
|
|
283
|
+
The rule: **pipeline by default; barrier when a stage names the cross-item fact it
|
|
284
|
+
needs.** If it cannot name one, it does not need one.
|
|
285
|
+
|
|
286
|
+
## 11. Project defaults, written once
|
|
287
|
+
|
|
288
|
+
Anything you run more than twice deserves its graph conventions recorded where the agent
|
|
289
|
+
reads them (`CLAUDE.md`, or the equivalent for the host), so they are not re-derived per
|
|
290
|
+
session:
|
|
291
|
+
|
|
292
|
+
```markdown
|
|
293
|
+
## Workflow defaults
|
|
294
|
+
|
|
295
|
+
- A node with no declared dependency starts immediately; do not serialise by habit.
|
|
296
|
+
- Every declared dependency names the data it carries. No payload named ⇒ delete the edge.
|
|
297
|
+
- A checker sits between any parallel layer and the node that consumes it, and the
|
|
298
|
+
consumer depends on the checker rather than on the layer.
|
|
299
|
+
- A checker flags; it never silently passes an incomplete output.
|
|
300
|
+
- A node that fails pauses the run and reports; nothing downstream consumes a flagged output.
|
|
301
|
+
- Outputs are files with the node's name; the graph passes paths, not transcripts.
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
The last line is `context-engineering.md`'s *filesystem as context* stated as a graph
|
|
305
|
+
rule: an edge that carries a path costs a few tokens, and an edge that carries a
|
|
306
|
+
transcript costs the window.
|
|
307
|
+
|
|
308
|
+
## 12. The source's four diagrams, and what each one is for
|
|
309
|
+
|
|
310
|
+
The article carries four hand-drawn figures. They are not decoration — each one is doing
|
|
311
|
+
a specific job, and knowing which one saves re-reading the prose:
|
|
312
|
+
|
|
313
|
+
| Figure | What it shows | The job it does |
|
|
314
|
+
|---|---|---|
|
|
315
|
+
| **Cover — "Graph Engineering explained"** | `START` (define the task) → `SPLIT` (break into nodes) → a fan of three workers labelled *research / compare / check* → `CHECKER` (catch errors early) → `OUTPUT` (one clean answer) | The whole argument in one line, and the only figure in which the checker appears as a first-class stage rather than an afterthought |
|
|
316
|
+
| **Node / edge** | Three boxes — `Research` (in: topic, out: findings) → `Write` (in: findings, out: draft) → `Verify` (in: draft, out: final) — with `NODE` and `EDGE` labelled, and **the arrows themselves labelled with the data they carry** | Makes §1 concrete: the payload written on the arrow is what turns "comes after" into "depends on". This is the figure to copy when teaching the model |
|
|
317
|
+
| **The diamond** | One `RESEARCH` node fanning into `SOURCE 1/2/3` (bracketed *parallel layer*), all three converging on `SYNTHESIZE` | The ideal shape, drawn **before** the failure modes — deliberately without a checker, which is what §5 then attacks |
|
|
318
|
+
| **Workflow — how the code runs** | The same shape in code terms: `research_a/b/c` in a parallel layer, three arrows into `checker` annotated *waits for all three*, one arrow from `checker` into `compare` | The repaired shape. Its point is the single edge out of the checker: `compare` depends on the **gate**, not on the branches — §6's last paragraph, drawn |
|
|
319
|
+
|
|
320
|
+
## Where this file disagrees with its source
|
|
321
|
+
|
|
322
|
+
- **The keyword.** The source's `workflow` is `ultracode` since v2.1.160 (§9). Treated as
|
|
323
|
+
a version-dated fact rather than a correction of the author: it was true when written.
|
|
324
|
+
- **The YAML.** The source presents `workflow:` / `nodes:` / `depends_on:` as a syntax the
|
|
325
|
+
host parses. It is a way of *describing* a graph in a prompt, and it works for that; the
|
|
326
|
+
execution contract is the script in §9.
|
|
327
|
+
- **The barrier.** The source's diamond always synchronises. This file makes the barrier a
|
|
328
|
+
decision with a named justification (§10).
|
|
329
|
+
- **The checker's own reliability.** The source introduces the checker and stops. This
|
|
330
|
+
file requires it to have been watched refusing a planted input, and treats a checker
|
|
331
|
+
that has never rejected anything as a finding (§6).
|
|
@@ -248,7 +248,7 @@ Map your public model names to provider ids in **one** function, and give every
|
|
|
248
248
|
provider a default and a fallback:
|
|
249
249
|
|
|
250
250
|
```
|
|
251
|
-
toUpstreamModel(provider, model) // "
|
|
251
|
+
toUpstreamModel(provider, model) // "<public-name>" → "<provider>/<upstream-id>"
|
|
252
252
|
getDefaultModel(provider) // when the caller names none
|
|
253
253
|
getFallbackModels(provider) // ordered, tried on 5xx / overload
|
|
254
254
|
```
|
|
@@ -139,6 +139,10 @@ class ExecutionPlan:
|
|
|
139
139
|
@classmethod
|
|
140
140
|
def from_json(cls, raw: str) -> ExecutionPlan: ...
|
|
141
141
|
|
|
142
|
+
def layers(self) -> list[list[PlanStage]]:
|
|
143
|
+
"""Kahn's algorithm over `depends_on`. Each returned list may run concurrently."""
|
|
144
|
+
...
|
|
145
|
+
|
|
142
146
|
@dataclass
|
|
143
147
|
class StageResult:
|
|
144
148
|
stage_id: str
|
|
@@ -149,6 +153,14 @@ class StageResult:
|
|
|
149
153
|
error: str | None = None
|
|
150
154
|
```
|
|
151
155
|
|
|
156
|
+
**`depends_on` is a claim the executor has to honour.** A plan that declares dependencies
|
|
157
|
+
and is then executed in list order has serialised itself: `stages[3]` waits for
|
|
158
|
+
`stages[2]` whether or not it consumes anything it produced. Execute by **layer** —
|
|
159
|
+
everything whose dependencies are satisfied goes together — and the declaration starts
|
|
160
|
+
paying for itself. Two rules come with it: a cycle is a plan defect and fails the plan
|
|
161
|
+
rather than deadlocking the run, and a layer of more than one stage needs the convergence
|
|
162
|
+
check in `graph-engineering.md` §6 before anything downstream consumes it.
|
|
163
|
+
|
|
152
164
|
---
|
|
153
165
|
|
|
154
166
|
## Validation Loop (SQL Execution)
|
|
@@ -187,21 +199,33 @@ for attempt in range(1, max_retries + 1):
|
|
|
187
199
|
|
|
188
200
|
## Context Window Sizes
|
|
189
201
|
|
|
202
|
+
**Do not ship a table of model ids.** This file carried one until 2026-08-15 — nine
|
|
203
|
+
vendor ids with their windows, and a `DEFAULT_CONTEXT_WINDOW` of 16 000. Every number in
|
|
204
|
+
it was correct when written and none of it survived a year: generations shipped, ids were
|
|
205
|
+
renamed, long-context variants appeared under the same family name, and a system reading
|
|
206
|
+
that table would have sized its budget for a window an order of magnitude smaller than
|
|
207
|
+
the one it was actually given. A lookup table of somebody else's identifiers is a cache
|
|
208
|
+
with no invalidation.
|
|
209
|
+
|
|
210
|
+
**Resolve the window at one boundary instead**, in this order, and let every caller ask
|
|
211
|
+
that boundary rather than a constant:
|
|
212
|
+
|
|
213
|
+
1. **Configuration** — an explicit per-model entry the operator set. It outranks
|
|
214
|
+
everything, because it is the only source that can encode a limit you have chosen (a
|
|
215
|
+
budget cap below the real window, a provider tier).
|
|
216
|
+
2. **The provider** — the model list or metadata endpoint most APIs expose. Fetched once
|
|
217
|
+
per process, cached with a TTL, refreshed on a miss.
|
|
218
|
+
3. **A conservative floor** for a model nothing knows about, plus a **loud log line**
|
|
219
|
+
naming the model. A silent default is how a new model runs at a fraction of its
|
|
220
|
+
window for months with nobody noticing.
|
|
221
|
+
|
|
222
|
+
The floor is a number to be small about, not accurate about: being early to compact costs
|
|
223
|
+
one avoidable rung of the ladder, and being late costs the request.
|
|
224
|
+
|
|
190
225
|
```python
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
"gpt-4-turbo": 128_000,
|
|
195
|
-
"gpt-4": 8_192,
|
|
196
|
-
"gpt-3.5-turbo": 16_385,
|
|
197
|
-
"claude-sonnet-4-20250514": 200_000,
|
|
198
|
-
"claude-3-5-sonnet-20241022": 200_000,
|
|
199
|
-
"claude-3-haiku-20240307": 200_000,
|
|
200
|
-
"claude-3-opus-20240229": 200_000,
|
|
201
|
-
}
|
|
202
|
-
DEFAULT_CONTEXT_WINDOW = 16_000
|
|
203
|
-
|
|
204
|
-
# Token estimation: tiktoken for OpenAI models, ~4 chars/token fallback
|
|
226
|
+
# Token estimation: a real tokenizer where one is available, ~4 chars/token otherwise.
|
|
227
|
+
# The fallback runs LOW on code, JSON and non-Latin text — see context-engineering.md
|
|
228
|
+
# → Estimating what you have left, and apply the padding factor described there.
|
|
205
229
|
def estimate_tokens(text):
|
|
206
230
|
try:
|
|
207
231
|
import tiktoken
|