@archmax-ai/harness 1.1.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. package/README.md +167 -170
  2. package/dist/agent.d.ts +2 -2
  3. package/dist/agent.js +2 -2
  4. package/dist/assembly/compose.js +2 -2
  5. package/dist/assembly/compose.js.map +1 -1
  6. package/dist/authoring-skill/archmax-harness/README.md +13 -15
  7. package/dist/authoring-skill/archmax-harness/SKILL.md +174 -264
  8. package/dist/authoring-skill/archmax-harness/references/backend-integration.md +244 -776
  9. package/dist/authoring-skill/archmax-harness/references/cases.md +178 -53
  10. package/dist/authoring-skill/archmax-harness/references/code-states-and-scripts.md +211 -100
  11. package/dist/authoring-skill/archmax-harness/references/diagnostics.md +57 -78
  12. package/dist/authoring-skill/archmax-harness/references/workflow-schema.md +613 -2006
  13. package/dist/core/deepagents.d.ts +1 -2
  14. package/dist/core/deepagents.js +1 -2
  15. package/dist/core/deepagents.js.map +1 -1
  16. package/dist/kernel/kernel.js +2 -2
  17. package/dist/kernel/kernel.js.map +1 -1
  18. package/dist/machine/lint-spec.js +6 -5
  19. package/dist/machine/lint-spec.js.map +1 -1
  20. package/dist/machine/machine.d.ts +8 -10
  21. package/dist/machine/machine.js +11 -12
  22. package/dist/machine/machine.js.map +1 -1
  23. package/dist/machine/spec-schema.d.ts +1 -4
  24. package/dist/machine/spec-schema.js +66 -47
  25. package/dist/machine/spec-schema.js.map +1 -1
  26. package/dist/machine/tool-names.d.ts +24 -34
  27. package/dist/machine/tool-names.js +43 -63
  28. package/dist/machine/tool-names.js.map +1 -1
  29. package/dist/machine/types.d.ts +11 -10
  30. package/dist/machine/types.js.map +1 -1
  31. package/dist/public/spec.d.ts +5 -5
  32. package/dist/public/spec.js +5 -5
  33. package/dist/public/spec.js.map +1 -1
  34. package/dist/validate/validate.js +3 -3
  35. package/dist/validate/validate.js.map +1 -1
  36. package/dist/workflow/automated.js +6 -1
  37. package/dist/workflow/automated.js.map +1 -1
  38. package/dist/workflow/control-tools.js +0 -2
  39. package/dist/workflow/control-tools.js.map +1 -1
  40. package/dist/workflow/grader-state.d.ts +2 -2
  41. package/dist/workflow/grader-state.js +5 -15
  42. package/dist/workflow/grader-state.js.map +1 -1
  43. package/dist/workflow/middleware.js +2 -26
  44. package/dist/workflow/middleware.js.map +1 -1
  45. package/dist/workflow/on-error.js +0 -1
  46. package/dist/workflow/on-error.js.map +1 -1
  47. package/dist/workflow/parks.js +0 -1
  48. package/dist/workflow/parks.js.map +1 -1
  49. package/dist/workflow/render-prompt.d.ts +4 -5
  50. package/dist/workflow/render-prompt.js +24 -27
  51. package/dist/workflow/render-prompt.js.map +1 -1
  52. package/dist/workflow/state.d.ts +0 -9
  53. package/dist/workflow/state.js +0 -3
  54. package/dist/workflow/state.js.map +1 -1
  55. package/dist/workflow/turn-boundary.js +0 -1
  56. package/dist/workflow/turn-boundary.js.map +1 -1
  57. package/package.json +1 -2
  58. package/skills/archmax-harness/README.md +13 -15
  59. package/skills/archmax-harness/SKILL.md +174 -264
  60. package/skills/archmax-harness/references/backend-integration.md +244 -776
  61. package/skills/archmax-harness/references/cases.md +178 -53
  62. package/skills/archmax-harness/references/code-states-and-scripts.md +211 -100
  63. package/skills/archmax-harness/references/diagnostics.md +57 -78
  64. package/skills/archmax-harness/references/workflow-schema.md +613 -2006
  65. package/dist/authoring-skill/archmax-harness/references/code-state-and-test-scripts.md +0 -1424
  66. package/dist/authoring-skill/archmax-harness/references/wiring.md +0 -119
  67. package/dist/authoring-skill/archmax-harness/references/workflow-yaml.md +0 -402
  68. package/skills/archmax-harness/references/code-state-and-test-scripts.md +0 -1424
  69. package/skills/archmax-harness/references/wiring.md +0 -119
  70. package/skills/archmax-harness/references/workflow-yaml.md +0 -402
@@ -1,1424 +0,0 @@
1
- # Code-state scripts & cases — full reference
2
-
3
- The complete contract and worked examples for the two authored surfaces this
4
- reference covers:
5
-
6
- 1. **Code-state scripts** (`workflows/<slug>/scripts/*.js` — beside the spec
7
- that wires them, on the **authoring plane**) — the scripts a `type: code`
8
- state runs, **hand-authored** `.js` files. `workflow.yaml` wires
9
- `script: scripts/<name>.js` on the code state: the path is
10
- **workflow-relative**, confined to that workflow's `scripts/` directory and
11
- ends in `.js` or `.mjs`, so it resolves to `workflows/<slug>/scripts/<name>.js`.
12
- An absolute path, a `..` segment, or a path outside `scripts/` is a load error
13
- and a `validate` error. They run in the QuickJS sandbox: plain JavaScript with
14
- **top-level `await`**, no Node APIs, no filesystem beyond the `tools.*` bridge.
15
-
16
- **A code-state script is not an agent's script.** The runtime runs a code
17
- state; the agent runs an `archmax_run` source. The two live in different places
18
- on purpose — code-state scripts in `workflows/<slug>/scripts/`, runtime scripts
19
- in `skills/<capability>/scripts/` — and the classes cannot meet: the agent
20
- cannot read `workflows/**` at all, and `archmax_run` is confined to skill
21
- bundles by a non-overridable kernel rule (`script.skill-only`). An agent that
22
- could execute or read the check that judges its work could probe it for a
23
- bypass, which is the whole reason for the split. A code-state script's source
24
- never reaches the model: not in a prompt, not in an event, not in a session
25
- artifact — the next state reads the reason the script passed to
26
- `archmax_advance`, never its code.
27
- 2. **Cases** (`workflows/<name>/tests/*.test.yaml`) — declarative
28
- YAML documents run by `archmax test` and interpreted **host-side**: no
29
- sandbox executes a case. Suite config is the `tests:` block in
30
- `workflow.yaml` (there is no `tests.config.js`).
31
-
32
- Sandbox scripts are self-contained: the only permitted imports are **typed
33
- `@archmax-ai/harness/*` carriers** (stripped before evaluation — see
34
- [Typed imports](#typed-imports)); importing from any other specifier is an
35
- **error**. Each sandbox context gets its own globals — nothing leaks between
36
- them (`SANDBOX_CONTRACT.context` is `"code"` for a code-state script, or `"ptc"`
37
- for the agent's own `archmax_eval`/`archmax_run` code, which gets **no**
38
- `defineCodeState`; `SANDBOX_CONTRACT.version` is `3`).
39
-
40
- ## Contents
41
-
42
- - [Typed imports](#typed-imports)
43
- - [Part 1 — Code-state scripts and agent scripts](#part-1--code-state-scripts-and-agent-scripts)
44
- - [Authoring model: hand-authored](#authoring-model-hand-authored)
45
- - [The code-state script contract](#the-code-state-script-contract)
46
- - [Governance of `tools.*` calls](#governance-of-tools-calls)
47
- - [`archmax_run` scripts: the `args` contract](#archmax_run-scripts-the-args-contract)
48
- - [Scripts and sub-workflow fan-out](#scripts-and-sub-workflow-fan-out)
49
- - [Worked example: a check in front of a state](#worked-example-a-check-in-front-of-a-state)
50
- - [Worked example: a check behind a state](#worked-example-a-check-behind-a-state)
51
- - [Worked example: approve, then end](#worked-example-approve-then-end)
52
- - [When to use a grader state instead](#when-to-use-a-grader-state-instead)
53
- - [Part 2 — Cases (declarative YAML)](#part-2--cases-declarative-yaml)
54
- - [Layout and the prose rules](#layout-and-the-prose-rules)
55
- - [Suite config: the `tests:` block](#suite-config-the-tests-block)
56
- - [Top-level keys](#top-level-keys)
57
- - [Step entries: one flat script](#step-entries-one-flat-script)
58
- - [Assertion grammar](#assertion-grammar)
59
- - [Structural assertions halt the case](#structural-assertions-halt-the-case)
60
- - [Start conditions: trigger, variables, and workspace](#start-conditions-trigger-variables-and-workspace)
61
- - [Recovering a case's inputs from previous runs](#recovering-a-cases-inputs-from-previous-runs)
62
- - [File fixtures: `from:` references](#file-fixtures-from-references)
63
- - [Settle the mocking depth before writing a case](#settle-the-mocking-depth-before-writing-a-case)
64
- - [Worked example: happy path + grade](#worked-example-happy-path--grade)
65
- - [Worked example: declined at entry, leak check](#worked-example-declined-at-entry-leak-check)
66
- - [Worked example: mocking tool results](#worked-example-mocking-tool-results)
67
- - [Worked example: human-in-the-loop decisions (refine loop)](#worked-example-human-in-the-loop-decisions-refine-loop)
68
- - [Worked example: triggered start + fixture file](#worked-example-triggered-start--fixture-file)
69
- - [Migrating a JS case to YAML](#migrating-a-js-case-to-yaml)
70
-
71
- ---
72
-
73
- ## Typed imports
74
-
75
- A code-state script may open with a **typed import** — a type carrier that gives
76
- IDE completion for the state's input, and is **stripped before QuickJS
77
- evaluation** (the runtime implementation is an identity):
78
-
79
- ```js
80
- /** … */
81
- import { defineCodeState } from "@archmax-ai/harness/sandbox";
82
-
83
- export default defineCodeState(async ({ state, trigger, variables, tools }) => {
84
- /* ... */
85
- });
86
- ```
87
-
88
- The same subpath types the input and the history (`CodeStateInput`,
89
- `CodeStateArgs`, `CodeStateFn`, `SandboxTools`, `SessionHistory`,
90
- `HistoryMessage`, `TrailStep`). Importing from any
91
- **non-`@archmax-ai/harness/*`** specifier is an **error** — sandbox scripts are
92
- self-contained. **Bare-globals authoring** (no import) is equally valid:
93
- `defineCodeState` is a prelude global, and a plain
94
- `export default async function ({ tools }) { … }` registers the same way.
95
-
96
- ---
97
-
98
- ## Part 1 — Code-state scripts and agent scripts
99
-
100
- ### Authoring model: hand-authored
101
-
102
- A code state references an **ordinary hand-authored** `.js` file. You DESIGN AND
103
- WRITE the script directly — there is no generation step. `workflow.yaml` only
104
- wires it: the state's type, its script, and its transitions, which are where the
105
- script may send the session:
106
-
107
- ```yaml
108
- states:
109
- screen-request:
110
- type: code
111
- script: scripts/screen-request.js
112
- triggers:
113
- manual:
114
- skills:
115
- allow: [order-data] # the script reads the order records: granted here
116
- on_error: escalation # where a failure of the script routes
117
- transitions:
118
- - to: identify-case
119
- description: The requester is known and asks only about their own company's orders.
120
- - to: decline
121
- description: The request names no known requester, or reaches for another company's orders.
122
- ```
123
-
124
- `script` is **required** on a code state and **refused** on every other type
125
- (both load errors). A code state makes no model call, so `instructions`, `model`,
126
- `budget`, `evidence` and `approvers` on one are `validate` errors: say what the
127
- script does in its JSDoc block, and in the state's `summary`.
128
-
129
- A code-state script **MUST** open with a leading JSDoc block — a title line, a
130
- blank line, and a description of what the script does _and how_: the inputs it
131
- reads, the sequence of checks, and **where it sends the session in each case**
132
- (each `archmax_advance` target and why, each `archmax_raise` code). It is the only
133
- human-facing contract the file has (admin UIs display it instead of the code), and
134
- it is what a reviewer refines when the check is wrong. No `@specification` tag, no
135
- `// archmax:generated` marker, no generation semantics. See
136
- [SKILL.md → Workspace layout](../SKILL.md#workspace-layout-complete--nothing-else-exists).
137
-
138
- - `archmax validate` reports a code-state script problem when the script **file
139
- is missing** from `scripts/` or it has a **foreign (non-`@archmax-ai/harness/*`)
140
- import**. It does **not** check the JSDoc block on any script — the header is
141
- an authoring standard you uphold yourself. (Cases are YAML; there
142
- `archmax validate` **errors** on a missing `description` instead.)
143
- - The runtime never generates scripts: a missing script file at run time fails
144
- the state (fail-closed), routed through its `on_error`.
145
-
146
- ### The code-state script contract
147
-
148
- When the session's position reaches a code state — the agent's `archmax_advance`
149
- into it, another automated state's advance, a human decision, an `on_error`
150
- route, a turn that opens there, an `archmax_reset` back to it — the runtime runs
151
- its script **in place of the next model call**. The agent's model is not called
152
- while the session stands in a code state, so a chain of code states makes no
153
- model call at all. The script's **default export** is called with **one object**,
154
- and the script **moves the session itself**:
155
-
156
- ```js
157
- /**
158
- * Requester screen — reject a request whose requester is not a known customer.
159
- *
160
- * Reads the first user message from the session's history, extracts the first
161
- * email address in it, loads skills/order-data/assets/orders.json (this state
162
- * enables `order-data`), and looks the address up against every order's `email`.
163
- *
164
- * Advances to decline when the request carries no email address, and when the
165
- * address matches no order — each with a reason naming what was missing — and
166
- * otherwise to identify-case.
167
- */
168
- import { defineCodeState } from "@archmax-ai/harness/sandbox";
169
-
170
- export default defineCodeState(async ({ tools }) => {
171
- const { messages } = JSON.parse(await tools.archmax_get_history());
172
- const orders = JSON.parse(await tools.read_file({ file_path: "skills/order-data/assets/orders.json" }));
173
- const request = messages.find((m) => m.role === "user")?.text ?? "";
174
- const email = request.match(/[\w.+-]+@[\w.-]+/)?.[0];
175
- if (!email) return tools.archmax_advance({ to: "decline", reason: "No email address found in the request." });
176
- if (!orders.some((o) => o.email === email)) {
177
- return tools.archmax_advance({ to: "decline", reason: `"${email}" is not in the order database.` });
178
- }
179
- console.log(`Allowed: ${email}`);
180
- await tools.archmax_advance({ to: "identify-case", reason: `${email} is a known requester.` });
181
- });
182
- ```
183
-
184
- **The input** (`CodeStateInput` in `@archmax-ai/harness/sandbox`; the same fields
185
- minus `tools` are also the global `args`):
186
-
187
- | Field | Meaning |
188
- | --- | --- |
189
- | `state` | The code state's own slug. |
190
- | `trigger?` | The id **string** of the trigger the turn carries. It carries no payload — the session's input is in `variables`. |
191
- | `variables` | The session's variables as a plain `name → value` map, including the built-in `trigger`. Structured values are present whole (`variables.order.items[0].sku`). **Read-only**: assigning changes nothing. Write with `tools.archmax_set_variables`, which holds locks, `title` and typed returns exactly as the agent's call does. |
192
- | `tools` | **Exactly this state's tool surface** (async): its grants (the workflow-wide grant included), the host's essential tools, and its controls. A tool answers to its own name (`tools.read_file`, `tools["archmax_workflow_enrich-order"]`) and to its camelCased key (`tools.readFile`); `Object.keys(tools)` lists the camelCased keys. A call with no argument passes `{}` (`tools.archmax_get_history()`, `tools.ls()`). **A name outside the surface is `undefined`.** See [Governance](#governance-of-tools-calls). |
193
-
194
- **The session's history** is not part of the input: the script reads it the way
195
- a grader does, with `archmax_get_history` (`SessionHistory`):
196
-
197
- ```js
198
- const { messages, trail } = JSON.parse(await tools.archmax_get_history());
199
- ```
200
-
201
- It is the session as it stood when the state was entered, built from the
202
- checkpointed session (never a summarized request), whole and oldest first,
203
- across every turn:
204
-
205
- | Field | Meaning |
206
- | --- | --- |
207
- | `messages` | The transcript as `{ role, text, toolCalls?, tool?, id?, note? }`. `role` is `user`, `assistant`, `tool`, `system`, or **`runtime`** — a note the runtime wrote (an arrival, a decision, an error route, an automated state's move; `note` names the kind). An `assistant` entry lists the calls it made in `toolCalls`, as `{ id?, name, args }`; the `tool` entry answering one carries the tool's name in `tool`, the call's `id`, and the **result in `text`**, as the tool returned it (`JSON.parse` it when the tool returns JSON; a failed or refused call's `text` is the error). A script that reacts to narration matches `role === "runtime"`, never a `[bracket]` prefix in a user message. Not windowed: the sandbox's `maxResultChars` and `memoryLimitBytes` bound it. |
208
- | `trail` | The trail as `{ to, kind, reason?, workflow?, status?, ts }`. `kind` is `trigger`, `agent`, `human`, `grader`, `code`, `sub-workflow`, `reset` or `on_error`. Count earlier rounds here: `trail.filter((s) => s.kind === "code" && s.to === "refund-request").length`. |
209
-
210
- Only what the transcript holds is there: a call a script made through `tools.*`
211
- (an agent's `archmax_run`, or a code state's own) is not, and neither is a
212
- grader's.
213
-
214
- **What a code state checks.** The conversation is the model's memory, not
215
- evidence: a code state checks the session's records — `variables`, files, and
216
- the tool results the history carries as the tools returned them. Check a result
217
- itself, never a copy the model wrote into a variable, a file or its reply, and
218
- never call a tool again just to see its result (for a host whose tools reach an
219
- external system, that is a second action). To read a result, find the call in an
220
- `assistant` entry's `toolCalls`, then the `tool` entry with the same `id`:
221
-
222
- ```js
223
- /**
224
- * Status check — the recorded status must be the one get_order returned.
225
- *
226
- * Finds the last get_order call in the session's history and the tool message
227
- * answering it, and reads scratchpad/status.json ({ status }), which the agent
228
- * wrote. Advances to notify-customer when the recorded status is the returned
229
- * one; otherwise back to look-up-order, with the status to record, or with a
230
- * request to look the order up when no get_order call has a readable result.
231
- */
232
- import { defineCodeState } from "@archmax-ai/harness/sandbox";
233
-
234
- export default defineCodeState(async ({ tools }) => {
235
- const { messages } = JSON.parse(await tools.archmax_get_history());
236
- const call = messages.flatMap((m) => m.toolCalls ?? []).findLast((c) => c.name === "get_order");
237
- const answer = call && messages.find((m) => m.role === "tool" && m.id === call.id);
238
- let order;
239
- try {
240
- order = JSON.parse(answer.text);
241
- } catch {
242
- return tools.archmax_advance({ to: "look-up-order", reason: "Look the order up with get_order first." });
243
- }
244
- const recorded = JSON.parse(await tools.read_file({ file_path: "scratchpad/status.json" }));
245
- if (recorded.status !== order.status) {
246
- return tools.archmax_advance({
247
- to: "look-up-order",
248
- reason: `Order ${call.args.id} is ${order.status}: record that status exactly.`,
249
- });
250
- }
251
- await tools.archmax_advance({ to: "notify-customer", reason: `The recorded status ${order.status} is get_order's.` });
252
- });
253
- ```
254
-
255
- The history spans every turn and every entry into a state, so where only the
256
- latest round counts, take the last match, as above. What a later state or turn
257
- needs from the conversation (the person's words, a decision) is still best
258
- stored in a variable or a file by the state that has it, so no check has to
259
- search the transcript for it.
260
-
261
- **The controls** on `tools`:
262
-
263
- | Control | Effect |
264
- | --- | --- |
265
- | `archmax_advance({ to, reason, evidence? })` | Records the move along one of the state's transitions. Checked **at the call**: an undeclared `to` or an unmet `requires` rejects naming the valid targets or the missing variable, and the script may catch it and decide otherwise. `evidence` is accepted only into a human state. The `reason` is what the next state reads. Absent in a terminal state. |
266
- | `archmax_raise({ code, reason })` | Records the end of the session as **failed**, with the code and reason as its outcome's `exit`. |
267
- | `archmax_get_variables({ name?, path? })` / `archmax_set_variables({ variables, lock? })` | Read and write the session's variables, as the agent's calls do. A write is visible to the script's later calls, so a script can set what `requires` asks for and then advance. |
268
- | `archmax_get_history()` | Resolves to the session's history as JSON text, `{ messages, trail }` (above). |
269
-
270
- These are a grader's controls too: both automated types get the same set. There
271
- is no `archmax_wait` and no `archmax_reset`: a code state cannot park, since a
272
- script frame is not durable. To hand work to a person, advance into a human state.
273
-
274
- **How the run settles** — exactly one outcome:
275
-
276
- | The run… | Outcome |
277
- | --- | --- |
278
- | called `archmax_advance` and finished cleanly | **advanced**: the session moves. The trail records `{ to, kind: "code", reason }`, and a runtime note carries the reason to the next state: `[code] '<state>' moved the run to '<to>': <reason>. You are now in '<to>'.` Into a human state the session parks after the one reply-only handoff message. Into another automated state, that state runs next, still with no model call. |
279
- | called `archmax_raise` | **raised**: the session ends `failed` — no `on_error`, no `returns` check, nothing further runs in the turn. |
280
- | finished without either, in a **terminal** state | **completed**: the session completes, held to the trigger's `returns`. |
281
- | threw, timed out, could not be read, imported a foreign module, or — in a **non-terminal** state — finished without either call | **failed**: routed through the state's `on_error`, else the session ends rejected. Finishing undecided fails with `code state '<slug>' finished without calling archmax_advance or archmax_raise`. |
282
-
283
- Three rules hold the outcomes apart:
284
-
285
- - **The return value is not read.** Returning `false`, a string or an object does
286
- nothing; there is no verdict to return. Only the controls decide.
287
- - **One decision per run.** The first accepted `archmax_advance` or `archmax_raise`
288
- is the state's decision, and every later one is refused (it rejects).
289
- - **The decision takes effect when the script finishes.** A script that advances
290
- and then throws **fails**, and the session does not move. A raise wins over a
291
- later error.
292
-
293
- So parse defensively where malformed input should *send the work back*, and let a
294
- genuine infrastructure failure (an unreadable database) throw, so the state fails
295
- closed through its `on_error`.
296
-
297
- Other globals: `console.log(...)` (each line a `code-output` event, which the CLI's
298
- state flow prints under the state), and `defineCodeState(fn)`, an alias for
299
- `export default fn`. A file with **no default export** is evaluated as a plain
300
- script: it reads the input from the global `args` and calls the global `tools`.
301
- **Each run starts clean**: a fresh REPL every time the state is entered, shared
302
- with neither the agent's `archmax_eval` nor another code state. The script is one
303
- evaluation, so `settings.timeoutMs` (default 15000), `memoryLimitBytes`,
304
- `maxPtcCalls` and `maxResultChars` bound it whole, and a cancelled turn stops it at
305
- once.
306
-
307
- `${{…}}` does not apply inside a script: the runtime substitutes references in the
308
- arguments a *model* writes. A script already has the values in `variables` and
309
- composes strings in JavaScript, so `${{…}}` text in a script's `tools.*` call is
310
- ordinary data.
311
-
312
- ### Governance of `tools.*` calls
313
-
314
- Every `tools.*` call is checked by the decision kernel at the moment it runs —
315
- the same kernel, with the same rules, that governs a tool call the model makes
316
- directly. There is no runtime authority for scripts: **both kinds are governed
317
- against the state they run in.**
318
-
319
- | Script kind | Governed against | Must be granted there |
320
- | --- | --- | --- |
321
- | Code-state script (`type: code`) | its own code state: `tools.allow`/`forbid`, `skills.*`, `mounts.*`, plus `allow_always`, `forbid_always`, safety rules, consumer rules | every tool, skill and governed mount it uses |
322
- | `archmax_run` / `archmax_eval` script (the model chose to run it) | the agent's active state, exactly as the model's own calls | every tool it calls |
323
-
324
- So a code state that reads `skills/order-data/assets/orders.json` enables
325
- `order-data` in its own `skills.allow`, and one that calls a host tool names it in
326
- its own `tools.allow`. A tool another state grants is `undefined` on its `tools`;
327
- a bare `forbid` entry removes a tool from it, and a guarded one (`{ tool:
328
- write_file, paths: [...] }`) refuses the matching calls. A grant on a code state
329
- naming `write_todos`, `archmax_eval`, `archmax_run`, `archmax_wait` or
330
- `archmax_reset` is a `validate` error: no automated state is offered them, and
331
- the same tools in the workflow's `allow_always` are left out of its surface.
332
-
333
- Mounts work the same way: a governed mount is reachable from a code state only
334
- when `mounts.allow_always` or that state's own `mounts.allow` grants it, and
335
- writable only where the grant asks `access: read_write`:
336
-
337
- ```js
338
- // workflow.yaml: states.check-contract.mounts.allow: [contracts]
339
- const policy = await tools.read_file({ file_path: "contracts/policy.md" });
340
- ```
341
-
342
- Neither kind may **read** the authoring plane at all. A `tools.*` call naming
343
- `workflows/**` is blocked (`zone.governance-plane`) with a diagnostic naming the
344
- prefix — a script cannot read its own spec, a sibling's, or a grader's
345
- criteria. There is no `task()` in any sandbox context and no `tools.task`: work
346
- that needs a model's judgment is a grader state on the graph.
347
-
348
- A refused call **rejects** inside the script, carrying the governance reason.
349
- The underlying tool never runs whether or not you catch it:
350
-
351
- ```js
352
- try {
353
- await tools.write_file({ file_path: "scratchpad/report.md", content: body });
354
- } catch (err) {
355
- console.log(`could not write the report: ${err.message}`);
356
- }
357
- ```
358
-
359
- A **binary file** is not a refusal and does not throw: reads are text only, so
360
- `tools.read_file` on an image, audio, video, PDF or PowerPoint file, or on any
361
- file whose content holds a NUL byte (an archive, an Office document), resolves to
362
- the notice `Error: '<path>' is a binary file (<type>, <size>) and was not read;
363
- read_file returns text files only.` instead of content — an image too, even when
364
- the host turned on `images`, which shows images to the model only. XML and JSON
365
- are text even when the store labels them `application/xml` or a `+xml`/`+json`
366
- type, so an Office document's unpacked `word/document.xml` reads as text. A missing file reads as
367
- an `Error: …` string the same way. A script that parses what it read checks
368
- first:
369
-
370
- ```js
371
- const text = await tools.read_file({ file_path: "scratchpad/upload.json" });
372
- const upload = text.startsWith("Error:") ? null : JSON.parse(text);
373
- ```
374
-
375
- To **duplicate, move or delete** a file — a template, an image, a `.docx` —
376
- call `tools.copy_file({ source, destination, overwrite? })`,
377
- `tools.move_file({ source, destination, overwrite? })` or
378
- `tools.remove_file({ file_path })` instead of reading and writing it back. The
379
- bytes move, binary files included, with no `read_file` line limit, and each
380
- call resolves to one line (`Copied '<source>' to '<destination>' (<size>).`,
381
- `Moved …`, `Removed '<path>'.`). They are governed by their declared paths like
382
- any other call, so a governance refusal (a destination in a read-only mount, a
383
- disabled skill's file) rejects. A failure the operation itself meets (a missing
384
- source, a directory, an existing destination without `overwrite: true`)
385
- resolves to an `Error: …` string instead.
386
-
387
- **Authoring rule for every script:** if sandboxed code calls `tools.X`, the state
388
- it runs in must cover `tool: X` (with any argument constraints the call
389
- satisfies), or `tools.X` is `undefined` (or the call rejects). Grant it per state
390
- with `tools.allow`, or workflow-wide with `tools.allow_always`; a default tool such
391
- as `read_file` is covered wherever the workflow-wide grant includes it.
392
-
393
- ### `archmax_run` scripts: the `args` contract
394
-
395
- An `archmax_run` source (`skills/<capability>/scripts/*.js`) is a plain script:
396
- top-level `await`, `tools.*`, `console.log`, and the **last expression** is its
397
- result — returned to the model as the tool result, never into the variable
398
- store. It reads its input from the global **`args`**:
399
-
400
- - **the arguments the model passed** to `archmax_run` (`args.<name>`), plus
401
- - **`args.variables`** — the session's variables as a flat `name → value` map,
402
- the same map a code state gets. The runtime supplies it on every call, so a
403
- script reads the run's input (a seeded list, a delivered payload) without the
404
- model having to retype it.
405
-
406
- ```js
407
- /** One sub-run per requested order. */
408
- const orders = args.variables?.orders_to_enrich ?? [];
409
- ```
410
-
411
- `args.variables` is a read-only snapshot; a script that must persist a result
412
- returns it and the state's `requires:` makes the model record it with
413
- `archmax_set_variables`. An agent's script reaches **no control tool** — not
414
- `archmax_advance`, not the variable tools — and there is **no `task()`** in an
415
- `archmax_run` script or in `archmax_eval`. The model has no `task` tool at all: it
416
- is disclosed in no state and grantable by nothing, because judging work is a
417
- grader state and delegating it is a sub-workflow.
418
-
419
- ### Scripts and sub-workflow fan-out
420
-
421
- A script **is** the deterministic way to drive a fan-out. Delegation tools are on the `tools.*`
422
- surface — a delegation does work and returns a value, unlike the control tools an agent's script
423
- may never call — so a script decides how many sub-runs to start, with what inputs, and what to do
424
- with each answer:
425
-
426
- ```js
427
- /** One sub-run per delayed order, all in flight at once. */
428
- const orders = args.variables?.orders_to_enrich ?? [];
429
-
430
- const enriched = await Promise.all(
431
- // The bridge camelCases tool names, and a tool result arrives as text.
432
- orders.map(async (o) => {
433
- const answer = await tools.archmaxWorkflowEnrichOrder({ order_id: o.order_id });
434
- return { order_id: o.order_id, ...JSON.parse(answer).returns };
435
- }),
436
- );
437
-
438
- enriched;
439
- ```
440
-
441
- `Promise.all` genuinely parallelizes: the bridge starts each host call detached and returns its
442
- promise handle immediately. The runtime still bounds how many sub-runs are in flight
443
- (the dispatcher's `maxConcurrent`), queueing the rest. The list never enters the model's
444
- context. A code state may drive the same fan-out when its own state grants the delegation tool,
445
- and then record the results itself with `tools.archmax_set_variables` before it advances.
446
-
447
- Two limits still hold, both inherited rather than special to delegation. A script may only call a
448
- delegation tool its **calling state** allows — the same kernel verdict an agent's call gets. And a
449
- script's delegation **cannot park**: a QuickJS frame is not durable, so a child that reaches a human
450
- state fails that call closed with `parked`.
451
-
452
- An agent's script still may not call `archmax_advance` (moving the state machine is the model's
453
- governed decision) or `archmax_set_variables`. Concurrent calls succeed or fail **independently**,
454
- so a script can act on what worked — `Promise.allSettled` is the honest shape when partial success
455
- is meaningful. Each sub-run is its own session with its own `scratchpad/`, so concurrent sub-runs
456
- never collide on a path; a `file` return comes back as a copy under a name no other file holds.
457
-
458
- A script lives as long as its turn. When the host cancels the turn, a call still in flight fails
459
- with `Tool '<name>' failed: the turn was aborted`, and every call the script makes after that is
460
- refused the same way before it reaches the tool. A retry in a `catch` therefore dispatches nothing
461
- more, so there is no need to tell a cancel apart from an ordinary failure. Code-state scripts stop
462
- the same way.
463
-
464
- ### Worked example: a check in front of a state
465
-
466
- To screen every request before any model call, make the trigger's entry state a
467
- code state that advances to where the work starts, or to a state that declines it.
468
- The bundled example's `screen-request` (`workflows/order-lookup/scripts/screen-request.js`)
469
- merges a requester check and a tenancy check — wired as in
470
- [Authoring model](#authoring-model-hand-authored), with `decline` a terminal agent
471
- state that tells the person why:
472
-
473
- ```js
474
- /**
475
- * Request screen — the requester must be known, and may only reach their own
476
- * company's orders.
477
- *
478
- * Binds the requester's email (the first one in the first user message) to the
479
- * customers whose orders carry it in skills/order-data/assets/orders.json, then
480
- * checks the request two ways: it names another customer, or it references an
481
- * order id (`ORD-…`) owned by another customer. Matching is case-insensitive.
482
- *
483
- * Advances to decline — with the reason the decline state tells the person — when
484
- * there is no email, the email is unknown, or either tenancy check finds a
485
- * foreign customer; otherwise to identify-case. An order database that cannot
486
- * be read throws: the state fails closed through its on_error.
487
- */
488
- import { defineCodeState } from "@archmax-ai/harness/sandbox";
489
-
490
- export default defineCodeState(async ({ tools }) => {
491
- const { messages } = JSON.parse(await tools.archmax_get_history());
492
- const orders = JSON.parse(await tools.read_file({ file_path: "skills/order-data/assets/orders.json" }));
493
- if (!Array.isArray(orders)) throw new Error("skills/order-data/assets/orders.json is not a list of orders");
494
-
495
- const request = messages.find((m) => m.role === "user")?.text ?? "";
496
- const email = (request.match(/[\w.+-]+@[\w.-]+\.\w{2,}/)?.[0] ?? "").toLowerCase();
497
- if (!email) {
498
- return tools.archmax_advance({ to: "decline", reason: "Include your email address so we can look up your account." });
499
- }
500
- const own = new Set(
501
- orders.filter((o) => String(o.email).toLowerCase() === email).map((o) => String(o.customer)),
502
- );
503
- if (own.size === 0) {
504
- return tools.archmax_advance({ to: "decline", reason: `"${email}" is not in the order database.` });
505
- }
506
-
507
- const lower = request.toLowerCase();
508
- const foreignByName = [...new Set(orders.map((o) => String(o.customer)))].filter(
509
- (name) => !own.has(name) && lower.includes(name.toLowerCase()),
510
- );
511
- const referenced = new Set((request.match(/\bORD-\d+\b/gi) ?? []).map((id) => id.toUpperCase()));
512
- const foreignById = orders
513
- .filter((o) => referenced.has(String(o.id).toUpperCase()) && !own.has(String(o.customer)))
514
- .map((o) => String(o.id));
515
-
516
- if (foreignByName.length > 0 || foreignById.length > 0) {
517
- return tools.archmax_advance({
518
- to: "decline",
519
- reason: `"${email}" may only view orders for its own account. Do not name the other customer's orders.`,
520
- });
521
- }
522
- await tools.archmax_advance({ to: "identify-case", reason: `${email} is authorized for ${[...own].join(", ")}.` });
523
- });
524
- ```
525
-
526
- The pattern to internalize: **one `archmax_advance` per outcome, each with a
527
- `reason` the next state can act on** (the `decline` agent reads it from the `[code]`
528
- note and tells the person); `console.log` for anything worth seeing in a verbose
529
- run; a `throw` for what should fail the state rather than route it. To end the
530
- session as a failure instead, with no reply at all, call
531
- `tools.archmax_raise({ code: "unknown-requester", reason })`.
532
-
533
- ### Worked example: a check behind a state
534
-
535
- To check the agent's work before the session moves on, route the agent's
536
- transition into a code state, and give that state an edge onward and an edge
537
- **back**. The bundled example's `check-refund` sits between `refund-request` (the
538
- agent records a decision) and `refund-review` (a person reviews it), so a decision
539
- the policy forbids never reaches the reviewer:
540
-
541
- ```yaml
542
- states:
543
- refund-request:
544
- instructions: >-
545
- … Record the decision in scratchpad/refund.json, tell the customer, and in
546
- that same message archmax_advance to check-refund. If the check sends the
547
- case back, its note names the fix: correct the file and advance again.
548
- skills:
549
- allow: [order-data, refund-policy]
550
- transitions:
551
- - to: check-refund
552
- description: The refund decision has been recorded and explained to the customer.
553
- check-refund:
554
- type: code
555
- script: scripts/check-refund.js
556
- skills:
557
- allow: [order-data]
558
- on_error: escalation
559
- transitions:
560
- - to: refund-review
561
- description: The recorded decision matches the refund policy.
562
- - to: refund-request
563
- description: The recorded decision contradicts the policy; send it back with the fix.
564
- - to: escalation
565
- description: The decision still fails the check after two corrections.
566
- ```
567
-
568
- The check (`workflows/order-lookup/scripts/check-refund.js`), which counts its own
569
- earlier send-backs on the trail and escalates on the third:
570
-
571
- ```js
572
- /**
573
- * Refund policy check — the agent may not hand a refund the policy forbids to
574
- * the reviewer.
575
- *
576
- * Re-derives the refund decision from the order database instead of trusting
577
- * the one the agent wrote. Reads the recorded ticket from scratchpad/refund.json
578
- * ({ orderId, decision }), then reads skills/order-data/assets/orders.json and
579
- * looks the order id up, case-insensitively, against each record's `id`.
580
- *
581
- * Policy: a refund is "approved" only when the order exists with status
582
- * delivered or delayed; an unknown order and any other status must be "denied".
583
- *
584
- * Advances to refund-review when the recorded decision matches. Otherwise sends
585
- * the session back to refund-request with a reason naming the fix — the ticket
586
- * missing or unparseable, `decision` not "approved"/"denied", or the decision
587
- * contradicting the policy — and, once it has sent it back twice (counted on
588
- * the trail), advances to escalation instead. An unreadable order database
589
- * throws: the state fails through its on_error.
590
- */
591
- import { defineCodeState } from "@archmax-ai/harness/sandbox";
592
-
593
- const MAX_SEND_BACKS = 2;
594
-
595
- export default defineCodeState(async ({ tools }) => {
596
- const { trail } = JSON.parse(await tools.archmax_get_history());
597
- const sentBack = trail.filter((s) => s.kind === "code" && s.to === "refund-request").length;
598
- const sendBack = (reason) =>
599
- sentBack >= MAX_SEND_BACKS
600
- ? tools.archmax_advance({ to: "escalation", reason: `Still failing after ${sentBack} corrections: ${reason}` })
601
- : tools.archmax_advance({ to: "refund-request", reason });
602
-
603
- let refund;
604
- try {
605
- refund = JSON.parse(await tools.read_file({ file_path: "scratchpad/refund.json" }));
606
- } catch (e) {
607
- return sendBack(`scratchpad/refund.json could not be read or parsed (${e.message}) - write it as a JSON object with "orderId" and "decision", then advance again`);
608
- }
609
- const decision = refund?.decision;
610
- if (decision !== "approved" && decision !== "denied") {
611
- return sendBack(`scratchpad/refund.json "decision" must be exactly "approved" or "denied"; got ${JSON.stringify(decision)}`);
612
- }
613
-
614
- const orders = JSON.parse(await tools.read_file({ file_path: "skills/order-data/assets/orders.json" }));
615
- const wanted = typeof refund.orderId === "string" ? refund.orderId.trim().toLowerCase() : "";
616
- const order = orders.find((o) => typeof o?.id === "string" && o.id.toLowerCase() === wanted);
617
- const derived = order && ["delivered", "delayed"].includes(order.status) ? "approved" : "denied";
618
- console.log(`Refund policy check - derived "${derived}", recorded "${decision}"`);
619
-
620
- if (decision === derived) {
621
- return tools.archmax_advance({ to: "refund-review", reason: `"${decision}" for ${refund.orderId} matches the refund policy.` });
622
- }
623
- const why = !order ? "it does not exist in the order database" : `its status is "${order.status}"`;
624
- return sendBack(
625
- `Policy requires "${derived}" for order ${JSON.stringify(refund.orderId)} because ${why}, but the recorded decision is "${decision}" - update scratchpad/refund.json to decision: "${derived}", correct what you told the customer, and advance again`,
626
- );
627
- });
628
- ```
629
-
630
- Note how the send-back reasons carry the _fix_: the agent reads the reason as a
631
- `[code]` note, rewrites `scratchpad/refund.json`, and advances again. There is no
632
- retry budget to declare: the script counts the rounds itself, from the history's
633
- `trail`, and decides when to give up.
634
-
635
- ### Worked example: approve, then end
636
-
637
- A check that approves into a terminal **agent** state costs one more model call:
638
- that state's agent has to finish the turn. Approve into a **terminal code state**
639
- instead. Its script runs and the session completes, with no further model call.
640
- Say the enrichment path's grader approves into `enrichments-approved`, and the
641
- trigger declares `returns: [delayed_order_ids]`:
642
-
643
- ```yaml
644
- enrichments-approved:
645
- type: code
646
- script: scripts/enrichments-approved.js # no transitions: terminal
647
- ```
648
-
649
- ```js
650
- /**
651
- * Approved-enrichments record — the end of the enrichment path.
652
- *
653
- * Reads the `enrichments` the grader approved and sets the trigger's declared
654
- * return `delayed_order_ids` from them, so a caller has the result without
655
- * walking the list, then returns: the session completes.
656
- */
657
- export default async function ({ variables, tools }) {
658
- const delayed = variables.enrichments.filter((e) => e.delayed).map((e) => e.order_id);
659
- await tools.archmax_set_variables({ variables: { delayed_order_ids: delayed } });
660
- }
661
- ```
662
-
663
- A terminal code state's completion is held to the trigger's `returns` like any
664
- completion: set a declared return with `tools.archmax_set_variables` before the
665
- script returns.
666
-
667
- ### When to use a grader state instead
668
-
669
- Use a **code state** when the check is **deterministic** — derivable from
670
- workspace data with plain code (policy tables, schema checks, cross-file
671
- consistency). Use a **grader state** when the check requires **judgment** (is the
672
- report complete? is the summary faithful?). A grader is an isolated subagent the
673
- runtime runs on entry: its system prompt is a compiled-in preamble followed by the
674
- state's `instructions` (its criteria), it reads the session with
675
- `archmax_get_history` (the whole transcript, tool results included, and the
676
- trail) and the work with its own state's tools, and it decides with
677
- `archmax_advance` along one of its transitions, or ends the session with
678
- `archmax_raise`. The bundled example's enrichment path:
679
-
680
- ```yaml
681
- states:
682
- enrich-order:
683
- instructions: >-
684
- Run skills/order-enrichment/scripts/enrich-orders.js with archmax_run …
685
- Record that list as the `enrichments` variable with archmax_set_variables,
686
- then advance to review-enrichment, which checks it. If the review sends it
687
- back, fix what its note names and advance again.
688
- requires: [enrichments]
689
- transitions:
690
- - to: review-enrichment
691
- description: Every order has been enriched and `enrichments` is recorded.
692
- review-enrichment:
693
- type: grader
694
- budget: { maxTurns: 8 } # the grader's own model calls
695
- on_error: escalation # a grader that fails or finishes undecided routes here
696
- skills:
697
- allow: [] # read_file, from the workflow-wide grant, reads the returned files
698
- instructions: |-
699
- You check the enrichments recorded for this run before they are reported.
700
-
701
- 1. Read `orders_to_enrich` and `enrichments` with `archmax_get_variables`.
702
- 2. The enrichments are correct when `enrichments` holds exactly one entry
703
- per requested `order_id` — none missing, none extra — and each entry
704
- has an `enrichment_file` and a boolean `delayed`.
705
- 3. Read each `enrichment_file` with `read_file`: its `delayed` must match
706
- the entry's.
707
-
708
- - Correct: advance to `enrichment-reported`.
709
- - Wrong: advance back to `enrich-order`, naming exactly what is missing or
710
- does not match.
711
- - Before sending it back, call `archmax_get_history` and count the steps
712
- in `trail` with `kind: "grader"` and `to: "enrich-order"`: if there are
713
- already two, raise `enrichment-incorrect` instead.
714
- transitions:
715
- - to: enrichment-reported # a terminal agent state that summarizes the enrichments
716
- description: Every requested order has exactly one enrichment, and each matches its file.
717
- - to: enrich-order
718
- description: >-
719
- The enrichments are incomplete or do not match their files: send them
720
- back, naming exactly what to fix.
721
- ```
722
-
723
- What to know:
724
-
725
- - A grader state **must** declare non-empty `instructions` and at least one
726
- transition (`validate` errors): a grader with nowhere to send the session could
727
- only approve by saying nothing, which would fail open. `evidence`, `approvers`
728
- and `budget.maxParks` are errors on it. Every edge **into** a grader carries a
729
- `description`: its instructions are criteria the work it judges is never shown,
730
- so an undescribed edge into one is a load error.
731
- - `archmax_get_history` returns `{ messages, trail }` — the same history a code
732
- state reads: the whole transcript, a `tool` entry carrying its result in `text`,
733
- and every trail step. Not windowed: a grader judges the whole session, so its
734
- criteria say which part counts ("the files `enrichments` names", "the last
735
- assistant message with text"). Check a claim against the tool result or
736
- the file itself, never against the agent's retelling of it.
737
- - There is **no iteration budget**. A grader that should give up after N rounds
738
- says so in its `instructions` and counts the rounds from the history's trail
739
- (each earlier send-back is `{ kind: "grader", to: "<graded state>" }`), as
740
- above. The turn's recursion limit is the only backstop.
741
- - The grader is offered exactly its state's surface — its grants (never
742
- `write_todos`, `archmax_eval`, `archmax_run`, `archmax_wait` or `archmax_reset`,
743
- even when `allow_always` lists them) and the controls a code state gets:
744
- `archmax_advance`, `archmax_raise`, the variable tools and `archmax_get_history`
745
- — governed against the grader state. Its messages never enter the session's
746
- transcript and its tokens never stream; the session reads its decision as a
747
- `[grader]` note and a `kind: grader` trail step.
748
- - A grader that answers in text without deciding **fails closed** (`grader state
749
- '<slug>' finished without calling archmax_advance or archmax_raise`), routed
750
- through `on_error`. `budget.maxTurns` / `timeoutMs` bound its own model calls.
751
- - `model` on a grader state names the id it runs on, resolved through the
752
- assembly's `modelFactory` under the **`grader`** role.
753
- - The checked state's `instructions` tell the agent that advancing into the
754
- grader is what hands the work in, and to fix the work when a note sends it back
755
- — never to argue with the note.
756
- - **Put a grader where the agent hands work on, not on a live answer.** A turn's
757
- `reply` is every assistant text of the turn, in order, so a draft the grader
758
- sends back has still reached the person. Check work that waits for nobody — a
759
- report, a record, a decision a person reviews next — and let a live answer end
760
- where it is given.
761
-
762
- ---
763
-
764
- ## Part 2 — Cases (declarative YAML)
765
-
766
- ### Layout and the prose rules
767
-
768
- ```
769
- workflows/<name>/tests/
770
- <case>.test.yaml # ONE case per file (.test.yml also accepted)
771
- fixtures/… # fixture files, reachable via `workspace: { <path>: { from: … } }`
772
- ```
773
-
774
- - **One YAML document per file, one case per document — one conversation on
775
- one session.** The case id is the tests/-relative path minus the extension
776
- (`tests/refunds/approved.test.yaml` → case `refunds/approved`); the
777
- `archmax test` filter argument matches that path as a substring.
778
- - **Cases are data, not code.** The host parses, validates, and interprets
779
- them directly — no QuickJS sandbox is created for a case. The sandbox still
780
- runs code-state scripts and `archmax_run` *inside the driven agent*, unchanged.
781
- - **A case reads like a script.** `steps` is one flat sequential list where
782
- actions (`send`, `decide`) and assertions are peers; each assertion
783
- evaluates against the view of the nearest action above it.
784
- - **Only YAML files are cases.** Discovery matches `*.test.yaml`/`*.test.yml`
785
- and nothing else: a leftover `*.test.js`/`*.test.ts` or `tests.config.js` is
786
- ignored **without a diagnostic**, so it neither runs nor warns. Delete such
787
- files once migrated (see
788
- [Migrating a JS case to YAML](#migrating-a-js-case-to-yaml)).
789
- - **Unknown keys are errors.** Unknown top-level keys and step-entry keys
790
- fail validation — a typo'd `reachedstate:` can never silently pass. So do
791
- an assertion with no preceding action, a nested `expect:` block (assertions
792
- are steps themselves), and any multi-session key (`sessions`, `session`,
793
- `parallel` — "not part of test spec version 1"). `archmax validate`
794
- statically checks every case: schema and unknown keys, a missing or
795
- over-long `title`/`description`, malformed regex strings, unknown trigger
796
- ids, and `from:` fixture existence.
797
- - **The title and description rules.** Every case declares two mandatory
798
- prose fields, both **length-capped and enforced** — over budget is an error,
799
- exactly like a missing field:
800
- - **`title`** — a short one-line label, **at most 60 characters**. It is
801
- what `archmax test` prints beside the verdict and what an admin UI lists,
802
- so it has to fit a line: name the behavior under test
803
- (`Unknown requester is rejected at entry`), not the mechanics.
804
- - **`description`** — **at most 200 characters**: one or two sentences
805
- stating the scenario driven and what is asserted. Not a transcript of the
806
- steps, which are right below it and read as a script already.
807
- - Everything longer goes in a **YAML comment above the document** — why the
808
- case exists, what a particular assertion is really pinning, what was
809
- deliberately mocked vs. left real, how the fixture is wired. The comment
810
- has no budget; use it freely.
811
- - Both budgets are measured on whitespace-collapsed text, so a folded
812
- (`>-`) and a literal (`|`) block of the same prose score alike.
813
- - Work title-first: write `title` and `description` before the steps, and
814
- never leave them contradicting each other. This replaces the JSDoc-header
815
- rule that applied to JS test files; JSDoc headers now apply to code-state
816
- scripts and `archmax_run` sources only.
817
-
818
- Run with `archmax test <workflow> [filter]`. A minimal case:
819
-
820
- ```yaml
821
- # The route assertion is the point: without it the case would still pass if
822
- # the agent answered the order question straight from the start state.
823
- title: Order question routed into handle-order
824
- description: >-
825
- One turn asking about an order: the run succeeds and the audit trail
826
- committed the route into handle-order.
827
- steps:
828
- - send: "What is the status of order ORD-1001?"
829
- - succeeded: true
830
- - reachedState: handle-order
831
- ```
832
-
833
- ### Suite config: the `tests:` block
834
-
835
- Declarative data in `workflow.yaml` — never executed code:
836
-
837
- ```yaml
838
- tests:
839
- maxConcurrency: 1 # cases run at a time (only 1 or absent is accepted)
840
- caseTimeoutMs:
841
- 120000 # per-case WALL-CLOCK budget (default 120000) —
842
- # deliberately named differently from settings.timeoutMs,
843
- # the per-script sandbox budget
844
- judge: {} # the model that scores `grade:` ({} = env-configured
845
- # default model; or { model, modelOptions }) — not a grader state
846
- ```
847
-
848
- There is no `reporters` option. A
849
- `judge:` entry is required for cases that declare `grade:` assertions —
850
- without it the assertion records an actionable failure instead of silently
851
- passing.
852
-
853
- ### Top-level keys
854
-
855
- Unknown keys are validation **errors**. All keys except `title` and
856
- `description` are optional:
857
-
858
- | Key | Shape | Meaning |
859
- | ------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
860
- | `title` | string (**required**, ≤ 60 chars, one line) | Short label naming the behavior under test — what `archmax test` prints beside the verdict. Missing, multi-line, or over budget ⇒ `archmax validate` **error**. |
861
- | `description` | string (**required**, ≤ 200 chars) | One or two sentences: the scenario driven and what is asserted. Missing or over budget ⇒ `archmax validate` **error**. Longer rationale belongs in a YAML comment above the document. |
862
- | `skip` | string | A reason; the case reports a skipped verdict and drives nothing. |
863
- | `variables` | `{ <name>: <value> }` | Session variables seeded for every driven turn, supplied as the runtime's `variables` and therefore **locked**. Values may be structured. This is where a case puts the run's input — a trigger carries none. Recover the real payload from a previous run's `artifacts/variables.json` rather than inventing one: see [Recovering a case's inputs from previous runs](#recovering-a-cases-inputs-from-previous-runs). |
864
- | `trigger` | `{ id }` | The trigger every driven turn of the case runs under. Carries **only an id**; `args` is rejected with a pointer to `variables:`. Validated against the workflow machine before any step — an unknown id fails the case (even with no steps), listing the workflow's declared triggers. Omitted ⇒ the manual trigger. See [Start conditions](#start-conditions-trigger-variables-and-workspace). |
865
- | `workspace` | `{ <path>: <content> }` | Files seeded into the case's run workspace **before any step**. Strings are written verbatim; non-string YAML values are JSON-serialized; a single-key `{ from: <tests/-relative path> }` mapping copies a file byte-for-byte. Only agent-visible run-zone paths are seedable (root-level files like `trigger.json`, or `scratchpad/…`); authored (`skills/…`), runtime-internal, and escaping paths are rejected before anything is written. |
866
- | `mocks` | `[{ tool, whenInput?, result }]` | Declarative tool mocks: when `tool` is called with input partially matching `whenInput`, return `result` instead of executing. Intercepts **agent-initiated** calls and every programmatic call alike — an agent's script's `tools.*`, a code state's `tools.*`, a grader's calls. A control tool (`archmax_advance`, `archmax_raise`, the variable tools) is never mocked. See [the mocking section](#settle-the-mocking-depth-before-writing-a-case). |
867
- | `clock` | `"2026-03-20"` / `"…T14:30:00Z"` | The date and time every model call of the case states as "now": an ISO 8601 date (midnight UTC) or a date-time with `Z` or an offset; a zoneless or impossible value is a schema error. Carried on every action, a handoff and a child session included, and outranks the host's `clock` option. See [Start conditions](#start-conditions-trigger-variables-and-workspace). |
868
- | `steps` | flat list of step entries | The case script: actions and assertions as peers (below). |
869
-
870
- There is **no multi-session surface** in test spec version 1 — no `sessions:`,
871
- no `session:` on a step, no `parallel:`, and no case-level `expect:` list;
872
- each is rejected with an actionable error ("not part of test spec version 1").
873
- A case is one conversation on one session; per-session isolation is an engine
874
- invariant covered by the SDK's own unit tests (`src/testing/driver.test.ts`),
875
- not something workflow authors re-test per workflow.
876
-
877
- ### Step entries: one flat script
878
-
879
- `steps` is a single flat, sequential list — a case reads like a script. Every
880
- entry is a **single-key mapping**: an **action**, or an **assertion** that
881
- evaluates against the view of the **nearest action above it** (the completed
882
- turn that action produced). An assertion before the first action is a schema
883
- error. The two actions:
884
-
885
- | Action | Shape | Meaning |
886
- | -------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
887
- | `send` | `send: "<message>"` | One user message — a plain string, nothing else (no transport fields; start conditions live at case scope). Runs the agent until it stops (final answer, park, or failure), under the case's declared trigger. Against a run **parked at a human state** it is a message instead: answered on a reply-only turn (every tool call refused), leaving the run parked at the same state. |
888
- | `decide` | `decide: { to, comment? }` | Resume a **parked** run: the simulated human picks the outgoing transition `to` (a state slug), optionally with a review comment (delivered to the agent as correction on `refine` edges). |
889
-
890
- ```yaml
891
- steps:
892
- - send: "I'd like a refund for order ORD-1001."
893
- - parked: true # <- asserts against the send's turn
894
- - reachedState: refund-review # <- same turn
895
- - send: "Any news?" # a message to the parked run, not a new turn
896
- - parked: decision # still parked: nothing it says routes the run
897
- - decide: { to: refund-closed, comment: "ok" }
898
- - succeeded: true # <- asserts against the decide's turn
899
- ```
900
-
901
- `usedNoTools`/`calledTool` read the **whole conversation's** tool calls, not one
902
- turn's, so do not use them to assert that a reply-only turn called nothing — in a
903
- multi-step case they still see the earlier turns' calls.
904
-
905
- There is no nested `expect:` block — writing one is an error with guidance
906
- that assertions are steps themselves.
907
-
908
- ### Assertion grammar
909
-
910
- Each assertion is a single-key step entry (repeatable; order-preserving),
911
- evaluated against the nearest action above it. **Every assertion is hard** —
912
- `grade` included: the `atLeast` you declare is the bar, and a score under it
913
- fails the case and the run.
914
-
915
- Assertions marked **structural** below assert *where the run went*, and a
916
- failure in one of them **halts the case** — see
917
- [Structural assertions halt the case](#structural-assertions-halt-the-case).
918
-
919
- | Assertion | Meaning |
920
- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
921
- | `succeeded: true` | **Structural.** Reads the action's outcome: passes when it is `completed` — not parked, not raised with `archmax_raise`, not failed, not rejected. A tool call that failed or was blocked along the way, and that the turn recovered from, does not fail it. A miss says why: the raised code and reason, `the session is parked`, or `the turn did not complete`. |
922
- | `parked: true` | **Structural.** The run is parked at a human state awaiting a decision. |
923
- | `raised: true` / `raised: <code>` / `raised: { code?, reason? }` | **Structural.** The agent ended the turn with `archmax_raise({ code, reason })`. A string pins the code exactly; the mapping pins the code and matches the reason — exactly, or as a regex when written `"/pattern/flags"`. A miss names the code that was raised, or says the run did not raise. |
924
- | `variables: { expect, path?, locked? }` | Every named variable is set to the expected value — deep equality, so a structured expectation compares by value. `path` addresses into a value per name (`items.-1.sku` is the last element); `locked` asserts the lock state. |
925
- | `reachedState: <slug>` | **Structural.** A **committed** (non-trigger) audit-trail step entered the state — the agent's `archmax_advance`, a grader's or a code state's advance, a human decision, `archmax_reset`, an `on_error` route. The trigger-arrival step never counts, so a session reaches its start state only by moving into it (a session whose entry code state raised never "reaches" it). |
926
- | `reply: { includes?, excludes? }` | `includes`: substrings or `/pattern/flags` regex strings that must **all** match; `excludes`: none may match (leak checks — a refusal that quotes the data it refuses still leaks it). Each accepts a **single token or a list** (`includes: "ORD-1003"` works). Matched against **every assistant message of the session so far** (earlier turns included), not just the final message (a parked run often closes with a short "moving on" message while the substantive answer came earlier). |
927
- | `calledTool: { name, input? }` | The turn recorded a call to tool `name`; `input` is a **partial** match — mappings match on the declared keys (nested mappings partially), arrays by deep equality, and `/pattern/flags` string values match as **regexes** (recommended for file paths, so the model's exact spelling doesn't matter). |
928
- | `notCalledTool: { name, input? }` | No matching call occurred. |
929
- | `blockedTool: { name, input? }` | A matching call was made **and refused by governance** — the assertion for testing a guard. Distinct from `notCalledTool` (nothing was tried, which proves nothing about the guard) and from a tool that ran and errored (only a policy rejection counts). Use it to test a state's `tools.allow` narrowing or its `skills.allow` set. |
930
- | `usedNoTools: true` | The turn's agent used no tools at all. An automated state's calls are not in the transcript, so they never count. |
931
- | `trail: { to?, kind?, reason?, count }` | **Structural.** The number of audit-trail steps matching **all** given fields equals `count` (declare at least one of `to`/`kind`/`reason`). `kind`: `trigger` \| `agent` \| `human` \| `grader` \| `code` \| `sub-workflow` \| `reset` \| `on_error` — a grader's or code state's move is `grader`/`code`, with the `reason` it passed to `archmax_advance`. `count: 0` asserts a state was never entered. The trail accumulates across turns — each turn starts with a `trigger` step. |
932
- | `noTraversal: true` | **Structural.** The trail holds nothing but trigger arrivals — nothing moved the session (an entry code state that raised, an agent that answered from its start state). A grader's or code state's move is a traversal. |
933
- | `triggerArrival: <trigger-id>` | **Structural.** Some trail step has `kind: trigger` and that trigger id as its reason — the run really started from the declared trigger. |
934
- | `grade: { closedQA, atLeast }` | LLM-grade the turn's **run evidence** against the `closedQA` criterion — what the action's turn said (every text of that turn, empty when it said nothing) *plus* a bounded chronological record of the session's assistant messages and tool calls so far (each call with input, output, and status; mocked calls appear like real ones). The two are weighed differently: a criterion about what the user was **told** is met only by the reply, one about what the agent **did** may be met by the record — so `closedQA: looked the order up before answering` is a legitimate criterion, not only phrasing checks. **`atLeast` decides the outcome** — score ≥ `atLeast` passes, below it **fails the case**; the judge model's own opinion never overrides your declared bar, so the record can't disagree with the verdict. The judge is instructed to keep its reason to 1–3 sentences, so it fits on the verdict line. Requires `tests.judge` in `workflow.yaml`. |
935
-
936
- **Regex form**: a string shaped `/pattern/flags` is compiled as a regular
937
- expression; anything else matches as a plain substring (in `reply` tokens) or
938
- by equality (in tool-input matchers). Malformed regex strings are **static
939
- validation errors**, never silently matched as literals. Quote them in YAML —
940
- `"/approv/i"`, `"/output\\/report\\.json/"`.
941
-
942
- **There is no generic predicate.** No `satisfies`, no inline JS — that is the
943
- point of the format. Bespoke logic belongs in a vitest unit test against the
944
- engine or in a code state on the graph.
945
-
946
- **An automated state is asserted by what it committed.** A code state's or a
947
- grader's own tool calls are not in the transcript, so `calledTool`,
948
- `notCalledTool`, `blockedTool` and `usedNoTools` see only the agent's, and a
949
- grader's words are never part of `reply`. Assert its move and reason with
950
- `reachedState` / `trail: { kind: code | grader, to, reason?, count }`, its raise
951
- with `raised`, what it set with `variables`, and its dispatches with
952
- `ranWorkflow`:
953
-
954
- ```yaml
955
- steps:
956
- - send: "Hi, I'm support@acmecorp.com. Which orders are delayed?"
957
- - trail: { kind: code, to: identify-case, count: 1 } # the entry screen passed it
958
- ```
959
-
960
- ```yaml
961
- trigger: { id: enrichment_requested } # the enrichment path
962
- variables: { orders_to_enrich: [{ order_id: "ORD-1002" }, { order_id: "ORD-1003" }] }
963
- steps:
964
- - send: "Enrich the requested orders."
965
- - trail: { kind: grader, to: enrichment-reported, count: 1 } # the grader approved the enrichments
966
- - trail: { kind: grader, to: enrich-order, count: 0 } # without a send-back
967
- ```
968
-
969
- ### Structural assertions halt the case
970
-
971
- A failed **structural** assertion (`succeeded`, `parked`, `raised`,
972
- `reachedState`, `trail`, `noTraversal`, `triggerArrival`) stops the case where it stands: no
973
- further `send`/`decide` is driven and no further assertion is evaluated. Once
974
- the run took a path the case did not describe, every later step would drive or
975
- grade the wrong run — a `decide` after a failed `parked` cannot succeed, and a
976
- `reply` check after a failed `reachedState` grades a state the run never
977
- entered.
978
-
979
- Content assertions (`reply`, `calledTool`, `notCalledTool`, `blockedTool`, `usedNoTools`,
980
- `grade`) do **not** halt: a wrong reply is a defect in the run, not evidence the
981
- run stopped being the one under test.
982
-
983
- **What this means for how you author a case.** Put the structural assertion you
984
- depend on immediately after the action that should have satisfied it, before the
985
- content assertions that only make sense in that state:
986
-
987
- ```yaml
988
- steps:
989
- - send: "I want a refund for ORD-1001"
990
- - reachedState: refund-review # ← structural: if this misses, stop here
991
- - parked: true
992
- - reply: # ← only meaningful in refund-review
993
- includes: "approved"
994
- - decide: { to: refund-closed }
995
- - succeeded: true
996
- ```
997
-
998
- A halt is reported as an ordinary failed verdict naming the assertion that
999
- missed — **no error is set**, because a halt is a verdict the case reached, not
1000
- a failure of the machinery. Every assertion step that never ran is reported with
1001
- `status: "not-executed"` (one record per step, carrying its `steps` index), so a
1002
- stopped case still accounts for everything it declared: `archmax test` prints an
1003
- `N assertions not executed` line, and a host reading records can grey them out
1004
- instead of showing nothing and implying they passed.
1005
-
1006
- Each record's `status` — `passed` \| `failed` \| `not-executed` — is the
1007
- authoritative outcome; the boolean `pass` is the alias `status === "passed"`.
1008
-
1009
- ### Start conditions: trigger, variables, and workspace
1010
-
1011
- A case declares its **start conditions** — which trigger its driven runs start
1012
- from, what input those runs carry, and which files exist in the run workspace
1013
- before anything happens — at **case scope, never per step**:
1014
-
1015
- ```yaml
1016
- title: Delayed-orders report compiled from a trigger
1017
- description: >-
1018
- The report_requested trigger starts every driven turn, carrying the company
1019
- the firing named as a locked run variable.
1020
- trigger:
1021
- id: report_requested
1022
- variables:
1023
- company: "Acme Corp" # locked, exactly as a host's variables would be
1024
- workspace:
1025
- trigger.json: { company: "Acme Corp" } # non-string ⇒ JSON-serialized
1026
- steps:
1027
- - send: "A delayed-orders report was requested."
1028
- - succeeded: true
1029
- - triggerArrival: report_requested
1030
- ```
1031
-
1032
- - **`trigger`** — every driven turn of the case runs under this trigger
1033
- (each turn restarts the run at that trigger's start state; the trail
1034
- records `{ kind: trigger, reason: <id> }` per turn). Omit it for the
1035
- `manual` trigger. It is validated **before any step executes**: an unknown
1036
- id fails the case with the workflow's declared triggers, even when the case
1037
- has no steps — and `archmax validate` reports the same statically. There is
1038
- deliberately no per-step trigger: a follow-up `send` cannot run under a
1039
- different trigger than the case declares. **When the workflow declares
1040
- non-manual triggers, every case exercising one MUST declare it.**
1041
- - **`variables`** — the run's input, seeded on every driven turn as the
1042
- runtime's `variables` and therefore **locked**, values structured or
1043
- scalar. A seed here is indistinguishable from a production host seed to
1044
- everything downstream: a `${{…}}` reference in a `tools.allow` glob resolves
1045
- from it, a state's `requires:` gate is satisfied by it, a code state's script
1046
- reads it in `variables`, and `archmax_set_variables` refuses to rewrite it.
1047
- **Recover the real payload rather than inventing one** — see
1048
- [Recovering a case's inputs from previous runs](#recovering-a-cases-inputs-from-previous-runs).
1049
- - **`workspace`** — input files seeded before any step. Keys are run-zone
1050
- paths (root-level files like `trigger.json`, or `scratchpad/…`); authored
1051
- paths (`skills/…`) and runtime areas are rejected before anything is
1052
- written. This mirrors the host convention of materializing a trigger's
1053
- payload as a workspace file — declare the payload exactly as production
1054
- would write it.
1055
-
1056
- Two operational notes:
1057
-
1058
- - Seeding writes the file; **reading it is still governed** — the triggered
1059
- start state must allow the read tool the agent needs (`read_file` is granted
1060
- wherever the workflow-wide grant includes it, unless a state narrows it).
1061
- - Hosts that pin case sessions via `sessionIdForCase` reuse sessions across test
1062
- runs: declared files are overwritten each run, but a file seeded by an
1063
- _earlier_ run whose declaration was since removed can linger. The default
1064
- (per-run unique sessions) starts clean.
1065
-
1066
- **Pin the date** with a case-level `clock:` whenever a fixture or a prompt
1067
- carries a date (an order placed on `2026-03-15`, "refundable for 30 days"):
1068
- without it the model reads the real "now" and the verdict drifts with the
1069
- calendar. Pin on a ten-minute mark (the line is rounded down to ten minutes,
1070
- UTC only). Only the prompt is pinned — code-state scripts, the sandbox's `Date`, a
1071
- park's `resumeAt` and every recorded timestamp stay real, so a script that needs
1072
- the case's "today" reads it from a seeded variable, and a wait resumed in the
1073
- case is resumed with `deliver`, not by its `until`.
1074
-
1075
- ### Recovering a case's inputs from previous runs
1076
-
1077
- A trigger carries only an id, so everything a firing passes arrives as
1078
- variables — and those names, shapes, and values live in the host's call site and
1079
- in sessions that already ran, not in the workflow. Read a real run rather than
1080
- guessing:
1081
-
1082
- ```bash
1083
- archmax sessions # ids, status, state, parked state, `vars=` names
1084
- archmax sessions <sessionId> # that session's variables: value + locked state
1085
- ```
1086
-
1087
- That reads the session's **checkpoint**, so it works for any session that has run,
1088
- including one driven by `archmax run` (the CLI emits no run artifacts). A host that
1089
- calls `emitRunArtifacts` leaves the same record on disk as
1090
- `sessions/<sessionId>/artifacts/variables.json`
1091
- (`{ sessionId, workflow, variables: { name: { value, locked } } }`).
1092
-
1093
- Pick the newest session whose recorded `trigger` matches the trigger the case
1094
- drives, then map its store onto the case:
1095
-
1096
- | Recorded entry | Where it belongs |
1097
- | --- | --- |
1098
- | `trigger` (always locked) | the case's `trigger: { id: … }`, never `variables:` |
1099
- | any other `locked: true` | the case's `variables:` block — the firing's payload |
1100
- | `locked: false` | nothing: the agent established it, so **assert** it with `variables: { expect: … }` |
1101
-
1102
- Never copy the whole store into `variables:`. Case seeds are locked, so seeding a
1103
- name the agent should establish hides a missing `archmax_set_variables` call *and*
1104
- makes the agent's own write fail against the lock — the case then exercises a run
1105
- production never performs.
1106
-
1107
- `locked` is the whole attribution, and it is sufficient: a locked entry came from
1108
- the host (an `variables` seed, a delivery's seeds, or the built-in
1109
- `trigger`), an unlocked one from the agent's own `set_variables`. What you read is
1110
- the store as the session's **latest** turn left it, so locked entries are also the
1111
- turn-invariant part.
1112
-
1113
- To produce a payload to read back, drive one yourself:
1114
- `archmax run <workflow> "<prompt>" --variables '{"from_email":"a@b.c"}'` then
1115
- `archmax sessions <id>`.
1116
-
1117
- With no previous run to read, derive the same facts statically: the `${{…}}`
1118
- references in `tools.allow`/`allow_always` and the names in every state's
1119
- `requires:` are the variables the workflow depends on (`archmax validate` warns
1120
- about references nothing guarantees), and the host's
1121
- `createAgent({ variables })` / `deliver(sessionId, { trigger,
1122
- variables })` call sites name what production supplies.
1123
-
1124
- ### File fixtures: `from:` references
1125
-
1126
- Bulk fixtures (a captured `trigger.json`, a large orders dump) live as real
1127
- files under `tests/` rather than inline YAML, and a case names each one it
1128
- wants: any `workspace` entry may declare `{ from: <path> }` instead of inline
1129
- content. The path resolves relative to the workflow's `tests/` directory, so
1130
- `from: fixtures/trigger.json` and `from: shared/orders-dump.json` (a fixture
1131
- shared across cases) both work. There is **no auto-seeded sibling data
1132
- directory**: a file is seeded only when a case's `workspace:` says so.
1133
-
1134
- Fail-closed rules — reported by `archmax validate` and enforced again at run
1135
- bootstrap: a `from:` path that escapes `tests/` or names a missing file is an
1136
- error before any step runs, and a `*.test.yaml` file anywhere under `tests/`
1137
- is a case.
1138
-
1139
- ### Settle the mocking depth _before_ writing a case
1140
-
1141
- An unmocked tool call in a test is a **real** tool call, on every run of the
1142
- suite: a real email sent, a real card charged, a real row written to a live
1143
- system. That is the one decision about a test its author cannot make alone, so
1144
- **ask the owner how much of the workflow's tool surface to mock before you
1145
- write the file**.
1146
-
1147
- Ask concretely rather than in the abstract — name the tools this workflow
1148
- actually calls and propose a split:
1149
-
1150
- - **Mock by default** anything that sends, pays, posts, deletes, or writes
1151
- outside the session zone, and anything whose result drifts between runs (live
1152
- inventory, today's date-dependent data). A test that is not safe to re-run is
1153
- not a test.
1154
- - **Leave real by default** reads against fixtures committed to the workspace,
1155
- the workflow's own scripts and code states, and run-zone file writes — mocking
1156
- those would test the mock instead of the workflow.
1157
- - **Ask** about reads against a live system: cheap and stable enough to keep
1158
- real, or slow, rate-limited, or credential-bound enough to mock?
1159
-
1160
- Record the answer in the **YAML comment above the case document** — the
1161
- `description` is capped at a sentence or two and belongs to what is asserted,
1162
- while a reader checking the case against the workflow needs to know what was
1163
- deliberately left real, which is exactly the kind of rationale the comment
1164
- exists for.
1165
-
1166
- Mocks are the top-level `mocks:` list; they intercept **agent-initiated** calls
1167
- and every programmatic call — an agent's script, a code state's script, a
1168
- grader — with the same partial-input matching as `calledTool`. So a mock of a
1169
- file an entry code state reads changes what that state decides too. Interception is guaranteed under `archmax test`. A host running
1170
- the same cases through `runTests` with its own `createTarget` must wire
1171
- `createToolMockMiddleware()` into that agent — a case that declares `mocks:`
1172
- against a target that cannot intercept them is refused with an error **before
1173
- the agent runs**, never run half-mocked. Mockless cases run against any
1174
- workflow-governed target. See
1175
- [backend-integration.md](backend-integration.md#cases-against-your-agent).
1176
-
1177
- ### Worked example: happy path + grade
1178
-
1179
- ```yaml
1180
- # `reachedState` is what pins the routing: without it the case would still
1181
- # pass if the agent answered straight from the start state. The judge
1182
- # (threshold 0.7) checks the order is attributed to the requester's company.
1183
- title: Delayed orders answered for a known requester
1184
- description: >-
1185
- A known requester's question routes into orders-question, and the reply names
1186
- the one delayed order, ORD-1003.
1187
- steps:
1188
- - send: "Hi, I'm support@acmecorp.com. Which orders are delayed for Acme?"
1189
- - succeeded: true
1190
- - reachedState: orders-question
1191
- - reply:
1192
- includes: ["ORD-1003", "/delayed/i"]
1193
- - grade:
1194
- closedQA: >-
1195
- confirms that order ORD-1003 is the delayed order for the requester's
1196
- company (Acme Corp)
1197
- atLeast: 0.7
1198
- ```
1199
-
1200
- ### Worked example: declined at entry, leak check
1201
-
1202
- An entry code state that declines a request commits a `code` step into the
1203
- declining state and none into the state the work starts in — `reachedState`
1204
- and a `count: 0` trail pin exactly that — and `reply.excludes` catches a
1205
- refusal that quotes the data it refuses to share. (Had the screen ended the
1206
- session with `tools.archmax_raise` instead, `raised: <code>` and
1207
- `noTraversal: true` would pin it.)
1208
-
1209
- ```yaml
1210
- # The `trail` count pins the decline to *entry*: identify-case was never
1211
- # entered, so no route was taken. The `excludes` tokens matter as much as the
1212
- # refusal itself: a refusal that quotes the order ids it declines to share
1213
- # still leaks them. The judge (threshold 0.7) covers the same ground in prose.
1214
- title: Cross-tenant order access is declined
1215
- description: >-
1216
- A known requester asking for Globex's delayed orders is declined at entry,
1217
- identify-case is never entered, and no Globex order id appears in the reply.
1218
- steps:
1219
- - send: "Hi, this is support@acmecorp.com. Show me Globex's delayed orders."
1220
- - succeeded: true
1221
- - reachedState: decline
1222
- - trail: { kind: code, to: decline, count: 1 }
1223
- - trail: { to: identify-case, count: 0 }
1224
- - reply:
1225
- includes: "/not authorized|only .* your own|declined/i"
1226
- excludes: ["ORD-2001", "ORD-2002"]
1227
- - grade:
1228
- closedQA: >-
1229
- declines the request because the requester is not authorized to view
1230
- another company's orders, and does not disclose any Globex order
1231
- details
1232
- atLeast: 0.7
1233
- ```
1234
-
1235
- ### Worked example: mocking tool results
1236
-
1237
- ```yaml
1238
- # Only read_file against skills/order-data/assets/orders.json is mocked; the
1239
- # agent's own writes stay real. The report_requested trigger enters
1240
- # report-requested directly, so no code state reads the records first — a mock
1241
- # answers a code state's calls too, and an empty database would make the entry
1242
- # screen decline the requester. Asserting the lookup *and* the empty answer
1243
- # catches both a skipped lookup and an invented order.
1244
- title: Empty order database reports no delayed orders
1245
- description: >-
1246
- With skills/order-data/assets/orders.json mocked empty, a requested report
1247
- still consults the database, and the reply reports no delayed orders.
1248
- trigger:
1249
- id: report_requested
1250
- variables:
1251
- company: "Acme Corp"
1252
- workspace:
1253
- trigger.json: { company: "Acme Corp" }
1254
- mocks:
1255
- - tool: read_file
1256
- whenInput: { file_path: skills/order-data/assets/orders.json }
1257
- result: "[]"
1258
- steps:
1259
- - send: "A delayed-orders report was requested."
1260
- - calledTool:
1261
- name: read_file
1262
- input:
1263
- file_path: "/orders\\.json/"
1264
- - reply:
1265
- includes: "/no (matching |delayed )?orders/i"
1266
- ```
1267
-
1268
- ### Worked example: mocking a sub-workflow
1269
-
1270
- A delegation is a tool call, so it mocks like one — keyed by the tool's own name,
1271
- matched on the call's arguments. Each call gets its own answer, which is what
1272
- proves a fan-out is not sharing inputs:
1273
-
1274
- ```yaml
1275
- mocks:
1276
- - tool: archmax_workflow_enrich-order
1277
- whenInput: { order_id: "ORD-1002" }
1278
- result:
1279
- message: "ORD-1002 is shipped."
1280
- # The target's `manual` trigger declares these, so a mock standing in
1281
- # for that sub-run has to supply them: a mock replaces the run, never its
1282
- # contract, and one omitting a declared name fails the dispatch.
1283
- returns:
1284
- enrichment_file: "scratchpad/enrichment/ORD-1002.json"
1285
- delayed: false
1286
- steps:
1287
- - send: "Enrich the requested orders."
1288
- - ranWorkflow: { workflow: enrich-order, count: 2 }
1289
- - variables:
1290
- expect:
1291
- enrichments:
1292
- - order_id: "ORD-1002"
1293
- delayed: false
1294
- ```
1295
-
1296
- Assert with **`ranWorkflow`**, not `calledTool`, whenever a script drives the
1297
- fan-out: `calledTool` reads the *model's* calls, and the point of a script-driven
1298
- fan-out is that the model makes none of them. `ranWorkflow` reads the trail, which
1299
- records a step per sub-run whoever made the call — including a mocked one.
1300
-
1301
- ### Worked example: human-in-the-loop decisions (refine loop)
1302
-
1303
- Drive a run into a human state, assert it parked, then decide as the reviewer —
1304
- including the send-back-for-changes (`refine`) loop, whose second parking is
1305
- pinned with a `trail` count:
1306
-
1307
- ```yaml
1308
- # The `trail` count is the point: refund-review is entered exactly twice —
1309
- # initial submit plus one resubmit — so a third entry would mean the refine
1310
- # edge looped. The final `reply` check pins that the correction did not lose
1311
- # its subject.
1312
- title: Refund sent back for changes, then approved
1313
- description: >-
1314
- The reviewer refines once and then approves: the run parks at both review
1315
- points, refund-review is entered exactly twice, the run succeeds, and the
1316
- final reply still names ORD-1001.
1317
- steps:
1318
- - send: "Hi, I'm support@acmecorp.com. I'd like a refund for order ORD-1001."
1319
- - parked: true
1320
- - decide:
1321
- to: refund-request
1322
- comment: >-
1323
- Please restate the refund amount and confirm the order was delivered
1324
- before it can be approved.
1325
- - parked: true
1326
- - trail:
1327
- to: refund-review
1328
- count: 2
1329
- - decide:
1330
- to: refund-closed
1331
- comment: "The explanation is clear now."
1332
- - succeeded: true
1333
- - reply:
1334
- includes: "ORD-1001"
1335
- ```
1336
-
1337
- ### Worked example: triggered start + fixture file
1338
-
1339
- The case declares its trigger on the document, its input as a locked variable,
1340
- and a payload fixture the host would materialize as a real file under `tests/`,
1341
- seeded byte-for-byte before any step:
1342
-
1343
- ```yaml
1344
- # tests/triggered-report.test.yaml — tests/fixtures/trigger.json holds the
1345
- # payload fixture. `triggerArrival` is what pins the start: it names the
1346
- # trigger the run really arrived on, not merely the state it ended up in.
1347
- title: Delayed-orders report compiled from a trigger
1348
- description: >-
1349
- The report_requested trigger starts the run: the agent reads the seeded
1350
- trigger.json, writes scratchpad/report.json, and the reply names Acme's one
1351
- delayed order, ORD-1003.
1352
- trigger:
1353
- id: report_requested
1354
- variables:
1355
- company: "Acme Corp"
1356
- workspace:
1357
- trigger.json: { from: fixtures/trigger.json }
1358
- steps:
1359
- - send: "A delayed-orders report was requested."
1360
- - succeeded: true
1361
- - triggerArrival: report_requested
1362
- # `/pattern/` input values match as regexes, so the assertion holds
1363
- # whether the model spells the path with or without a leading slash.
1364
- - calledTool:
1365
- name: read_file
1366
- input:
1367
- file_path: "/trigger\\.json/"
1368
- - calledTool:
1369
- name: write_file
1370
- input:
1371
- file_path: "/scratchpad\\/report\\.json/"
1372
- - reply:
1373
- includes: ["ORD-1003", "Acme"]
1374
- ```
1375
-
1376
- ### Migrating a JS case to YAML
1377
-
1378
- JS test cases (`defineEval`, the sandbox `t` driver, `@archmax-ai/harness/testing`) are
1379
- **removed**; a `*.test.js`/`*.test.ts` file is no longer recognized and is
1380
- skipped silently. The mapping:
1381
-
1382
- | JS surface (removed) | YAML equivalent |
1383
- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
1384
- | File JSDoc header + `defineEval({ description })` | The mandatory `title` (≤ 60 chars) + `description` (≤ 200 chars); rationale that fits neither moves to a YAML comment above the document |
1385
- | `defineEval({ trigger })` / `({ workspace })` | Top-level `trigger:` / `workspace:` (inline content or `{ from: … }`) |
1386
- | `await t.send("msg")` | `- send: "msg"` step entry |
1387
- | `await t.decide(to, comment)` | `- decide: { to, comment }` step entry |
1388
- | `t.newSession(...)` / `Promise.all` over sessions | **Not part of test spec version 1** — a case is one conversation on one session; per-session isolation is an engine invariant covered by the SDK's own unit tests |
1389
- | `t.mockTool(name, { whenInput, result })` | `mocks: [{ tool, whenInput, result }]` |
1390
- | `t.succeeded()` / `t.parked()` | `- succeeded: true` / `- parked: true` step entries after the action |
1391
- | `t.messageIncludes("x")` / `t.messageIncludes(/x/i)` | `- reply: { includes: ["x", "/x/i"] }` step entry (single token or list) |
1392
- | Leak checks via `satisfies(...)` on the reply | `- reply: { excludes: [...] }` step entry |
1393
- | `t.calledTool(name, { input })` / `t.notCalledTool` / `t.usedNoTools()` | `- calledTool: { name, input? }` / `- notCalledTool:` / `- usedNoTools: true` step entries (RegExp input values become `"/pattern/flags"` strings) |
1394
- | `t.reachedState(name)` | `- reachedState: <name>` step entry |
1395
- | `t.auditTrail()` + `filter(...).length === n` | `- trail: { to?, kind?, reason?, count }`; "nothing but the trigger arrival" ⇒ `- noTraversal: true`; trigger-arrival reason ⇒ `- triggerArrival: <id>` |
1396
- | `t.judge.autoevals.closedQA(c).atLeast(n)` | `- grade: { closedQA: c, atLeast: n }` step entry |
1397
- | `t.skip(reason)` | `skip: <reason>` |
1398
- | bespoke predicates | **No YAML equivalent — deliberate.** Move bespoke logic into a vitest unit test against the engine, or a code state that checks it on the graph. |
1399
- | `import { defineEval } from "@archmax-ai/harness/testing"` | Nothing — the subpath is removed; YAML imports nothing |
1400
-
1401
- Removed package exports: `createDriveTool`, `DriveRequest`, `DriveSession`,
1402
- and the `@archmax-ai/harness/testing` subpath. New root exports for hosts that parse
1403
- cases themselves: `parseCaseDocument`, `CaseSchemaError`, the case types
1404
- (`CaseDocument`, `CaseStep`, `CaseExpectation`, …), and the test-spec
1405
- versioning surface (`DEFAULT_TEST_SPEC_VERSION`, `SUPPORTED_TEST_SPEC_VERSIONS`,
1406
- `resolveTestSpecVersion`, `isTestSpecVersionSupported`,
1407
- `UnsupportedTestSpecVersionError`). `@archmax-ai/harness/sandbox` carries the
1408
- code-state contract (`defineCodeState`, `CodeStateInput`, `CodeStateArgs`,
1409
- `CodeStateFn`, `SandboxTools`, `SessionHistory`, `HistoryMessage`, `TrailStep`).
1410
-
1411
- ---
1412
-
1413
- ### Source of truth in code
1414
-
1415
- | Concern | File in the SDK |
1416
- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
1417
- | Sandbox prelude (the `code` context, `defineCodeState`, the scoped `tools` view) | `src/sandbox/assets/parts/code-state.js`; `src/sandbox/prelude.ts` |
1418
- | Script source preparation (`@archmax-ai/harness/*` import stripping, `export default`, foreign-import errors) | `src/sandbox/imports.ts` |
1419
- | Executor, sandbox tools, PTC gateway | `src/sandbox/executor.ts`, `tools.ts`, `ptc-gateway.ts` |
1420
- | Typed entry point (`defineCodeState`, `CodeStateInput`, `SessionHistory`, `HistoryMessage`, `TrailStep`) | `src/public/sandbox.ts` |
1421
- | Running an automated state, its outcomes and controls | `src/workflow/automated.ts`, `automated-controls.ts` |
1422
- | Code-state and grader runs; the history (`messages`, `trail`) | `src/workflow/code-state.ts`, `grader-state.ts`, `history.ts` |
1423
- | Case engine (host-side) | `src/testing/` — case schema `case-schema.ts`, discovery `discovery.ts`, assertion evaluation `assertions.ts`, grader `grade.ts`, runner `runner.ts` |
1424
- | Validation diagnostics (missing code-state script, foreign import, YAML case checks) | `src/validate/validate.ts` |