@archmax-ai/harness 1.0.1 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +167 -165
- package/dist/agent.d.ts +2 -2
- package/dist/agent.js +2 -2
- package/dist/assembly/compose.js +5 -3
- package/dist/assembly/compose.js.map +1 -1
- package/dist/authoring-skill/archmax-harness/README.md +13 -15
- package/dist/authoring-skill/archmax-harness/SKILL.md +174 -261
- package/dist/authoring-skill/archmax-harness/references/backend-integration.md +244 -764
- package/dist/authoring-skill/archmax-harness/references/cases.md +178 -53
- package/dist/authoring-skill/archmax-harness/references/code-states-and-scripts.md +211 -100
- package/dist/authoring-skill/archmax-harness/references/diagnostics.md +57 -78
- package/dist/authoring-skill/archmax-harness/references/workflow-schema.md +614 -1977
- package/dist/core/attachments.d.ts +2 -0
- package/dist/core/attachments.js +2 -1
- package/dist/core/attachments.js.map +1 -1
- package/dist/core/deepagents.d.ts +1 -2
- package/dist/core/deepagents.js +1 -2
- package/dist/core/deepagents.js.map +1 -1
- package/dist/core/file-operations.d.ts +6 -0
- package/dist/core/file-operations.js +29 -0
- package/dist/core/file-operations.js.map +1 -1
- package/dist/index.d.ts +5 -1
- package/dist/index.js +4 -0
- package/dist/index.js.map +1 -1
- package/dist/kernel/kernel.js +2 -2
- package/dist/kernel/kernel.js.map +1 -1
- package/dist/machine/lint-spec.js +6 -5
- package/dist/machine/lint-spec.js.map +1 -1
- package/dist/machine/machine.d.ts +8 -10
- package/dist/machine/machine.js +11 -12
- package/dist/machine/machine.js.map +1 -1
- package/dist/machine/signature-pattern.d.ts +24 -0
- package/dist/machine/signature-pattern.js +67 -0
- package/dist/machine/signature-pattern.js.map +1 -0
- package/dist/machine/signature.d.ts +137 -35
- package/dist/machine/signature.js +476 -25
- package/dist/machine/signature.js.map +1 -1
- package/dist/machine/spec-schema.d.ts +203 -5
- package/dist/machine/spec-schema.js +209 -51
- package/dist/machine/spec-schema.js.map +1 -1
- package/dist/machine/tool-names.d.ts +24 -34
- package/dist/machine/tool-names.js +43 -63
- package/dist/machine/tool-names.js.map +1 -1
- package/dist/machine/types.d.ts +11 -10
- package/dist/machine/types.js.map +1 -1
- package/dist/public/spec.d.ts +10 -6
- package/dist/public/spec.js +9 -5
- package/dist/public/spec.js.map +1 -1
- package/dist/validate/validate.js +3 -3
- package/dist/validate/validate.js.map +1 -1
- package/dist/workflow/automated.js +6 -1
- package/dist/workflow/automated.js.map +1 -1
- package/dist/workflow/control-tools.js +3 -3
- package/dist/workflow/control-tools.js.map +1 -1
- package/dist/workflow/file-values.d.ts +4 -1
- package/dist/workflow/file-values.js +2 -0
- package/dist/workflow/file-values.js.map +1 -1
- package/dist/workflow/grader-state.d.ts +2 -2
- package/dist/workflow/grader-state.js +5 -15
- package/dist/workflow/grader-state.js.map +1 -1
- package/dist/workflow/middleware.d.ts +2 -0
- package/dist/workflow/middleware.js +3 -26
- package/dist/workflow/middleware.js.map +1 -1
- package/dist/workflow/on-error.js +0 -1
- package/dist/workflow/on-error.js.map +1 -1
- package/dist/workflow/parks.js +0 -1
- package/dist/workflow/parks.js.map +1 -1
- package/dist/workflow/render-prompt.d.ts +4 -5
- package/dist/workflow/render-prompt.js +31 -29
- package/dist/workflow/render-prompt.js.map +1 -1
- package/dist/workflow/signature-checks.d.ts +8 -3
- package/dist/workflow/signature-checks.js +17 -4
- package/dist/workflow/signature-checks.js.map +1 -1
- package/dist/workflow/state.d.ts +0 -9
- package/dist/workflow/state.js +0 -3
- package/dist/workflow/state.js.map +1 -1
- package/dist/workflow/sub-workflow.js +4 -1
- package/dist/workflow/sub-workflow.js.map +1 -1
- package/dist/workflow/turn-boundary.js +0 -1
- package/dist/workflow/turn-boundary.js.map +1 -1
- package/package.json +2 -2
- package/skills/archmax-harness/README.md +13 -15
- package/skills/archmax-harness/SKILL.md +174 -261
- package/skills/archmax-harness/references/backend-integration.md +244 -764
- package/skills/archmax-harness/references/cases.md +178 -53
- package/skills/archmax-harness/references/code-states-and-scripts.md +211 -100
- package/skills/archmax-harness/references/diagnostics.md +57 -78
- package/skills/archmax-harness/references/workflow-schema.md +614 -1977
- package/dist/authoring-skill/archmax-harness/references/code-state-and-test-scripts.md +0 -1424
- package/dist/authoring-skill/archmax-harness/references/wiring.md +0 -119
- package/dist/authoring-skill/archmax-harness/references/workflow-yaml.md +0 -399
- package/skills/archmax-harness/references/code-state-and-test-scripts.md +0 -1424
- package/skills/archmax-harness/references/wiring.md +0 -119
- package/skills/archmax-harness/references/workflow-yaml.md +0 -399
|
@@ -1,1424 +0,0 @@
|
|
|
1
|
-
# Code-state scripts & cases — full reference
|
|
2
|
-
|
|
3
|
-
The complete contract and worked examples for the two authored surfaces this
|
|
4
|
-
reference covers:
|
|
5
|
-
|
|
6
|
-
1. **Code-state scripts** (`workflows/<slug>/scripts/*.js` — beside the spec
|
|
7
|
-
that wires them, on the **authoring plane**) — the scripts a `type: code`
|
|
8
|
-
state runs, **hand-authored** `.js` files. `workflow.yaml` wires
|
|
9
|
-
`script: scripts/<name>.js` on the code state: the path is
|
|
10
|
-
**workflow-relative**, confined to that workflow's `scripts/` directory and
|
|
11
|
-
ends in `.js` or `.mjs`, so it resolves to `workflows/<slug>/scripts/<name>.js`.
|
|
12
|
-
An absolute path, a `..` segment, or a path outside `scripts/` is a load error
|
|
13
|
-
and a `validate` error. They run in the QuickJS sandbox: plain JavaScript with
|
|
14
|
-
**top-level `await`**, no Node APIs, no filesystem beyond the `tools.*` bridge.
|
|
15
|
-
|
|
16
|
-
**A code-state script is not an agent's script.** The runtime runs a code
|
|
17
|
-
state; the agent runs an `archmax_run` source. The two live in different places
|
|
18
|
-
on purpose — code-state scripts in `workflows/<slug>/scripts/`, runtime scripts
|
|
19
|
-
in `skills/<capability>/scripts/` — and the classes cannot meet: the agent
|
|
20
|
-
cannot read `workflows/**` at all, and `archmax_run` is confined to skill
|
|
21
|
-
bundles by a non-overridable kernel rule (`script.skill-only`). An agent that
|
|
22
|
-
could execute or read the check that judges its work could probe it for a
|
|
23
|
-
bypass, which is the whole reason for the split. A code-state script's source
|
|
24
|
-
never reaches the model: not in a prompt, not in an event, not in a session
|
|
25
|
-
artifact — the next state reads the reason the script passed to
|
|
26
|
-
`archmax_advance`, never its code.
|
|
27
|
-
2. **Cases** (`workflows/<name>/tests/*.test.yaml`) — declarative
|
|
28
|
-
YAML documents run by `archmax test` and interpreted **host-side**: no
|
|
29
|
-
sandbox executes a case. Suite config is the `tests:` block in
|
|
30
|
-
`workflow.yaml` (there is no `tests.config.js`).
|
|
31
|
-
|
|
32
|
-
Sandbox scripts are self-contained: the only permitted imports are **typed
|
|
33
|
-
`@archmax-ai/harness/*` carriers** (stripped before evaluation — see
|
|
34
|
-
[Typed imports](#typed-imports)); importing from any other specifier is an
|
|
35
|
-
**error**. Each sandbox context gets its own globals — nothing leaks between
|
|
36
|
-
them (`SANDBOX_CONTRACT.context` is `"code"` for a code-state script, or `"ptc"`
|
|
37
|
-
for the agent's own `archmax_eval`/`archmax_run` code, which gets **no**
|
|
38
|
-
`defineCodeState`; `SANDBOX_CONTRACT.version` is `3`).
|
|
39
|
-
|
|
40
|
-
## Contents
|
|
41
|
-
|
|
42
|
-
- [Typed imports](#typed-imports)
|
|
43
|
-
- [Part 1 — Code-state scripts and agent scripts](#part-1--code-state-scripts-and-agent-scripts)
|
|
44
|
-
- [Authoring model: hand-authored](#authoring-model-hand-authored)
|
|
45
|
-
- [The code-state script contract](#the-code-state-script-contract)
|
|
46
|
-
- [Governance of `tools.*` calls](#governance-of-tools-calls)
|
|
47
|
-
- [`archmax_run` scripts: the `args` contract](#archmax_run-scripts-the-args-contract)
|
|
48
|
-
- [Scripts and sub-workflow fan-out](#scripts-and-sub-workflow-fan-out)
|
|
49
|
-
- [Worked example: a check in front of a state](#worked-example-a-check-in-front-of-a-state)
|
|
50
|
-
- [Worked example: a check behind a state](#worked-example-a-check-behind-a-state)
|
|
51
|
-
- [Worked example: approve, then end](#worked-example-approve-then-end)
|
|
52
|
-
- [When to use a grader state instead](#when-to-use-a-grader-state-instead)
|
|
53
|
-
- [Part 2 — Cases (declarative YAML)](#part-2--cases-declarative-yaml)
|
|
54
|
-
- [Layout and the prose rules](#layout-and-the-prose-rules)
|
|
55
|
-
- [Suite config: the `tests:` block](#suite-config-the-tests-block)
|
|
56
|
-
- [Top-level keys](#top-level-keys)
|
|
57
|
-
- [Step entries: one flat script](#step-entries-one-flat-script)
|
|
58
|
-
- [Assertion grammar](#assertion-grammar)
|
|
59
|
-
- [Structural assertions halt the case](#structural-assertions-halt-the-case)
|
|
60
|
-
- [Start conditions: trigger, variables, and workspace](#start-conditions-trigger-variables-and-workspace)
|
|
61
|
-
- [Recovering a case's inputs from previous runs](#recovering-a-cases-inputs-from-previous-runs)
|
|
62
|
-
- [File fixtures: `from:` references](#file-fixtures-from-references)
|
|
63
|
-
- [Settle the mocking depth before writing a case](#settle-the-mocking-depth-before-writing-a-case)
|
|
64
|
-
- [Worked example: happy path + grade](#worked-example-happy-path--grade)
|
|
65
|
-
- [Worked example: declined at entry, leak check](#worked-example-declined-at-entry-leak-check)
|
|
66
|
-
- [Worked example: mocking tool results](#worked-example-mocking-tool-results)
|
|
67
|
-
- [Worked example: human-in-the-loop decisions (refine loop)](#worked-example-human-in-the-loop-decisions-refine-loop)
|
|
68
|
-
- [Worked example: triggered start + fixture file](#worked-example-triggered-start--fixture-file)
|
|
69
|
-
- [Migrating a JS case to YAML](#migrating-a-js-case-to-yaml)
|
|
70
|
-
|
|
71
|
-
---
|
|
72
|
-
|
|
73
|
-
## Typed imports
|
|
74
|
-
|
|
75
|
-
A code-state script may open with a **typed import** — a type carrier that gives
|
|
76
|
-
IDE completion for the state's input, and is **stripped before QuickJS
|
|
77
|
-
evaluation** (the runtime implementation is an identity):
|
|
78
|
-
|
|
79
|
-
```js
|
|
80
|
-
/** … */
|
|
81
|
-
import { defineCodeState } from "@archmax-ai/harness/sandbox";
|
|
82
|
-
|
|
83
|
-
export default defineCodeState(async ({ state, trigger, variables, tools }) => {
|
|
84
|
-
/* ... */
|
|
85
|
-
});
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
The same subpath types the input and the history (`CodeStateInput`,
|
|
89
|
-
`CodeStateArgs`, `CodeStateFn`, `SandboxTools`, `SessionHistory`,
|
|
90
|
-
`HistoryMessage`, `TrailStep`). Importing from any
|
|
91
|
-
**non-`@archmax-ai/harness/*`** specifier is an **error** — sandbox scripts are
|
|
92
|
-
self-contained. **Bare-globals authoring** (no import) is equally valid:
|
|
93
|
-
`defineCodeState` is a prelude global, and a plain
|
|
94
|
-
`export default async function ({ tools }) { … }` registers the same way.
|
|
95
|
-
|
|
96
|
-
---
|
|
97
|
-
|
|
98
|
-
## Part 1 — Code-state scripts and agent scripts
|
|
99
|
-
|
|
100
|
-
### Authoring model: hand-authored
|
|
101
|
-
|
|
102
|
-
A code state references an **ordinary hand-authored** `.js` file. You DESIGN AND
|
|
103
|
-
WRITE the script directly — there is no generation step. `workflow.yaml` only
|
|
104
|
-
wires it: the state's type, its script, and its transitions, which are where the
|
|
105
|
-
script may send the session:
|
|
106
|
-
|
|
107
|
-
```yaml
|
|
108
|
-
states:
|
|
109
|
-
screen-request:
|
|
110
|
-
type: code
|
|
111
|
-
script: scripts/screen-request.js
|
|
112
|
-
triggers:
|
|
113
|
-
manual:
|
|
114
|
-
skills:
|
|
115
|
-
allow: [order-data] # the script reads the order records: granted here
|
|
116
|
-
on_error: escalation # where a failure of the script routes
|
|
117
|
-
transitions:
|
|
118
|
-
- to: identify-case
|
|
119
|
-
description: The requester is known and asks only about their own company's orders.
|
|
120
|
-
- to: decline
|
|
121
|
-
description: The request names no known requester, or reaches for another company's orders.
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
`script` is **required** on a code state and **refused** on every other type
|
|
125
|
-
(both load errors). A code state makes no model call, so `instructions`, `model`,
|
|
126
|
-
`budget`, `evidence` and `approvers` on one are `validate` errors: say what the
|
|
127
|
-
script does in its JSDoc block, and in the state's `summary`.
|
|
128
|
-
|
|
129
|
-
A code-state script **MUST** open with a leading JSDoc block — a title line, a
|
|
130
|
-
blank line, and a description of what the script does _and how_: the inputs it
|
|
131
|
-
reads, the sequence of checks, and **where it sends the session in each case**
|
|
132
|
-
(each `archmax_advance` target and why, each `archmax_raise` code). It is the only
|
|
133
|
-
human-facing contract the file has (admin UIs display it instead of the code), and
|
|
134
|
-
it is what a reviewer refines when the check is wrong. No `@specification` tag, no
|
|
135
|
-
`// archmax:generated` marker, no generation semantics. See
|
|
136
|
-
[SKILL.md → Workspace layout](../SKILL.md#workspace-layout-complete--nothing-else-exists).
|
|
137
|
-
|
|
138
|
-
- `archmax validate` reports a code-state script problem when the script **file
|
|
139
|
-
is missing** from `scripts/` or it has a **foreign (non-`@archmax-ai/harness/*`)
|
|
140
|
-
import**. It does **not** check the JSDoc block on any script — the header is
|
|
141
|
-
an authoring standard you uphold yourself. (Cases are YAML; there
|
|
142
|
-
`archmax validate` **errors** on a missing `description` instead.)
|
|
143
|
-
- The runtime never generates scripts: a missing script file at run time fails
|
|
144
|
-
the state (fail-closed), routed through its `on_error`.
|
|
145
|
-
|
|
146
|
-
### The code-state script contract
|
|
147
|
-
|
|
148
|
-
When the session's position reaches a code state — the agent's `archmax_advance`
|
|
149
|
-
into it, another automated state's advance, a human decision, an `on_error`
|
|
150
|
-
route, a turn that opens there, an `archmax_reset` back to it — the runtime runs
|
|
151
|
-
its script **in place of the next model call**. The agent's model is not called
|
|
152
|
-
while the session stands in a code state, so a chain of code states makes no
|
|
153
|
-
model call at all. The script's **default export** is called with **one object**,
|
|
154
|
-
and the script **moves the session itself**:
|
|
155
|
-
|
|
156
|
-
```js
|
|
157
|
-
/**
|
|
158
|
-
* Requester screen — reject a request whose requester is not a known customer.
|
|
159
|
-
*
|
|
160
|
-
* Reads the first user message from the session's history, extracts the first
|
|
161
|
-
* email address in it, loads skills/order-data/assets/orders.json (this state
|
|
162
|
-
* enables `order-data`), and looks the address up against every order's `email`.
|
|
163
|
-
*
|
|
164
|
-
* Advances to decline when the request carries no email address, and when the
|
|
165
|
-
* address matches no order — each with a reason naming what was missing — and
|
|
166
|
-
* otherwise to identify-case.
|
|
167
|
-
*/
|
|
168
|
-
import { defineCodeState } from "@archmax-ai/harness/sandbox";
|
|
169
|
-
|
|
170
|
-
export default defineCodeState(async ({ tools }) => {
|
|
171
|
-
const { messages } = JSON.parse(await tools.archmax_get_history());
|
|
172
|
-
const orders = JSON.parse(await tools.read_file({ file_path: "skills/order-data/assets/orders.json" }));
|
|
173
|
-
const request = messages.find((m) => m.role === "user")?.text ?? "";
|
|
174
|
-
const email = request.match(/[\w.+-]+@[\w.-]+/)?.[0];
|
|
175
|
-
if (!email) return tools.archmax_advance({ to: "decline", reason: "No email address found in the request." });
|
|
176
|
-
if (!orders.some((o) => o.email === email)) {
|
|
177
|
-
return tools.archmax_advance({ to: "decline", reason: `"${email}" is not in the order database.` });
|
|
178
|
-
}
|
|
179
|
-
console.log(`Allowed: ${email}`);
|
|
180
|
-
await tools.archmax_advance({ to: "identify-case", reason: `${email} is a known requester.` });
|
|
181
|
-
});
|
|
182
|
-
```
|
|
183
|
-
|
|
184
|
-
**The input** (`CodeStateInput` in `@archmax-ai/harness/sandbox`; the same fields
|
|
185
|
-
minus `tools` are also the global `args`):
|
|
186
|
-
|
|
187
|
-
| Field | Meaning |
|
|
188
|
-
| --- | --- |
|
|
189
|
-
| `state` | The code state's own slug. |
|
|
190
|
-
| `trigger?` | The id **string** of the trigger the turn carries. It carries no payload — the session's input is in `variables`. |
|
|
191
|
-
| `variables` | The session's variables as a plain `name → value` map, including the built-in `trigger`. Structured values are present whole (`variables.order.items[0].sku`). **Read-only**: assigning changes nothing. Write with `tools.archmax_set_variables`, which holds locks, `title` and typed returns exactly as the agent's call does. |
|
|
192
|
-
| `tools` | **Exactly this state's tool surface** (async): its grants (the workflow-wide grant included), the host's essential tools, and its controls. A tool answers to its own name (`tools.read_file`, `tools["archmax_workflow_enrich-order"]`) and to its camelCased key (`tools.readFile`); `Object.keys(tools)` lists the camelCased keys. A call with no argument passes `{}` (`tools.archmax_get_history()`, `tools.ls()`). **A name outside the surface is `undefined`.** See [Governance](#governance-of-tools-calls). |
|
|
193
|
-
|
|
194
|
-
**The session's history** is not part of the input: the script reads it the way
|
|
195
|
-
a grader does, with `archmax_get_history` (`SessionHistory`):
|
|
196
|
-
|
|
197
|
-
```js
|
|
198
|
-
const { messages, trail } = JSON.parse(await tools.archmax_get_history());
|
|
199
|
-
```
|
|
200
|
-
|
|
201
|
-
It is the session as it stood when the state was entered, built from the
|
|
202
|
-
checkpointed session (never a summarized request), whole and oldest first,
|
|
203
|
-
across every turn:
|
|
204
|
-
|
|
205
|
-
| Field | Meaning |
|
|
206
|
-
| --- | --- |
|
|
207
|
-
| `messages` | The transcript as `{ role, text, toolCalls?, tool?, id?, note? }`. `role` is `user`, `assistant`, `tool`, `system`, or **`runtime`** — a note the runtime wrote (an arrival, a decision, an error route, an automated state's move; `note` names the kind). An `assistant` entry lists the calls it made in `toolCalls`, as `{ id?, name, args }`; the `tool` entry answering one carries the tool's name in `tool`, the call's `id`, and the **result in `text`**, as the tool returned it (`JSON.parse` it when the tool returns JSON; a failed or refused call's `text` is the error). A script that reacts to narration matches `role === "runtime"`, never a `[bracket]` prefix in a user message. Not windowed: the sandbox's `maxResultChars` and `memoryLimitBytes` bound it. |
|
|
208
|
-
| `trail` | The trail as `{ to, kind, reason?, workflow?, status?, ts }`. `kind` is `trigger`, `agent`, `human`, `grader`, `code`, `sub-workflow`, `reset` or `on_error`. Count earlier rounds here: `trail.filter((s) => s.kind === "code" && s.to === "refund-request").length`. |
|
|
209
|
-
|
|
210
|
-
Only what the transcript holds is there: a call a script made through `tools.*`
|
|
211
|
-
(an agent's `archmax_run`, or a code state's own) is not, and neither is a
|
|
212
|
-
grader's.
|
|
213
|
-
|
|
214
|
-
**What a code state checks.** The conversation is the model's memory, not
|
|
215
|
-
evidence: a code state checks the session's records — `variables`, files, and
|
|
216
|
-
the tool results the history carries as the tools returned them. Check a result
|
|
217
|
-
itself, never a copy the model wrote into a variable, a file or its reply, and
|
|
218
|
-
never call a tool again just to see its result (for a host whose tools reach an
|
|
219
|
-
external system, that is a second action). To read a result, find the call in an
|
|
220
|
-
`assistant` entry's `toolCalls`, then the `tool` entry with the same `id`:
|
|
221
|
-
|
|
222
|
-
```js
|
|
223
|
-
/**
|
|
224
|
-
* Status check — the recorded status must be the one get_order returned.
|
|
225
|
-
*
|
|
226
|
-
* Finds the last get_order call in the session's history and the tool message
|
|
227
|
-
* answering it, and reads scratchpad/status.json ({ status }), which the agent
|
|
228
|
-
* wrote. Advances to notify-customer when the recorded status is the returned
|
|
229
|
-
* one; otherwise back to look-up-order, with the status to record, or with a
|
|
230
|
-
* request to look the order up when no get_order call has a readable result.
|
|
231
|
-
*/
|
|
232
|
-
import { defineCodeState } from "@archmax-ai/harness/sandbox";
|
|
233
|
-
|
|
234
|
-
export default defineCodeState(async ({ tools }) => {
|
|
235
|
-
const { messages } = JSON.parse(await tools.archmax_get_history());
|
|
236
|
-
const call = messages.flatMap((m) => m.toolCalls ?? []).findLast((c) => c.name === "get_order");
|
|
237
|
-
const answer = call && messages.find((m) => m.role === "tool" && m.id === call.id);
|
|
238
|
-
let order;
|
|
239
|
-
try {
|
|
240
|
-
order = JSON.parse(answer.text);
|
|
241
|
-
} catch {
|
|
242
|
-
return tools.archmax_advance({ to: "look-up-order", reason: "Look the order up with get_order first." });
|
|
243
|
-
}
|
|
244
|
-
const recorded = JSON.parse(await tools.read_file({ file_path: "scratchpad/status.json" }));
|
|
245
|
-
if (recorded.status !== order.status) {
|
|
246
|
-
return tools.archmax_advance({
|
|
247
|
-
to: "look-up-order",
|
|
248
|
-
reason: `Order ${call.args.id} is ${order.status}: record that status exactly.`,
|
|
249
|
-
});
|
|
250
|
-
}
|
|
251
|
-
await tools.archmax_advance({ to: "notify-customer", reason: `The recorded status ${order.status} is get_order's.` });
|
|
252
|
-
});
|
|
253
|
-
```
|
|
254
|
-
|
|
255
|
-
The history spans every turn and every entry into a state, so where only the
|
|
256
|
-
latest round counts, take the last match, as above. What a later state or turn
|
|
257
|
-
needs from the conversation (the person's words, a decision) is still best
|
|
258
|
-
stored in a variable or a file by the state that has it, so no check has to
|
|
259
|
-
search the transcript for it.
|
|
260
|
-
|
|
261
|
-
**The controls** on `tools`:
|
|
262
|
-
|
|
263
|
-
| Control | Effect |
|
|
264
|
-
| --- | --- |
|
|
265
|
-
| `archmax_advance({ to, reason, evidence? })` | Records the move along one of the state's transitions. Checked **at the call**: an undeclared `to` or an unmet `requires` rejects naming the valid targets or the missing variable, and the script may catch it and decide otherwise. `evidence` is accepted only into a human state. The `reason` is what the next state reads. Absent in a terminal state. |
|
|
266
|
-
| `archmax_raise({ code, reason })` | Records the end of the session as **failed**, with the code and reason as its outcome's `exit`. |
|
|
267
|
-
| `archmax_get_variables({ name?, path? })` / `archmax_set_variables({ variables, lock? })` | Read and write the session's variables, as the agent's calls do. A write is visible to the script's later calls, so a script can set what `requires` asks for and then advance. |
|
|
268
|
-
| `archmax_get_history()` | Resolves to the session's history as JSON text, `{ messages, trail }` (above). |
|
|
269
|
-
|
|
270
|
-
These are a grader's controls too: both automated types get the same set. There
|
|
271
|
-
is no `archmax_wait` and no `archmax_reset`: a code state cannot park, since a
|
|
272
|
-
script frame is not durable. To hand work to a person, advance into a human state.
|
|
273
|
-
|
|
274
|
-
**How the run settles** — exactly one outcome:
|
|
275
|
-
|
|
276
|
-
| The run… | Outcome |
|
|
277
|
-
| --- | --- |
|
|
278
|
-
| called `archmax_advance` and finished cleanly | **advanced**: the session moves. The trail records `{ to, kind: "code", reason }`, and a runtime note carries the reason to the next state: `[code] '<state>' moved the run to '<to>': <reason>. You are now in '<to>'.` Into a human state the session parks after the one reply-only handoff message. Into another automated state, that state runs next, still with no model call. |
|
|
279
|
-
| called `archmax_raise` | **raised**: the session ends `failed` — no `on_error`, no `returns` check, nothing further runs in the turn. |
|
|
280
|
-
| finished without either, in a **terminal** state | **completed**: the session completes, held to the trigger's `returns`. |
|
|
281
|
-
| threw, timed out, could not be read, imported a foreign module, or — in a **non-terminal** state — finished without either call | **failed**: routed through the state's `on_error`, else the session ends rejected. Finishing undecided fails with `code state '<slug>' finished without calling archmax_advance or archmax_raise`. |
|
|
282
|
-
|
|
283
|
-
Three rules hold the outcomes apart:
|
|
284
|
-
|
|
285
|
-
- **The return value is not read.** Returning `false`, a string or an object does
|
|
286
|
-
nothing; there is no verdict to return. Only the controls decide.
|
|
287
|
-
- **One decision per run.** The first accepted `archmax_advance` or `archmax_raise`
|
|
288
|
-
is the state's decision, and every later one is refused (it rejects).
|
|
289
|
-
- **The decision takes effect when the script finishes.** A script that advances
|
|
290
|
-
and then throws **fails**, and the session does not move. A raise wins over a
|
|
291
|
-
later error.
|
|
292
|
-
|
|
293
|
-
So parse defensively where malformed input should *send the work back*, and let a
|
|
294
|
-
genuine infrastructure failure (an unreadable database) throw, so the state fails
|
|
295
|
-
closed through its `on_error`.
|
|
296
|
-
|
|
297
|
-
Other globals: `console.log(...)` (each line a `code-output` event, which the CLI's
|
|
298
|
-
state flow prints under the state), and `defineCodeState(fn)`, an alias for
|
|
299
|
-
`export default fn`. A file with **no default export** is evaluated as a plain
|
|
300
|
-
script: it reads the input from the global `args` and calls the global `tools`.
|
|
301
|
-
**Each run starts clean**: a fresh REPL every time the state is entered, shared
|
|
302
|
-
with neither the agent's `archmax_eval` nor another code state. The script is one
|
|
303
|
-
evaluation, so `settings.timeoutMs` (default 15000), `memoryLimitBytes`,
|
|
304
|
-
`maxPtcCalls` and `maxResultChars` bound it whole, and a cancelled turn stops it at
|
|
305
|
-
once.
|
|
306
|
-
|
|
307
|
-
`${{…}}` does not apply inside a script: the runtime substitutes references in the
|
|
308
|
-
arguments a *model* writes. A script already has the values in `variables` and
|
|
309
|
-
composes strings in JavaScript, so `${{…}}` text in a script's `tools.*` call is
|
|
310
|
-
ordinary data.
|
|
311
|
-
|
|
312
|
-
### Governance of `tools.*` calls
|
|
313
|
-
|
|
314
|
-
Every `tools.*` call is checked by the decision kernel at the moment it runs —
|
|
315
|
-
the same kernel, with the same rules, that governs a tool call the model makes
|
|
316
|
-
directly. There is no runtime authority for scripts: **both kinds are governed
|
|
317
|
-
against the state they run in.**
|
|
318
|
-
|
|
319
|
-
| Script kind | Governed against | Must be granted there |
|
|
320
|
-
| --- | --- | --- |
|
|
321
|
-
| Code-state script (`type: code`) | its own code state: `tools.allow`/`forbid`, `skills.*`, `mounts.*`, plus `allow_always`, `forbid_always`, safety rules, consumer rules | every tool, skill and governed mount it uses |
|
|
322
|
-
| `archmax_run` / `archmax_eval` script (the model chose to run it) | the agent's active state, exactly as the model's own calls | every tool it calls |
|
|
323
|
-
|
|
324
|
-
So a code state that reads `skills/order-data/assets/orders.json` enables
|
|
325
|
-
`order-data` in its own `skills.allow`, and one that calls a host tool names it in
|
|
326
|
-
its own `tools.allow`. A tool another state grants is `undefined` on its `tools`;
|
|
327
|
-
a bare `forbid` entry removes a tool from it, and a guarded one (`{ tool:
|
|
328
|
-
write_file, paths: [...] }`) refuses the matching calls. A grant on a code state
|
|
329
|
-
naming `write_todos`, `archmax_eval`, `archmax_run`, `archmax_wait` or
|
|
330
|
-
`archmax_reset` is a `validate` error: no automated state is offered them, and
|
|
331
|
-
the same tools in the workflow's `allow_always` are left out of its surface.
|
|
332
|
-
|
|
333
|
-
Mounts work the same way: a governed mount is reachable from a code state only
|
|
334
|
-
when `mounts.allow_always` or that state's own `mounts.allow` grants it, and
|
|
335
|
-
writable only where the grant asks `access: read_write`:
|
|
336
|
-
|
|
337
|
-
```js
|
|
338
|
-
// workflow.yaml: states.check-contract.mounts.allow: [contracts]
|
|
339
|
-
const policy = await tools.read_file({ file_path: "contracts/policy.md" });
|
|
340
|
-
```
|
|
341
|
-
|
|
342
|
-
Neither kind may **read** the authoring plane at all. A `tools.*` call naming
|
|
343
|
-
`workflows/**` is blocked (`zone.governance-plane`) with a diagnostic naming the
|
|
344
|
-
prefix — a script cannot read its own spec, a sibling's, or a grader's
|
|
345
|
-
criteria. There is no `task()` in any sandbox context and no `tools.task`: work
|
|
346
|
-
that needs a model's judgment is a grader state on the graph.
|
|
347
|
-
|
|
348
|
-
A refused call **rejects** inside the script, carrying the governance reason.
|
|
349
|
-
The underlying tool never runs whether or not you catch it:
|
|
350
|
-
|
|
351
|
-
```js
|
|
352
|
-
try {
|
|
353
|
-
await tools.write_file({ file_path: "scratchpad/report.md", content: body });
|
|
354
|
-
} catch (err) {
|
|
355
|
-
console.log(`could not write the report: ${err.message}`);
|
|
356
|
-
}
|
|
357
|
-
```
|
|
358
|
-
|
|
359
|
-
A **binary file** is not a refusal and does not throw: reads are text only, so
|
|
360
|
-
`tools.read_file` on an image, audio, video, PDF or PowerPoint file, or on any
|
|
361
|
-
file whose content holds a NUL byte (an archive, an Office document), resolves to
|
|
362
|
-
the notice `Error: '<path>' is a binary file (<type>, <size>) and was not read;
|
|
363
|
-
read_file returns text files only.` instead of content — an image too, even when
|
|
364
|
-
the host turned on `images`, which shows images to the model only. XML and JSON
|
|
365
|
-
are text even when the store labels them `application/xml` or a `+xml`/`+json`
|
|
366
|
-
type, so an Office document's unpacked `word/document.xml` reads as text. A missing file reads as
|
|
367
|
-
an `Error: …` string the same way. A script that parses what it read checks
|
|
368
|
-
first:
|
|
369
|
-
|
|
370
|
-
```js
|
|
371
|
-
const text = await tools.read_file({ file_path: "scratchpad/upload.json" });
|
|
372
|
-
const upload = text.startsWith("Error:") ? null : JSON.parse(text);
|
|
373
|
-
```
|
|
374
|
-
|
|
375
|
-
To **duplicate, move or delete** a file — a template, an image, a `.docx` —
|
|
376
|
-
call `tools.copy_file({ source, destination, overwrite? })`,
|
|
377
|
-
`tools.move_file({ source, destination, overwrite? })` or
|
|
378
|
-
`tools.remove_file({ file_path })` instead of reading and writing it back. The
|
|
379
|
-
bytes move, binary files included, with no `read_file` line limit, and each
|
|
380
|
-
call resolves to one line (`Copied '<source>' to '<destination>' (<size>).`,
|
|
381
|
-
`Moved …`, `Removed '<path>'.`). They are governed by their declared paths like
|
|
382
|
-
any other call, so a governance refusal (a destination in a read-only mount, a
|
|
383
|
-
disabled skill's file) rejects. A failure the operation itself meets (a missing
|
|
384
|
-
source, a directory, an existing destination without `overwrite: true`)
|
|
385
|
-
resolves to an `Error: …` string instead.
|
|
386
|
-
|
|
387
|
-
**Authoring rule for every script:** if sandboxed code calls `tools.X`, the state
|
|
388
|
-
it runs in must cover `tool: X` (with any argument constraints the call
|
|
389
|
-
satisfies), or `tools.X` is `undefined` (or the call rejects). Grant it per state
|
|
390
|
-
with `tools.allow`, or workflow-wide with `tools.allow_always`; a default tool such
|
|
391
|
-
as `read_file` is covered wherever the workflow-wide grant includes it.
|
|
392
|
-
|
|
393
|
-
### `archmax_run` scripts: the `args` contract
|
|
394
|
-
|
|
395
|
-
An `archmax_run` source (`skills/<capability>/scripts/*.js`) is a plain script:
|
|
396
|
-
top-level `await`, `tools.*`, `console.log`, and the **last expression** is its
|
|
397
|
-
result — returned to the model as the tool result, never into the variable
|
|
398
|
-
store. It reads its input from the global **`args`**:
|
|
399
|
-
|
|
400
|
-
- **the arguments the model passed** to `archmax_run` (`args.<name>`), plus
|
|
401
|
-
- **`args.variables`** — the session's variables as a flat `name → value` map,
|
|
402
|
-
the same map a code state gets. The runtime supplies it on every call, so a
|
|
403
|
-
script reads the run's input (a seeded list, a delivered payload) without the
|
|
404
|
-
model having to retype it.
|
|
405
|
-
|
|
406
|
-
```js
|
|
407
|
-
/** One sub-run per requested order. */
|
|
408
|
-
const orders = args.variables?.orders_to_enrich ?? [];
|
|
409
|
-
```
|
|
410
|
-
|
|
411
|
-
`args.variables` is a read-only snapshot; a script that must persist a result
|
|
412
|
-
returns it and the state's `requires:` makes the model record it with
|
|
413
|
-
`archmax_set_variables`. An agent's script reaches **no control tool** — not
|
|
414
|
-
`archmax_advance`, not the variable tools — and there is **no `task()`** in an
|
|
415
|
-
`archmax_run` script or in `archmax_eval`. The model has no `task` tool at all: it
|
|
416
|
-
is disclosed in no state and grantable by nothing, because judging work is a
|
|
417
|
-
grader state and delegating it is a sub-workflow.
|
|
418
|
-
|
|
419
|
-
### Scripts and sub-workflow fan-out
|
|
420
|
-
|
|
421
|
-
A script **is** the deterministic way to drive a fan-out. Delegation tools are on the `tools.*`
|
|
422
|
-
surface — a delegation does work and returns a value, unlike the control tools an agent's script
|
|
423
|
-
may never call — so a script decides how many sub-runs to start, with what inputs, and what to do
|
|
424
|
-
with each answer:
|
|
425
|
-
|
|
426
|
-
```js
|
|
427
|
-
/** One sub-run per delayed order, all in flight at once. */
|
|
428
|
-
const orders = args.variables?.orders_to_enrich ?? [];
|
|
429
|
-
|
|
430
|
-
const enriched = await Promise.all(
|
|
431
|
-
// The bridge camelCases tool names, and a tool result arrives as text.
|
|
432
|
-
orders.map(async (o) => {
|
|
433
|
-
const answer = await tools.archmaxWorkflowEnrichOrder({ order_id: o.order_id });
|
|
434
|
-
return { order_id: o.order_id, ...JSON.parse(answer).returns };
|
|
435
|
-
}),
|
|
436
|
-
);
|
|
437
|
-
|
|
438
|
-
enriched;
|
|
439
|
-
```
|
|
440
|
-
|
|
441
|
-
`Promise.all` genuinely parallelizes: the bridge starts each host call detached and returns its
|
|
442
|
-
promise handle immediately. The runtime still bounds how many sub-runs are in flight
|
|
443
|
-
(the dispatcher's `maxConcurrent`), queueing the rest. The list never enters the model's
|
|
444
|
-
context. A code state may drive the same fan-out when its own state grants the delegation tool,
|
|
445
|
-
and then record the results itself with `tools.archmax_set_variables` before it advances.
|
|
446
|
-
|
|
447
|
-
Two limits still hold, both inherited rather than special to delegation. A script may only call a
|
|
448
|
-
delegation tool its **calling state** allows — the same kernel verdict an agent's call gets. And a
|
|
449
|
-
script's delegation **cannot park**: a QuickJS frame is not durable, so a child that reaches a human
|
|
450
|
-
state fails that call closed with `parked`.
|
|
451
|
-
|
|
452
|
-
An agent's script still may not call `archmax_advance` (moving the state machine is the model's
|
|
453
|
-
governed decision) or `archmax_set_variables`. Concurrent calls succeed or fail **independently**,
|
|
454
|
-
so a script can act on what worked — `Promise.allSettled` is the honest shape when partial success
|
|
455
|
-
is meaningful. Each sub-run is its own session with its own `scratchpad/`, so concurrent sub-runs
|
|
456
|
-
never collide on a path; a `file` return comes back as a copy under a name no other file holds.
|
|
457
|
-
|
|
458
|
-
A script lives as long as its turn. When the host cancels the turn, a call still in flight fails
|
|
459
|
-
with `Tool '<name>' failed: the turn was aborted`, and every call the script makes after that is
|
|
460
|
-
refused the same way before it reaches the tool. A retry in a `catch` therefore dispatches nothing
|
|
461
|
-
more, so there is no need to tell a cancel apart from an ordinary failure. Code-state scripts stop
|
|
462
|
-
the same way.
|
|
463
|
-
|
|
464
|
-
### Worked example: a check in front of a state
|
|
465
|
-
|
|
466
|
-
To screen every request before any model call, make the trigger's entry state a
|
|
467
|
-
code state that advances to where the work starts, or to a state that declines it.
|
|
468
|
-
The bundled example's `screen-request` (`workflows/order-lookup/scripts/screen-request.js`)
|
|
469
|
-
merges a requester check and a tenancy check — wired as in
|
|
470
|
-
[Authoring model](#authoring-model-hand-authored), with `decline` a terminal agent
|
|
471
|
-
state that tells the person why:
|
|
472
|
-
|
|
473
|
-
```js
|
|
474
|
-
/**
|
|
475
|
-
* Request screen — the requester must be known, and may only reach their own
|
|
476
|
-
* company's orders.
|
|
477
|
-
*
|
|
478
|
-
* Binds the requester's email (the first one in the first user message) to the
|
|
479
|
-
* customers whose orders carry it in skills/order-data/assets/orders.json, then
|
|
480
|
-
* checks the request two ways: it names another customer, or it references an
|
|
481
|
-
* order id (`ORD-…`) owned by another customer. Matching is case-insensitive.
|
|
482
|
-
*
|
|
483
|
-
* Advances to decline — with the reason the decline state tells the person — when
|
|
484
|
-
* there is no email, the email is unknown, or either tenancy check finds a
|
|
485
|
-
* foreign customer; otherwise to identify-case. An order database that cannot
|
|
486
|
-
* be read throws: the state fails closed through its on_error.
|
|
487
|
-
*/
|
|
488
|
-
import { defineCodeState } from "@archmax-ai/harness/sandbox";
|
|
489
|
-
|
|
490
|
-
export default defineCodeState(async ({ tools }) => {
|
|
491
|
-
const { messages } = JSON.parse(await tools.archmax_get_history());
|
|
492
|
-
const orders = JSON.parse(await tools.read_file({ file_path: "skills/order-data/assets/orders.json" }));
|
|
493
|
-
if (!Array.isArray(orders)) throw new Error("skills/order-data/assets/orders.json is not a list of orders");
|
|
494
|
-
|
|
495
|
-
const request = messages.find((m) => m.role === "user")?.text ?? "";
|
|
496
|
-
const email = (request.match(/[\w.+-]+@[\w.-]+\.\w{2,}/)?.[0] ?? "").toLowerCase();
|
|
497
|
-
if (!email) {
|
|
498
|
-
return tools.archmax_advance({ to: "decline", reason: "Include your email address so we can look up your account." });
|
|
499
|
-
}
|
|
500
|
-
const own = new Set(
|
|
501
|
-
orders.filter((o) => String(o.email).toLowerCase() === email).map((o) => String(o.customer)),
|
|
502
|
-
);
|
|
503
|
-
if (own.size === 0) {
|
|
504
|
-
return tools.archmax_advance({ to: "decline", reason: `"${email}" is not in the order database.` });
|
|
505
|
-
}
|
|
506
|
-
|
|
507
|
-
const lower = request.toLowerCase();
|
|
508
|
-
const foreignByName = [...new Set(orders.map((o) => String(o.customer)))].filter(
|
|
509
|
-
(name) => !own.has(name) && lower.includes(name.toLowerCase()),
|
|
510
|
-
);
|
|
511
|
-
const referenced = new Set((request.match(/\bORD-\d+\b/gi) ?? []).map((id) => id.toUpperCase()));
|
|
512
|
-
const foreignById = orders
|
|
513
|
-
.filter((o) => referenced.has(String(o.id).toUpperCase()) && !own.has(String(o.customer)))
|
|
514
|
-
.map((o) => String(o.id));
|
|
515
|
-
|
|
516
|
-
if (foreignByName.length > 0 || foreignById.length > 0) {
|
|
517
|
-
return tools.archmax_advance({
|
|
518
|
-
to: "decline",
|
|
519
|
-
reason: `"${email}" may only view orders for its own account. Do not name the other customer's orders.`,
|
|
520
|
-
});
|
|
521
|
-
}
|
|
522
|
-
await tools.archmax_advance({ to: "identify-case", reason: `${email} is authorized for ${[...own].join(", ")}.` });
|
|
523
|
-
});
|
|
524
|
-
```
|
|
525
|
-
|
|
526
|
-
The pattern to internalize: **one `archmax_advance` per outcome, each with a
|
|
527
|
-
`reason` the next state can act on** (the `decline` agent reads it from the `[code]`
|
|
528
|
-
note and tells the person); `console.log` for anything worth seeing in a verbose
|
|
529
|
-
run; a `throw` for what should fail the state rather than route it. To end the
|
|
530
|
-
session as a failure instead, with no reply at all, call
|
|
531
|
-
`tools.archmax_raise({ code: "unknown-requester", reason })`.
|
|
532
|
-
|
|
533
|
-
### Worked example: a check behind a state
|
|
534
|
-
|
|
535
|
-
To check the agent's work before the session moves on, route the agent's
|
|
536
|
-
transition into a code state, and give that state an edge onward and an edge
|
|
537
|
-
**back**. The bundled example's `check-refund` sits between `refund-request` (the
|
|
538
|
-
agent records a decision) and `refund-review` (a person reviews it), so a decision
|
|
539
|
-
the policy forbids never reaches the reviewer:
|
|
540
|
-
|
|
541
|
-
```yaml
|
|
542
|
-
states:
|
|
543
|
-
refund-request:
|
|
544
|
-
instructions: >-
|
|
545
|
-
… Record the decision in scratchpad/refund.json, tell the customer, and in
|
|
546
|
-
that same message archmax_advance to check-refund. If the check sends the
|
|
547
|
-
case back, its note names the fix: correct the file and advance again.
|
|
548
|
-
skills:
|
|
549
|
-
allow: [order-data, refund-policy]
|
|
550
|
-
transitions:
|
|
551
|
-
- to: check-refund
|
|
552
|
-
description: The refund decision has been recorded and explained to the customer.
|
|
553
|
-
check-refund:
|
|
554
|
-
type: code
|
|
555
|
-
script: scripts/check-refund.js
|
|
556
|
-
skills:
|
|
557
|
-
allow: [order-data]
|
|
558
|
-
on_error: escalation
|
|
559
|
-
transitions:
|
|
560
|
-
- to: refund-review
|
|
561
|
-
description: The recorded decision matches the refund policy.
|
|
562
|
-
- to: refund-request
|
|
563
|
-
description: The recorded decision contradicts the policy; send it back with the fix.
|
|
564
|
-
- to: escalation
|
|
565
|
-
description: The decision still fails the check after two corrections.
|
|
566
|
-
```
|
|
567
|
-
|
|
568
|
-
The check (`workflows/order-lookup/scripts/check-refund.js`), which counts its own
|
|
569
|
-
earlier send-backs on the trail and escalates on the third:
|
|
570
|
-
|
|
571
|
-
```js
|
|
572
|
-
/**
|
|
573
|
-
* Refund policy check — the agent may not hand a refund the policy forbids to
|
|
574
|
-
* the reviewer.
|
|
575
|
-
*
|
|
576
|
-
* Re-derives the refund decision from the order database instead of trusting
|
|
577
|
-
* the one the agent wrote. Reads the recorded ticket from scratchpad/refund.json
|
|
578
|
-
* ({ orderId, decision }), then reads skills/order-data/assets/orders.json and
|
|
579
|
-
* looks the order id up, case-insensitively, against each record's `id`.
|
|
580
|
-
*
|
|
581
|
-
* Policy: a refund is "approved" only when the order exists with status
|
|
582
|
-
* delivered or delayed; an unknown order and any other status must be "denied".
|
|
583
|
-
*
|
|
584
|
-
* Advances to refund-review when the recorded decision matches. Otherwise sends
|
|
585
|
-
* the session back to refund-request with a reason naming the fix — the ticket
|
|
586
|
-
* missing or unparseable, `decision` not "approved"/"denied", or the decision
|
|
587
|
-
* contradicting the policy — and, once it has sent it back twice (counted on
|
|
588
|
-
* the trail), advances to escalation instead. An unreadable order database
|
|
589
|
-
* throws: the state fails through its on_error.
|
|
590
|
-
*/
|
|
591
|
-
import { defineCodeState } from "@archmax-ai/harness/sandbox";
|
|
592
|
-
|
|
593
|
-
const MAX_SEND_BACKS = 2;
|
|
594
|
-
|
|
595
|
-
export default defineCodeState(async ({ tools }) => {
|
|
596
|
-
const { trail } = JSON.parse(await tools.archmax_get_history());
|
|
597
|
-
const sentBack = trail.filter((s) => s.kind === "code" && s.to === "refund-request").length;
|
|
598
|
-
const sendBack = (reason) =>
|
|
599
|
-
sentBack >= MAX_SEND_BACKS
|
|
600
|
-
? tools.archmax_advance({ to: "escalation", reason: `Still failing after ${sentBack} corrections: ${reason}` })
|
|
601
|
-
: tools.archmax_advance({ to: "refund-request", reason });
|
|
602
|
-
|
|
603
|
-
let refund;
|
|
604
|
-
try {
|
|
605
|
-
refund = JSON.parse(await tools.read_file({ file_path: "scratchpad/refund.json" }));
|
|
606
|
-
} catch (e) {
|
|
607
|
-
return sendBack(`scratchpad/refund.json could not be read or parsed (${e.message}) - write it as a JSON object with "orderId" and "decision", then advance again`);
|
|
608
|
-
}
|
|
609
|
-
const decision = refund?.decision;
|
|
610
|
-
if (decision !== "approved" && decision !== "denied") {
|
|
611
|
-
return sendBack(`scratchpad/refund.json "decision" must be exactly "approved" or "denied"; got ${JSON.stringify(decision)}`);
|
|
612
|
-
}
|
|
613
|
-
|
|
614
|
-
const orders = JSON.parse(await tools.read_file({ file_path: "skills/order-data/assets/orders.json" }));
|
|
615
|
-
const wanted = typeof refund.orderId === "string" ? refund.orderId.trim().toLowerCase() : "";
|
|
616
|
-
const order = orders.find((o) => typeof o?.id === "string" && o.id.toLowerCase() === wanted);
|
|
617
|
-
const derived = order && ["delivered", "delayed"].includes(order.status) ? "approved" : "denied";
|
|
618
|
-
console.log(`Refund policy check - derived "${derived}", recorded "${decision}"`);
|
|
619
|
-
|
|
620
|
-
if (decision === derived) {
|
|
621
|
-
return tools.archmax_advance({ to: "refund-review", reason: `"${decision}" for ${refund.orderId} matches the refund policy.` });
|
|
622
|
-
}
|
|
623
|
-
const why = !order ? "it does not exist in the order database" : `its status is "${order.status}"`;
|
|
624
|
-
return sendBack(
|
|
625
|
-
`Policy requires "${derived}" for order ${JSON.stringify(refund.orderId)} because ${why}, but the recorded decision is "${decision}" - update scratchpad/refund.json to decision: "${derived}", correct what you told the customer, and advance again`,
|
|
626
|
-
);
|
|
627
|
-
});
|
|
628
|
-
```
|
|
629
|
-
|
|
630
|
-
Note how the send-back reasons carry the _fix_: the agent reads the reason as a
|
|
631
|
-
`[code]` note, rewrites `scratchpad/refund.json`, and advances again. There is no
|
|
632
|
-
retry budget to declare: the script counts the rounds itself, from the history's
|
|
633
|
-
`trail`, and decides when to give up.
|
|
634
|
-
|
|
635
|
-
### Worked example: approve, then end
|
|
636
|
-
|
|
637
|
-
A check that approves into a terminal **agent** state costs one more model call:
|
|
638
|
-
that state's agent has to finish the turn. Approve into a **terminal code state**
|
|
639
|
-
instead. Its script runs and the session completes, with no further model call.
|
|
640
|
-
Say the enrichment path's grader approves into `enrichments-approved`, and the
|
|
641
|
-
trigger declares `returns: [delayed_order_ids]`:
|
|
642
|
-
|
|
643
|
-
```yaml
|
|
644
|
-
enrichments-approved:
|
|
645
|
-
type: code
|
|
646
|
-
script: scripts/enrichments-approved.js # no transitions: terminal
|
|
647
|
-
```
|
|
648
|
-
|
|
649
|
-
```js
|
|
650
|
-
/**
|
|
651
|
-
* Approved-enrichments record — the end of the enrichment path.
|
|
652
|
-
*
|
|
653
|
-
* Reads the `enrichments` the grader approved and sets the trigger's declared
|
|
654
|
-
* return `delayed_order_ids` from them, so a caller has the result without
|
|
655
|
-
* walking the list, then returns: the session completes.
|
|
656
|
-
*/
|
|
657
|
-
export default async function ({ variables, tools }) {
|
|
658
|
-
const delayed = variables.enrichments.filter((e) => e.delayed).map((e) => e.order_id);
|
|
659
|
-
await tools.archmax_set_variables({ variables: { delayed_order_ids: delayed } });
|
|
660
|
-
}
|
|
661
|
-
```
|
|
662
|
-
|
|
663
|
-
A terminal code state's completion is held to the trigger's `returns` like any
|
|
664
|
-
completion: set a declared return with `tools.archmax_set_variables` before the
|
|
665
|
-
script returns.
|
|
666
|
-
|
|
667
|
-
### When to use a grader state instead
|
|
668
|
-
|
|
669
|
-
Use a **code state** when the check is **deterministic** — derivable from
|
|
670
|
-
workspace data with plain code (policy tables, schema checks, cross-file
|
|
671
|
-
consistency). Use a **grader state** when the check requires **judgment** (is the
|
|
672
|
-
report complete? is the summary faithful?). A grader is an isolated subagent the
|
|
673
|
-
runtime runs on entry: its system prompt is a compiled-in preamble followed by the
|
|
674
|
-
state's `instructions` (its criteria), it reads the session with
|
|
675
|
-
`archmax_get_history` (the whole transcript, tool results included, and the
|
|
676
|
-
trail) and the work with its own state's tools, and it decides with
|
|
677
|
-
`archmax_advance` along one of its transitions, or ends the session with
|
|
678
|
-
`archmax_raise`. The bundled example's enrichment path:
|
|
679
|
-
|
|
680
|
-
```yaml
|
|
681
|
-
states:
|
|
682
|
-
enrich-order:
|
|
683
|
-
instructions: >-
|
|
684
|
-
Run skills/order-enrichment/scripts/enrich-orders.js with archmax_run …
|
|
685
|
-
Record that list as the `enrichments` variable with archmax_set_variables,
|
|
686
|
-
then advance to review-enrichment, which checks it. If the review sends it
|
|
687
|
-
back, fix what its note names and advance again.
|
|
688
|
-
requires: [enrichments]
|
|
689
|
-
transitions:
|
|
690
|
-
- to: review-enrichment
|
|
691
|
-
description: Every order has been enriched and `enrichments` is recorded.
|
|
692
|
-
review-enrichment:
|
|
693
|
-
type: grader
|
|
694
|
-
budget: { maxTurns: 8 } # the grader's own model calls
|
|
695
|
-
on_error: escalation # a grader that fails or finishes undecided routes here
|
|
696
|
-
skills:
|
|
697
|
-
allow: [] # read_file, from the workflow-wide grant, reads the returned files
|
|
698
|
-
instructions: |-
|
|
699
|
-
You check the enrichments recorded for this run before they are reported.
|
|
700
|
-
|
|
701
|
-
1. Read `orders_to_enrich` and `enrichments` with `archmax_get_variables`.
|
|
702
|
-
2. The enrichments are correct when `enrichments` holds exactly one entry
|
|
703
|
-
per requested `order_id` — none missing, none extra — and each entry
|
|
704
|
-
has an `enrichment_file` and a boolean `delayed`.
|
|
705
|
-
3. Read each `enrichment_file` with `read_file`: its `delayed` must match
|
|
706
|
-
the entry's.
|
|
707
|
-
|
|
708
|
-
- Correct: advance to `enrichment-reported`.
|
|
709
|
-
- Wrong: advance back to `enrich-order`, naming exactly what is missing or
|
|
710
|
-
does not match.
|
|
711
|
-
- Before sending it back, call `archmax_get_history` and count the steps
|
|
712
|
-
in `trail` with `kind: "grader"` and `to: "enrich-order"`: if there are
|
|
713
|
-
already two, raise `enrichment-incorrect` instead.
|
|
714
|
-
transitions:
|
|
715
|
-
- to: enrichment-reported # a terminal agent state that summarizes the enrichments
|
|
716
|
-
description: Every requested order has exactly one enrichment, and each matches its file.
|
|
717
|
-
- to: enrich-order
|
|
718
|
-
description: >-
|
|
719
|
-
The enrichments are incomplete or do not match their files: send them
|
|
720
|
-
back, naming exactly what to fix.
|
|
721
|
-
```
|
|
722
|
-
|
|
723
|
-
What to know:
|
|
724
|
-
|
|
725
|
-
- A grader state **must** declare non-empty `instructions` and at least one
|
|
726
|
-
transition (`validate` errors): a grader with nowhere to send the session could
|
|
727
|
-
only approve by saying nothing, which would fail open. `evidence`, `approvers`
|
|
728
|
-
and `budget.maxParks` are errors on it. Every edge **into** a grader carries a
|
|
729
|
-
`description`: its instructions are criteria the work it judges is never shown,
|
|
730
|
-
so an undescribed edge into one is a load error.
|
|
731
|
-
- `archmax_get_history` returns `{ messages, trail }` — the same history a code
|
|
732
|
-
state reads: the whole transcript, a `tool` entry carrying its result in `text`,
|
|
733
|
-
and every trail step. Not windowed: a grader judges the whole session, so its
|
|
734
|
-
criteria say which part counts ("the files `enrichments` names", "the last
|
|
735
|
-
assistant message with text"). Check a claim against the tool result or
|
|
736
|
-
the file itself, never against the agent's retelling of it.
|
|
737
|
-
- There is **no iteration budget**. A grader that should give up after N rounds
|
|
738
|
-
says so in its `instructions` and counts the rounds from the history's trail
|
|
739
|
-
(each earlier send-back is `{ kind: "grader", to: "<graded state>" }`), as
|
|
740
|
-
above. The turn's recursion limit is the only backstop.
|
|
741
|
-
- The grader is offered exactly its state's surface — its grants (never
|
|
742
|
-
`write_todos`, `archmax_eval`, `archmax_run`, `archmax_wait` or `archmax_reset`,
|
|
743
|
-
even when `allow_always` lists them) and the controls a code state gets:
|
|
744
|
-
`archmax_advance`, `archmax_raise`, the variable tools and `archmax_get_history`
|
|
745
|
-
— governed against the grader state. Its messages never enter the session's
|
|
746
|
-
transcript and its tokens never stream; the session reads its decision as a
|
|
747
|
-
`[grader]` note and a `kind: grader` trail step.
|
|
748
|
-
- A grader that answers in text without deciding **fails closed** (`grader state
|
|
749
|
-
'<slug>' finished without calling archmax_advance or archmax_raise`), routed
|
|
750
|
-
through `on_error`. `budget.maxTurns` / `timeoutMs` bound its own model calls.
|
|
751
|
-
- `model` on a grader state names the id it runs on, resolved through the
|
|
752
|
-
assembly's `modelFactory` under the **`grader`** role.
|
|
753
|
-
- The checked state's `instructions` tell the agent that advancing into the
|
|
754
|
-
grader is what hands the work in, and to fix the work when a note sends it back
|
|
755
|
-
— never to argue with the note.
|
|
756
|
-
- **Put a grader where the agent hands work on, not on a live answer.** A turn's
|
|
757
|
-
`reply` is every assistant text of the turn, in order, so a draft the grader
|
|
758
|
-
sends back has still reached the person. Check work that waits for nobody — a
|
|
759
|
-
report, a record, a decision a person reviews next — and let a live answer end
|
|
760
|
-
where it is given.
|
|
761
|
-
|
|
762
|
-
---
|
|
763
|
-
|
|
764
|
-
## Part 2 — Cases (declarative YAML)
|
|
765
|
-
|
|
766
|
-
### Layout and the prose rules
|
|
767
|
-
|
|
768
|
-
```
|
|
769
|
-
workflows/<name>/tests/
|
|
770
|
-
<case>.test.yaml # ONE case per file (.test.yml also accepted)
|
|
771
|
-
fixtures/… # fixture files, reachable via `workspace: { <path>: { from: … } }`
|
|
772
|
-
```
|
|
773
|
-
|
|
774
|
-
- **One YAML document per file, one case per document — one conversation on
|
|
775
|
-
one session.** The case id is the tests/-relative path minus the extension
|
|
776
|
-
(`tests/refunds/approved.test.yaml` → case `refunds/approved`); the
|
|
777
|
-
`archmax test` filter argument matches that path as a substring.
|
|
778
|
-
- **Cases are data, not code.** The host parses, validates, and interprets
|
|
779
|
-
them directly — no QuickJS sandbox is created for a case. The sandbox still
|
|
780
|
-
runs code-state scripts and `archmax_run` *inside the driven agent*, unchanged.
|
|
781
|
-
- **A case reads like a script.** `steps` is one flat sequential list where
|
|
782
|
-
actions (`send`, `decide`) and assertions are peers; each assertion
|
|
783
|
-
evaluates against the view of the nearest action above it.
|
|
784
|
-
- **Only YAML files are cases.** Discovery matches `*.test.yaml`/`*.test.yml`
|
|
785
|
-
and nothing else: a leftover `*.test.js`/`*.test.ts` or `tests.config.js` is
|
|
786
|
-
ignored **without a diagnostic**, so it neither runs nor warns. Delete such
|
|
787
|
-
files once migrated (see
|
|
788
|
-
[Migrating a JS case to YAML](#migrating-a-js-case-to-yaml)).
|
|
789
|
-
- **Unknown keys are errors.** Unknown top-level keys and step-entry keys
|
|
790
|
-
fail validation — a typo'd `reachedstate:` can never silently pass. So do
|
|
791
|
-
an assertion with no preceding action, a nested `expect:` block (assertions
|
|
792
|
-
are steps themselves), and any multi-session key (`sessions`, `session`,
|
|
793
|
-
`parallel` — "not part of test spec version 1"). `archmax validate`
|
|
794
|
-
statically checks every case: schema and unknown keys, a missing or
|
|
795
|
-
over-long `title`/`description`, malformed regex strings, unknown trigger
|
|
796
|
-
ids, and `from:` fixture existence.
|
|
797
|
-
- **The title and description rules.** Every case declares two mandatory
|
|
798
|
-
prose fields, both **length-capped and enforced** — over budget is an error,
|
|
799
|
-
exactly like a missing field:
|
|
800
|
-
- **`title`** — a short one-line label, **at most 60 characters**. It is
|
|
801
|
-
what `archmax test` prints beside the verdict and what an admin UI lists,
|
|
802
|
-
so it has to fit a line: name the behavior under test
|
|
803
|
-
(`Unknown requester is rejected at entry`), not the mechanics.
|
|
804
|
-
- **`description`** — **at most 200 characters**: one or two sentences
|
|
805
|
-
stating the scenario driven and what is asserted. Not a transcript of the
|
|
806
|
-
steps, which are right below it and read as a script already.
|
|
807
|
-
- Everything longer goes in a **YAML comment above the document** — why the
|
|
808
|
-
case exists, what a particular assertion is really pinning, what was
|
|
809
|
-
deliberately mocked vs. left real, how the fixture is wired. The comment
|
|
810
|
-
has no budget; use it freely.
|
|
811
|
-
- Both budgets are measured on whitespace-collapsed text, so a folded
|
|
812
|
-
(`>-`) and a literal (`|`) block of the same prose score alike.
|
|
813
|
-
- Work title-first: write `title` and `description` before the steps, and
|
|
814
|
-
never leave them contradicting each other. This replaces the JSDoc-header
|
|
815
|
-
rule that applied to JS test files; JSDoc headers now apply to code-state
|
|
816
|
-
scripts and `archmax_run` sources only.
|
|
817
|
-
|
|
818
|
-
Run with `archmax test <workflow> [filter]`. A minimal case:
|
|
819
|
-
|
|
820
|
-
```yaml
|
|
821
|
-
# The route assertion is the point: without it the case would still pass if
|
|
822
|
-
# the agent answered the order question straight from the start state.
|
|
823
|
-
title: Order question routed into handle-order
|
|
824
|
-
description: >-
|
|
825
|
-
One turn asking about an order: the run succeeds and the audit trail
|
|
826
|
-
committed the route into handle-order.
|
|
827
|
-
steps:
|
|
828
|
-
- send: "What is the status of order ORD-1001?"
|
|
829
|
-
- succeeded: true
|
|
830
|
-
- reachedState: handle-order
|
|
831
|
-
```
|
|
832
|
-
|
|
833
|
-
### Suite config: the `tests:` block
|
|
834
|
-
|
|
835
|
-
Declarative data in `workflow.yaml` — never executed code:
|
|
836
|
-
|
|
837
|
-
```yaml
|
|
838
|
-
tests:
|
|
839
|
-
maxConcurrency: 1 # cases run at a time (only 1 or absent is accepted)
|
|
840
|
-
caseTimeoutMs:
|
|
841
|
-
120000 # per-case WALL-CLOCK budget (default 120000) —
|
|
842
|
-
# deliberately named differently from settings.timeoutMs,
|
|
843
|
-
# the per-script sandbox budget
|
|
844
|
-
judge: {} # the model that scores `grade:` ({} = env-configured
|
|
845
|
-
# default model; or { model, modelOptions }) — not a grader state
|
|
846
|
-
```
|
|
847
|
-
|
|
848
|
-
There is no `reporters` option. A
|
|
849
|
-
`judge:` entry is required for cases that declare `grade:` assertions —
|
|
850
|
-
without it the assertion records an actionable failure instead of silently
|
|
851
|
-
passing.
|
|
852
|
-
|
|
853
|
-
### Top-level keys
|
|
854
|
-
|
|
855
|
-
Unknown keys are validation **errors**. All keys except `title` and
|
|
856
|
-
`description` are optional:
|
|
857
|
-
|
|
858
|
-
| Key | Shape | Meaning |
|
|
859
|
-
| ------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
860
|
-
| `title` | string (**required**, ≤ 60 chars, one line) | Short label naming the behavior under test — what `archmax test` prints beside the verdict. Missing, multi-line, or over budget ⇒ `archmax validate` **error**. |
|
|
861
|
-
| `description` | string (**required**, ≤ 200 chars) | One or two sentences: the scenario driven and what is asserted. Missing or over budget ⇒ `archmax validate` **error**. Longer rationale belongs in a YAML comment above the document. |
|
|
862
|
-
| `skip` | string | A reason; the case reports a skipped verdict and drives nothing. |
|
|
863
|
-
| `variables` | `{ <name>: <value> }` | Session variables seeded for every driven turn, supplied as the runtime's `variables` and therefore **locked**. Values may be structured. This is where a case puts the run's input — a trigger carries none. Recover the real payload from a previous run's `artifacts/variables.json` rather than inventing one: see [Recovering a case's inputs from previous runs](#recovering-a-cases-inputs-from-previous-runs). |
|
|
864
|
-
| `trigger` | `{ id }` | The trigger every driven turn of the case runs under. Carries **only an id**; `args` is rejected with a pointer to `variables:`. Validated against the workflow machine before any step — an unknown id fails the case (even with no steps), listing the workflow's declared triggers. Omitted ⇒ the manual trigger. See [Start conditions](#start-conditions-trigger-variables-and-workspace). |
|
|
865
|
-
| `workspace` | `{ <path>: <content> }` | Files seeded into the case's run workspace **before any step**. Strings are written verbatim; non-string YAML values are JSON-serialized; a single-key `{ from: <tests/-relative path> }` mapping copies a file byte-for-byte. Only agent-visible run-zone paths are seedable (root-level files like `trigger.json`, or `scratchpad/…`); authored (`skills/…`), runtime-internal, and escaping paths are rejected before anything is written. |
|
|
866
|
-
| `mocks` | `[{ tool, whenInput?, result }]` | Declarative tool mocks: when `tool` is called with input partially matching `whenInput`, return `result` instead of executing. Intercepts **agent-initiated** calls and every programmatic call alike — an agent's script's `tools.*`, a code state's `tools.*`, a grader's calls. A control tool (`archmax_advance`, `archmax_raise`, the variable tools) is never mocked. See [the mocking section](#settle-the-mocking-depth-before-writing-a-case). |
|
|
867
|
-
| `clock` | `"2026-03-20"` / `"…T14:30:00Z"` | The date and time every model call of the case states as "now": an ISO 8601 date (midnight UTC) or a date-time with `Z` or an offset; a zoneless or impossible value is a schema error. Carried on every action, a handoff and a child session included, and outranks the host's `clock` option. See [Start conditions](#start-conditions-trigger-variables-and-workspace). |
|
|
868
|
-
| `steps` | flat list of step entries | The case script: actions and assertions as peers (below). |
|
|
869
|
-
|
|
870
|
-
There is **no multi-session surface** in test spec version 1 — no `sessions:`,
|
|
871
|
-
no `session:` on a step, no `parallel:`, and no case-level `expect:` list;
|
|
872
|
-
each is rejected with an actionable error ("not part of test spec version 1").
|
|
873
|
-
A case is one conversation on one session; per-session isolation is an engine
|
|
874
|
-
invariant covered by the SDK's own unit tests (`src/testing/driver.test.ts`),
|
|
875
|
-
not something workflow authors re-test per workflow.
|
|
876
|
-
|
|
877
|
-
### Step entries: one flat script
|
|
878
|
-
|
|
879
|
-
`steps` is a single flat, sequential list — a case reads like a script. Every
|
|
880
|
-
entry is a **single-key mapping**: an **action**, or an **assertion** that
|
|
881
|
-
evaluates against the view of the **nearest action above it** (the completed
|
|
882
|
-
turn that action produced). An assertion before the first action is a schema
|
|
883
|
-
error. The two actions:
|
|
884
|
-
|
|
885
|
-
| Action | Shape | Meaning |
|
|
886
|
-
| -------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
887
|
-
| `send` | `send: "<message>"` | One user message — a plain string, nothing else (no transport fields; start conditions live at case scope). Runs the agent until it stops (final answer, park, or failure), under the case's declared trigger. Against a run **parked at a human state** it is a message instead: answered on a reply-only turn (every tool call refused), leaving the run parked at the same state. |
|
|
888
|
-
| `decide` | `decide: { to, comment? }` | Resume a **parked** run: the simulated human picks the outgoing transition `to` (a state slug), optionally with a review comment (delivered to the agent as correction on `refine` edges). |
|
|
889
|
-
|
|
890
|
-
```yaml
|
|
891
|
-
steps:
|
|
892
|
-
- send: "I'd like a refund for order ORD-1001."
|
|
893
|
-
- parked: true # <- asserts against the send's turn
|
|
894
|
-
- reachedState: refund-review # <- same turn
|
|
895
|
-
- send: "Any news?" # a message to the parked run, not a new turn
|
|
896
|
-
- parked: decision # still parked: nothing it says routes the run
|
|
897
|
-
- decide: { to: refund-closed, comment: "ok" }
|
|
898
|
-
- succeeded: true # <- asserts against the decide's turn
|
|
899
|
-
```
|
|
900
|
-
|
|
901
|
-
`usedNoTools`/`calledTool` read the **whole conversation's** tool calls, not one
|
|
902
|
-
turn's, so do not use them to assert that a reply-only turn called nothing — in a
|
|
903
|
-
multi-step case they still see the earlier turns' calls.
|
|
904
|
-
|
|
905
|
-
There is no nested `expect:` block — writing one is an error with guidance
|
|
906
|
-
that assertions are steps themselves.
|
|
907
|
-
|
|
908
|
-
### Assertion grammar
|
|
909
|
-
|
|
910
|
-
Each assertion is a single-key step entry (repeatable; order-preserving),
|
|
911
|
-
evaluated against the nearest action above it. **Every assertion is hard** —
|
|
912
|
-
`grade` included: the `atLeast` you declare is the bar, and a score under it
|
|
913
|
-
fails the case and the run.
|
|
914
|
-
|
|
915
|
-
Assertions marked **structural** below assert *where the run went*, and a
|
|
916
|
-
failure in one of them **halts the case** — see
|
|
917
|
-
[Structural assertions halt the case](#structural-assertions-halt-the-case).
|
|
918
|
-
|
|
919
|
-
| Assertion | Meaning |
|
|
920
|
-
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
921
|
-
| `succeeded: true` | **Structural.** Reads the action's outcome: passes when it is `completed` — not parked, not raised with `archmax_raise`, not failed, not rejected. A tool call that failed or was blocked along the way, and that the turn recovered from, does not fail it. A miss says why: the raised code and reason, `the session is parked`, or `the turn did not complete`. |
|
|
922
|
-
| `parked: true` | **Structural.** The run is parked at a human state awaiting a decision. |
|
|
923
|
-
| `raised: true` / `raised: <code>` / `raised: { code?, reason? }` | **Structural.** The agent ended the turn with `archmax_raise({ code, reason })`. A string pins the code exactly; the mapping pins the code and matches the reason — exactly, or as a regex when written `"/pattern/flags"`. A miss names the code that was raised, or says the run did not raise. |
|
|
924
|
-
| `variables: { expect, path?, locked? }` | Every named variable is set to the expected value — deep equality, so a structured expectation compares by value. `path` addresses into a value per name (`items.-1.sku` is the last element); `locked` asserts the lock state. |
|
|
925
|
-
| `reachedState: <slug>` | **Structural.** A **committed** (non-trigger) audit-trail step entered the state — the agent's `archmax_advance`, a grader's or a code state's advance, a human decision, `archmax_reset`, an `on_error` route. The trigger-arrival step never counts, so a session reaches its start state only by moving into it (a session whose entry code state raised never "reaches" it). |
|
|
926
|
-
| `reply: { includes?, excludes? }` | `includes`: substrings or `/pattern/flags` regex strings that must **all** match; `excludes`: none may match (leak checks — a refusal that quotes the data it refuses still leaks it). Each accepts a **single token or a list** (`includes: "ORD-1003"` works). Matched against **every assistant message of the session so far** (earlier turns included), not just the final message (a parked run often closes with a short "moving on" message while the substantive answer came earlier). |
|
|
927
|
-
| `calledTool: { name, input? }` | The turn recorded a call to tool `name`; `input` is a **partial** match — mappings match on the declared keys (nested mappings partially), arrays by deep equality, and `/pattern/flags` string values match as **regexes** (recommended for file paths, so the model's exact spelling doesn't matter). |
|
|
928
|
-
| `notCalledTool: { name, input? }` | No matching call occurred. |
|
|
929
|
-
| `blockedTool: { name, input? }` | A matching call was made **and refused by governance** — the assertion for testing a guard. Distinct from `notCalledTool` (nothing was tried, which proves nothing about the guard) and from a tool that ran and errored (only a policy rejection counts). Use it to test a state's `tools.allow` narrowing or its `skills.allow` set. |
|
|
930
|
-
| `usedNoTools: true` | The turn's agent used no tools at all. An automated state's calls are not in the transcript, so they never count. |
|
|
931
|
-
| `trail: { to?, kind?, reason?, count }` | **Structural.** The number of audit-trail steps matching **all** given fields equals `count` (declare at least one of `to`/`kind`/`reason`). `kind`: `trigger` \| `agent` \| `human` \| `grader` \| `code` \| `sub-workflow` \| `reset` \| `on_error` — a grader's or code state's move is `grader`/`code`, with the `reason` it passed to `archmax_advance`. `count: 0` asserts a state was never entered. The trail accumulates across turns — each turn starts with a `trigger` step. |
|
|
932
|
-
| `noTraversal: true` | **Structural.** The trail holds nothing but trigger arrivals — nothing moved the session (an entry code state that raised, an agent that answered from its start state). A grader's or code state's move is a traversal. |
|
|
933
|
-
| `triggerArrival: <trigger-id>` | **Structural.** Some trail step has `kind: trigger` and that trigger id as its reason — the run really started from the declared trigger. |
|
|
934
|
-
| `grade: { closedQA, atLeast }` | LLM-grade the turn's **run evidence** against the `closedQA` criterion — what the action's turn said (every text of that turn, empty when it said nothing) *plus* a bounded chronological record of the session's assistant messages and tool calls so far (each call with input, output, and status; mocked calls appear like real ones). The two are weighed differently: a criterion about what the user was **told** is met only by the reply, one about what the agent **did** may be met by the record — so `closedQA: looked the order up before answering` is a legitimate criterion, not only phrasing checks. **`atLeast` decides the outcome** — score ≥ `atLeast` passes, below it **fails the case**; the judge model's own opinion never overrides your declared bar, so the record can't disagree with the verdict. The judge is instructed to keep its reason to 1–3 sentences, so it fits on the verdict line. Requires `tests.judge` in `workflow.yaml`. |
|
|
935
|
-
|
|
936
|
-
**Regex form**: a string shaped `/pattern/flags` is compiled as a regular
|
|
937
|
-
expression; anything else matches as a plain substring (in `reply` tokens) or
|
|
938
|
-
by equality (in tool-input matchers). Malformed regex strings are **static
|
|
939
|
-
validation errors**, never silently matched as literals. Quote them in YAML —
|
|
940
|
-
`"/approv/i"`, `"/output\\/report\\.json/"`.
|
|
941
|
-
|
|
942
|
-
**There is no generic predicate.** No `satisfies`, no inline JS — that is the
|
|
943
|
-
point of the format. Bespoke logic belongs in a vitest unit test against the
|
|
944
|
-
engine or in a code state on the graph.
|
|
945
|
-
|
|
946
|
-
**An automated state is asserted by what it committed.** A code state's or a
|
|
947
|
-
grader's own tool calls are not in the transcript, so `calledTool`,
|
|
948
|
-
`notCalledTool`, `blockedTool` and `usedNoTools` see only the agent's, and a
|
|
949
|
-
grader's words are never part of `reply`. Assert its move and reason with
|
|
950
|
-
`reachedState` / `trail: { kind: code | grader, to, reason?, count }`, its raise
|
|
951
|
-
with `raised`, what it set with `variables`, and its dispatches with
|
|
952
|
-
`ranWorkflow`:
|
|
953
|
-
|
|
954
|
-
```yaml
|
|
955
|
-
steps:
|
|
956
|
-
- send: "Hi, I'm support@acmecorp.com. Which orders are delayed?"
|
|
957
|
-
- trail: { kind: code, to: identify-case, count: 1 } # the entry screen passed it
|
|
958
|
-
```
|
|
959
|
-
|
|
960
|
-
```yaml
|
|
961
|
-
trigger: { id: enrichment_requested } # the enrichment path
|
|
962
|
-
variables: { orders_to_enrich: [{ order_id: "ORD-1002" }, { order_id: "ORD-1003" }] }
|
|
963
|
-
steps:
|
|
964
|
-
- send: "Enrich the requested orders."
|
|
965
|
-
- trail: { kind: grader, to: enrichment-reported, count: 1 } # the grader approved the enrichments
|
|
966
|
-
- trail: { kind: grader, to: enrich-order, count: 0 } # without a send-back
|
|
967
|
-
```
|
|
968
|
-
|
|
969
|
-
### Structural assertions halt the case
|
|
970
|
-
|
|
971
|
-
A failed **structural** assertion (`succeeded`, `parked`, `raised`,
|
|
972
|
-
`reachedState`, `trail`, `noTraversal`, `triggerArrival`) stops the case where it stands: no
|
|
973
|
-
further `send`/`decide` is driven and no further assertion is evaluated. Once
|
|
974
|
-
the run took a path the case did not describe, every later step would drive or
|
|
975
|
-
grade the wrong run — a `decide` after a failed `parked` cannot succeed, and a
|
|
976
|
-
`reply` check after a failed `reachedState` grades a state the run never
|
|
977
|
-
entered.
|
|
978
|
-
|
|
979
|
-
Content assertions (`reply`, `calledTool`, `notCalledTool`, `blockedTool`, `usedNoTools`,
|
|
980
|
-
`grade`) do **not** halt: a wrong reply is a defect in the run, not evidence the
|
|
981
|
-
run stopped being the one under test.
|
|
982
|
-
|
|
983
|
-
**What this means for how you author a case.** Put the structural assertion you
|
|
984
|
-
depend on immediately after the action that should have satisfied it, before the
|
|
985
|
-
content assertions that only make sense in that state:
|
|
986
|
-
|
|
987
|
-
```yaml
|
|
988
|
-
steps:
|
|
989
|
-
- send: "I want a refund for ORD-1001"
|
|
990
|
-
- reachedState: refund-review # ← structural: if this misses, stop here
|
|
991
|
-
- parked: true
|
|
992
|
-
- reply: # ← only meaningful in refund-review
|
|
993
|
-
includes: "approved"
|
|
994
|
-
- decide: { to: refund-closed }
|
|
995
|
-
- succeeded: true
|
|
996
|
-
```
|
|
997
|
-
|
|
998
|
-
A halt is reported as an ordinary failed verdict naming the assertion that
|
|
999
|
-
missed — **no error is set**, because a halt is a verdict the case reached, not
|
|
1000
|
-
a failure of the machinery. Every assertion step that never ran is reported with
|
|
1001
|
-
`status: "not-executed"` (one record per step, carrying its `steps` index), so a
|
|
1002
|
-
stopped case still accounts for everything it declared: `archmax test` prints an
|
|
1003
|
-
`N assertions not executed` line, and a host reading records can grey them out
|
|
1004
|
-
instead of showing nothing and implying they passed.
|
|
1005
|
-
|
|
1006
|
-
Each record's `status` — `passed` \| `failed` \| `not-executed` — is the
|
|
1007
|
-
authoritative outcome; the boolean `pass` is the alias `status === "passed"`.
|
|
1008
|
-
|
|
1009
|
-
### Start conditions: trigger, variables, and workspace
|
|
1010
|
-
|
|
1011
|
-
A case declares its **start conditions** — which trigger its driven runs start
|
|
1012
|
-
from, what input those runs carry, and which files exist in the run workspace
|
|
1013
|
-
before anything happens — at **case scope, never per step**:
|
|
1014
|
-
|
|
1015
|
-
```yaml
|
|
1016
|
-
title: Delayed-orders report compiled from a trigger
|
|
1017
|
-
description: >-
|
|
1018
|
-
The report_requested trigger starts every driven turn, carrying the company
|
|
1019
|
-
the firing named as a locked run variable.
|
|
1020
|
-
trigger:
|
|
1021
|
-
id: report_requested
|
|
1022
|
-
variables:
|
|
1023
|
-
company: "Acme Corp" # locked, exactly as a host's variables would be
|
|
1024
|
-
workspace:
|
|
1025
|
-
trigger.json: { company: "Acme Corp" } # non-string ⇒ JSON-serialized
|
|
1026
|
-
steps:
|
|
1027
|
-
- send: "A delayed-orders report was requested."
|
|
1028
|
-
- succeeded: true
|
|
1029
|
-
- triggerArrival: report_requested
|
|
1030
|
-
```
|
|
1031
|
-
|
|
1032
|
-
- **`trigger`** — every driven turn of the case runs under this trigger
|
|
1033
|
-
(each turn restarts the run at that trigger's start state; the trail
|
|
1034
|
-
records `{ kind: trigger, reason: <id> }` per turn). Omit it for the
|
|
1035
|
-
`manual` trigger. It is validated **before any step executes**: an unknown
|
|
1036
|
-
id fails the case with the workflow's declared triggers, even when the case
|
|
1037
|
-
has no steps — and `archmax validate` reports the same statically. There is
|
|
1038
|
-
deliberately no per-step trigger: a follow-up `send` cannot run under a
|
|
1039
|
-
different trigger than the case declares. **When the workflow declares
|
|
1040
|
-
non-manual triggers, every case exercising one MUST declare it.**
|
|
1041
|
-
- **`variables`** — the run's input, seeded on every driven turn as the
|
|
1042
|
-
runtime's `variables` and therefore **locked**, values structured or
|
|
1043
|
-
scalar. A seed here is indistinguishable from a production host seed to
|
|
1044
|
-
everything downstream: a `${{…}}` reference in a `tools.allow` glob resolves
|
|
1045
|
-
from it, a state's `requires:` gate is satisfied by it, a code state's script
|
|
1046
|
-
reads it in `variables`, and `archmax_set_variables` refuses to rewrite it.
|
|
1047
|
-
**Recover the real payload rather than inventing one** — see
|
|
1048
|
-
[Recovering a case's inputs from previous runs](#recovering-a-cases-inputs-from-previous-runs).
|
|
1049
|
-
- **`workspace`** — input files seeded before any step. Keys are run-zone
|
|
1050
|
-
paths (root-level files like `trigger.json`, or `scratchpad/…`); authored
|
|
1051
|
-
paths (`skills/…`) and runtime areas are rejected before anything is
|
|
1052
|
-
written. This mirrors the host convention of materializing a trigger's
|
|
1053
|
-
payload as a workspace file — declare the payload exactly as production
|
|
1054
|
-
would write it.
|
|
1055
|
-
|
|
1056
|
-
Two operational notes:
|
|
1057
|
-
|
|
1058
|
-
- Seeding writes the file; **reading it is still governed** — the triggered
|
|
1059
|
-
start state must allow the read tool the agent needs (`read_file` is granted
|
|
1060
|
-
wherever the workflow-wide grant includes it, unless a state narrows it).
|
|
1061
|
-
- Hosts that pin case sessions via `sessionIdForCase` reuse sessions across test
|
|
1062
|
-
runs: declared files are overwritten each run, but a file seeded by an
|
|
1063
|
-
_earlier_ run whose declaration was since removed can linger. The default
|
|
1064
|
-
(per-run unique sessions) starts clean.
|
|
1065
|
-
|
|
1066
|
-
**Pin the date** with a case-level `clock:` whenever a fixture or a prompt
|
|
1067
|
-
carries a date (an order placed on `2026-03-15`, "refundable for 30 days"):
|
|
1068
|
-
without it the model reads the real "now" and the verdict drifts with the
|
|
1069
|
-
calendar. Pin on a ten-minute mark (the line is rounded down to ten minutes,
|
|
1070
|
-
UTC only). Only the prompt is pinned — code-state scripts, the sandbox's `Date`, a
|
|
1071
|
-
park's `resumeAt` and every recorded timestamp stay real, so a script that needs
|
|
1072
|
-
the case's "today" reads it from a seeded variable, and a wait resumed in the
|
|
1073
|
-
case is resumed with `deliver`, not by its `until`.
|
|
1074
|
-
|
|
1075
|
-
### Recovering a case's inputs from previous runs
|
|
1076
|
-
|
|
1077
|
-
A trigger carries only an id, so everything a firing passes arrives as
|
|
1078
|
-
variables — and those names, shapes, and values live in the host's call site and
|
|
1079
|
-
in sessions that already ran, not in the workflow. Read a real run rather than
|
|
1080
|
-
guessing:
|
|
1081
|
-
|
|
1082
|
-
```bash
|
|
1083
|
-
archmax sessions # ids, status, state, parked state, `vars=` names
|
|
1084
|
-
archmax sessions <sessionId> # that session's variables: value + locked state
|
|
1085
|
-
```
|
|
1086
|
-
|
|
1087
|
-
That reads the session's **checkpoint**, so it works for any session that has run,
|
|
1088
|
-
including one driven by `archmax run` (the CLI emits no run artifacts). A host that
|
|
1089
|
-
calls `emitRunArtifacts` leaves the same record on disk as
|
|
1090
|
-
`sessions/<sessionId>/artifacts/variables.json`
|
|
1091
|
-
(`{ sessionId, workflow, variables: { name: { value, locked } } }`).
|
|
1092
|
-
|
|
1093
|
-
Pick the newest session whose recorded `trigger` matches the trigger the case
|
|
1094
|
-
drives, then map its store onto the case:
|
|
1095
|
-
|
|
1096
|
-
| Recorded entry | Where it belongs |
|
|
1097
|
-
| --- | --- |
|
|
1098
|
-
| `trigger` (always locked) | the case's `trigger: { id: … }`, never `variables:` |
|
|
1099
|
-
| any other `locked: true` | the case's `variables:` block — the firing's payload |
|
|
1100
|
-
| `locked: false` | nothing: the agent established it, so **assert** it with `variables: { expect: … }` |
|
|
1101
|
-
|
|
1102
|
-
Never copy the whole store into `variables:`. Case seeds are locked, so seeding a
|
|
1103
|
-
name the agent should establish hides a missing `archmax_set_variables` call *and*
|
|
1104
|
-
makes the agent's own write fail against the lock — the case then exercises a run
|
|
1105
|
-
production never performs.
|
|
1106
|
-
|
|
1107
|
-
`locked` is the whole attribution, and it is sufficient: a locked entry came from
|
|
1108
|
-
the host (an `variables` seed, a delivery's seeds, or the built-in
|
|
1109
|
-
`trigger`), an unlocked one from the agent's own `set_variables`. What you read is
|
|
1110
|
-
the store as the session's **latest** turn left it, so locked entries are also the
|
|
1111
|
-
turn-invariant part.
|
|
1112
|
-
|
|
1113
|
-
To produce a payload to read back, drive one yourself:
|
|
1114
|
-
`archmax run <workflow> "<prompt>" --variables '{"from_email":"a@b.c"}'` then
|
|
1115
|
-
`archmax sessions <id>`.
|
|
1116
|
-
|
|
1117
|
-
With no previous run to read, derive the same facts statically: the `${{…}}`
|
|
1118
|
-
references in `tools.allow`/`allow_always` and the names in every state's
|
|
1119
|
-
`requires:` are the variables the workflow depends on (`archmax validate` warns
|
|
1120
|
-
about references nothing guarantees), and the host's
|
|
1121
|
-
`createAgent({ variables })` / `deliver(sessionId, { trigger,
|
|
1122
|
-
variables })` call sites name what production supplies.
|
|
1123
|
-
|
|
1124
|
-
### File fixtures: `from:` references
|
|
1125
|
-
|
|
1126
|
-
Bulk fixtures (a captured `trigger.json`, a large orders dump) live as real
|
|
1127
|
-
files under `tests/` rather than inline YAML, and a case names each one it
|
|
1128
|
-
wants: any `workspace` entry may declare `{ from: <path> }` instead of inline
|
|
1129
|
-
content. The path resolves relative to the workflow's `tests/` directory, so
|
|
1130
|
-
`from: fixtures/trigger.json` and `from: shared/orders-dump.json` (a fixture
|
|
1131
|
-
shared across cases) both work. There is **no auto-seeded sibling data
|
|
1132
|
-
directory**: a file is seeded only when a case's `workspace:` says so.
|
|
1133
|
-
|
|
1134
|
-
Fail-closed rules — reported by `archmax validate` and enforced again at run
|
|
1135
|
-
bootstrap: a `from:` path that escapes `tests/` or names a missing file is an
|
|
1136
|
-
error before any step runs, and a `*.test.yaml` file anywhere under `tests/`
|
|
1137
|
-
is a case.
|
|
1138
|
-
|
|
1139
|
-
### Settle the mocking depth _before_ writing a case
|
|
1140
|
-
|
|
1141
|
-
An unmocked tool call in a test is a **real** tool call, on every run of the
|
|
1142
|
-
suite: a real email sent, a real card charged, a real row written to a live
|
|
1143
|
-
system. That is the one decision about a test its author cannot make alone, so
|
|
1144
|
-
**ask the owner how much of the workflow's tool surface to mock before you
|
|
1145
|
-
write the file**.
|
|
1146
|
-
|
|
1147
|
-
Ask concretely rather than in the abstract — name the tools this workflow
|
|
1148
|
-
actually calls and propose a split:
|
|
1149
|
-
|
|
1150
|
-
- **Mock by default** anything that sends, pays, posts, deletes, or writes
|
|
1151
|
-
outside the session zone, and anything whose result drifts between runs (live
|
|
1152
|
-
inventory, today's date-dependent data). A test that is not safe to re-run is
|
|
1153
|
-
not a test.
|
|
1154
|
-
- **Leave real by default** reads against fixtures committed to the workspace,
|
|
1155
|
-
the workflow's own scripts and code states, and run-zone file writes — mocking
|
|
1156
|
-
those would test the mock instead of the workflow.
|
|
1157
|
-
- **Ask** about reads against a live system: cheap and stable enough to keep
|
|
1158
|
-
real, or slow, rate-limited, or credential-bound enough to mock?
|
|
1159
|
-
|
|
1160
|
-
Record the answer in the **YAML comment above the case document** — the
|
|
1161
|
-
`description` is capped at a sentence or two and belongs to what is asserted,
|
|
1162
|
-
while a reader checking the case against the workflow needs to know what was
|
|
1163
|
-
deliberately left real, which is exactly the kind of rationale the comment
|
|
1164
|
-
exists for.
|
|
1165
|
-
|
|
1166
|
-
Mocks are the top-level `mocks:` list; they intercept **agent-initiated** calls
|
|
1167
|
-
and every programmatic call — an agent's script, a code state's script, a
|
|
1168
|
-
grader — with the same partial-input matching as `calledTool`. So a mock of a
|
|
1169
|
-
file an entry code state reads changes what that state decides too. Interception is guaranteed under `archmax test`. A host running
|
|
1170
|
-
the same cases through `runTests` with its own `createTarget` must wire
|
|
1171
|
-
`createToolMockMiddleware()` into that agent — a case that declares `mocks:`
|
|
1172
|
-
against a target that cannot intercept them is refused with an error **before
|
|
1173
|
-
the agent runs**, never run half-mocked. Mockless cases run against any
|
|
1174
|
-
workflow-governed target. See
|
|
1175
|
-
[backend-integration.md](backend-integration.md#cases-against-your-agent).
|
|
1176
|
-
|
|
1177
|
-
### Worked example: happy path + grade
|
|
1178
|
-
|
|
1179
|
-
```yaml
|
|
1180
|
-
# `reachedState` is what pins the routing: without it the case would still
|
|
1181
|
-
# pass if the agent answered straight from the start state. The judge
|
|
1182
|
-
# (threshold 0.7) checks the order is attributed to the requester's company.
|
|
1183
|
-
title: Delayed orders answered for a known requester
|
|
1184
|
-
description: >-
|
|
1185
|
-
A known requester's question routes into orders-question, and the reply names
|
|
1186
|
-
the one delayed order, ORD-1003.
|
|
1187
|
-
steps:
|
|
1188
|
-
- send: "Hi, I'm support@acmecorp.com. Which orders are delayed for Acme?"
|
|
1189
|
-
- succeeded: true
|
|
1190
|
-
- reachedState: orders-question
|
|
1191
|
-
- reply:
|
|
1192
|
-
includes: ["ORD-1003", "/delayed/i"]
|
|
1193
|
-
- grade:
|
|
1194
|
-
closedQA: >-
|
|
1195
|
-
confirms that order ORD-1003 is the delayed order for the requester's
|
|
1196
|
-
company (Acme Corp)
|
|
1197
|
-
atLeast: 0.7
|
|
1198
|
-
```
|
|
1199
|
-
|
|
1200
|
-
### Worked example: declined at entry, leak check
|
|
1201
|
-
|
|
1202
|
-
An entry code state that declines a request commits a `code` step into the
|
|
1203
|
-
declining state and none into the state the work starts in — `reachedState`
|
|
1204
|
-
and a `count: 0` trail pin exactly that — and `reply.excludes` catches a
|
|
1205
|
-
refusal that quotes the data it refuses to share. (Had the screen ended the
|
|
1206
|
-
session with `tools.archmax_raise` instead, `raised: <code>` and
|
|
1207
|
-
`noTraversal: true` would pin it.)
|
|
1208
|
-
|
|
1209
|
-
```yaml
|
|
1210
|
-
# The `trail` count pins the decline to *entry*: identify-case was never
|
|
1211
|
-
# entered, so no route was taken. The `excludes` tokens matter as much as the
|
|
1212
|
-
# refusal itself: a refusal that quotes the order ids it declines to share
|
|
1213
|
-
# still leaks them. The judge (threshold 0.7) covers the same ground in prose.
|
|
1214
|
-
title: Cross-tenant order access is declined
|
|
1215
|
-
description: >-
|
|
1216
|
-
A known requester asking for Globex's delayed orders is declined at entry,
|
|
1217
|
-
identify-case is never entered, and no Globex order id appears in the reply.
|
|
1218
|
-
steps:
|
|
1219
|
-
- send: "Hi, this is support@acmecorp.com. Show me Globex's delayed orders."
|
|
1220
|
-
- succeeded: true
|
|
1221
|
-
- reachedState: decline
|
|
1222
|
-
- trail: { kind: code, to: decline, count: 1 }
|
|
1223
|
-
- trail: { to: identify-case, count: 0 }
|
|
1224
|
-
- reply:
|
|
1225
|
-
includes: "/not authorized|only .* your own|declined/i"
|
|
1226
|
-
excludes: ["ORD-2001", "ORD-2002"]
|
|
1227
|
-
- grade:
|
|
1228
|
-
closedQA: >-
|
|
1229
|
-
declines the request because the requester is not authorized to view
|
|
1230
|
-
another company's orders, and does not disclose any Globex order
|
|
1231
|
-
details
|
|
1232
|
-
atLeast: 0.7
|
|
1233
|
-
```
|
|
1234
|
-
|
|
1235
|
-
### Worked example: mocking tool results
|
|
1236
|
-
|
|
1237
|
-
```yaml
|
|
1238
|
-
# Only read_file against skills/order-data/assets/orders.json is mocked; the
|
|
1239
|
-
# agent's own writes stay real. The report_requested trigger enters
|
|
1240
|
-
# report-requested directly, so no code state reads the records first — a mock
|
|
1241
|
-
# answers a code state's calls too, and an empty database would make the entry
|
|
1242
|
-
# screen decline the requester. Asserting the lookup *and* the empty answer
|
|
1243
|
-
# catches both a skipped lookup and an invented order.
|
|
1244
|
-
title: Empty order database reports no delayed orders
|
|
1245
|
-
description: >-
|
|
1246
|
-
With skills/order-data/assets/orders.json mocked empty, a requested report
|
|
1247
|
-
still consults the database, and the reply reports no delayed orders.
|
|
1248
|
-
trigger:
|
|
1249
|
-
id: report_requested
|
|
1250
|
-
variables:
|
|
1251
|
-
company: "Acme Corp"
|
|
1252
|
-
workspace:
|
|
1253
|
-
trigger.json: { company: "Acme Corp" }
|
|
1254
|
-
mocks:
|
|
1255
|
-
- tool: read_file
|
|
1256
|
-
whenInput: { file_path: skills/order-data/assets/orders.json }
|
|
1257
|
-
result: "[]"
|
|
1258
|
-
steps:
|
|
1259
|
-
- send: "A delayed-orders report was requested."
|
|
1260
|
-
- calledTool:
|
|
1261
|
-
name: read_file
|
|
1262
|
-
input:
|
|
1263
|
-
file_path: "/orders\\.json/"
|
|
1264
|
-
- reply:
|
|
1265
|
-
includes: "/no (matching |delayed )?orders/i"
|
|
1266
|
-
```
|
|
1267
|
-
|
|
1268
|
-
### Worked example: mocking a sub-workflow
|
|
1269
|
-
|
|
1270
|
-
A delegation is a tool call, so it mocks like one — keyed by the tool's own name,
|
|
1271
|
-
matched on the call's arguments. Each call gets its own answer, which is what
|
|
1272
|
-
proves a fan-out is not sharing inputs:
|
|
1273
|
-
|
|
1274
|
-
```yaml
|
|
1275
|
-
mocks:
|
|
1276
|
-
- tool: archmax_workflow_enrich-order
|
|
1277
|
-
whenInput: { order_id: "ORD-1002" }
|
|
1278
|
-
result:
|
|
1279
|
-
message: "ORD-1002 is shipped."
|
|
1280
|
-
# The target's `manual` trigger declares these, so a mock standing in
|
|
1281
|
-
# for that sub-run has to supply them: a mock replaces the run, never its
|
|
1282
|
-
# contract, and one omitting a declared name fails the dispatch.
|
|
1283
|
-
returns:
|
|
1284
|
-
enrichment_file: "scratchpad/enrichment/ORD-1002.json"
|
|
1285
|
-
delayed: false
|
|
1286
|
-
steps:
|
|
1287
|
-
- send: "Enrich the requested orders."
|
|
1288
|
-
- ranWorkflow: { workflow: enrich-order, count: 2 }
|
|
1289
|
-
- variables:
|
|
1290
|
-
expect:
|
|
1291
|
-
enrichments:
|
|
1292
|
-
- order_id: "ORD-1002"
|
|
1293
|
-
delayed: false
|
|
1294
|
-
```
|
|
1295
|
-
|
|
1296
|
-
Assert with **`ranWorkflow`**, not `calledTool`, whenever a script drives the
|
|
1297
|
-
fan-out: `calledTool` reads the *model's* calls, and the point of a script-driven
|
|
1298
|
-
fan-out is that the model makes none of them. `ranWorkflow` reads the trail, which
|
|
1299
|
-
records a step per sub-run whoever made the call — including a mocked one.
|
|
1300
|
-
|
|
1301
|
-
### Worked example: human-in-the-loop decisions (refine loop)
|
|
1302
|
-
|
|
1303
|
-
Drive a run into a human state, assert it parked, then decide as the reviewer —
|
|
1304
|
-
including the send-back-for-changes (`refine`) loop, whose second parking is
|
|
1305
|
-
pinned with a `trail` count:
|
|
1306
|
-
|
|
1307
|
-
```yaml
|
|
1308
|
-
# The `trail` count is the point: refund-review is entered exactly twice —
|
|
1309
|
-
# initial submit plus one resubmit — so a third entry would mean the refine
|
|
1310
|
-
# edge looped. The final `reply` check pins that the correction did not lose
|
|
1311
|
-
# its subject.
|
|
1312
|
-
title: Refund sent back for changes, then approved
|
|
1313
|
-
description: >-
|
|
1314
|
-
The reviewer refines once and then approves: the run parks at both review
|
|
1315
|
-
points, refund-review is entered exactly twice, the run succeeds, and the
|
|
1316
|
-
final reply still names ORD-1001.
|
|
1317
|
-
steps:
|
|
1318
|
-
- send: "Hi, I'm support@acmecorp.com. I'd like a refund for order ORD-1001."
|
|
1319
|
-
- parked: true
|
|
1320
|
-
- decide:
|
|
1321
|
-
to: refund-request
|
|
1322
|
-
comment: >-
|
|
1323
|
-
Please restate the refund amount and confirm the order was delivered
|
|
1324
|
-
before it can be approved.
|
|
1325
|
-
- parked: true
|
|
1326
|
-
- trail:
|
|
1327
|
-
to: refund-review
|
|
1328
|
-
count: 2
|
|
1329
|
-
- decide:
|
|
1330
|
-
to: refund-closed
|
|
1331
|
-
comment: "The explanation is clear now."
|
|
1332
|
-
- succeeded: true
|
|
1333
|
-
- reply:
|
|
1334
|
-
includes: "ORD-1001"
|
|
1335
|
-
```
|
|
1336
|
-
|
|
1337
|
-
### Worked example: triggered start + fixture file
|
|
1338
|
-
|
|
1339
|
-
The case declares its trigger on the document, its input as a locked variable,
|
|
1340
|
-
and a payload fixture the host would materialize as a real file under `tests/`,
|
|
1341
|
-
seeded byte-for-byte before any step:
|
|
1342
|
-
|
|
1343
|
-
```yaml
|
|
1344
|
-
# tests/triggered-report.test.yaml — tests/fixtures/trigger.json holds the
|
|
1345
|
-
# payload fixture. `triggerArrival` is what pins the start: it names the
|
|
1346
|
-
# trigger the run really arrived on, not merely the state it ended up in.
|
|
1347
|
-
title: Delayed-orders report compiled from a trigger
|
|
1348
|
-
description: >-
|
|
1349
|
-
The report_requested trigger starts the run: the agent reads the seeded
|
|
1350
|
-
trigger.json, writes scratchpad/report.json, and the reply names Acme's one
|
|
1351
|
-
delayed order, ORD-1003.
|
|
1352
|
-
trigger:
|
|
1353
|
-
id: report_requested
|
|
1354
|
-
variables:
|
|
1355
|
-
company: "Acme Corp"
|
|
1356
|
-
workspace:
|
|
1357
|
-
trigger.json: { from: fixtures/trigger.json }
|
|
1358
|
-
steps:
|
|
1359
|
-
- send: "A delayed-orders report was requested."
|
|
1360
|
-
- succeeded: true
|
|
1361
|
-
- triggerArrival: report_requested
|
|
1362
|
-
# `/pattern/` input values match as regexes, so the assertion holds
|
|
1363
|
-
# whether the model spells the path with or without a leading slash.
|
|
1364
|
-
- calledTool:
|
|
1365
|
-
name: read_file
|
|
1366
|
-
input:
|
|
1367
|
-
file_path: "/trigger\\.json/"
|
|
1368
|
-
- calledTool:
|
|
1369
|
-
name: write_file
|
|
1370
|
-
input:
|
|
1371
|
-
file_path: "/scratchpad\\/report\\.json/"
|
|
1372
|
-
- reply:
|
|
1373
|
-
includes: ["ORD-1003", "Acme"]
|
|
1374
|
-
```
|
|
1375
|
-
|
|
1376
|
-
### Migrating a JS case to YAML
|
|
1377
|
-
|
|
1378
|
-
JS test cases (`defineEval`, the sandbox `t` driver, `@archmax-ai/harness/testing`) are
|
|
1379
|
-
**removed**; a `*.test.js`/`*.test.ts` file is no longer recognized and is
|
|
1380
|
-
skipped silently. The mapping:
|
|
1381
|
-
|
|
1382
|
-
| JS surface (removed) | YAML equivalent |
|
|
1383
|
-
| ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
|
|
1384
|
-
| File JSDoc header + `defineEval({ description })` | The mandatory `title` (≤ 60 chars) + `description` (≤ 200 chars); rationale that fits neither moves to a YAML comment above the document |
|
|
1385
|
-
| `defineEval({ trigger })` / `({ workspace })` | Top-level `trigger:` / `workspace:` (inline content or `{ from: … }`) |
|
|
1386
|
-
| `await t.send("msg")` | `- send: "msg"` step entry |
|
|
1387
|
-
| `await t.decide(to, comment)` | `- decide: { to, comment }` step entry |
|
|
1388
|
-
| `t.newSession(...)` / `Promise.all` over sessions | **Not part of test spec version 1** — a case is one conversation on one session; per-session isolation is an engine invariant covered by the SDK's own unit tests |
|
|
1389
|
-
| `t.mockTool(name, { whenInput, result })` | `mocks: [{ tool, whenInput, result }]` |
|
|
1390
|
-
| `t.succeeded()` / `t.parked()` | `- succeeded: true` / `- parked: true` step entries after the action |
|
|
1391
|
-
| `t.messageIncludes("x")` / `t.messageIncludes(/x/i)` | `- reply: { includes: ["x", "/x/i"] }` step entry (single token or list) |
|
|
1392
|
-
| Leak checks via `satisfies(...)` on the reply | `- reply: { excludes: [...] }` step entry |
|
|
1393
|
-
| `t.calledTool(name, { input })` / `t.notCalledTool` / `t.usedNoTools()` | `- calledTool: { name, input? }` / `- notCalledTool:` / `- usedNoTools: true` step entries (RegExp input values become `"/pattern/flags"` strings) |
|
|
1394
|
-
| `t.reachedState(name)` | `- reachedState: <name>` step entry |
|
|
1395
|
-
| `t.auditTrail()` + `filter(...).length === n` | `- trail: { to?, kind?, reason?, count }`; "nothing but the trigger arrival" ⇒ `- noTraversal: true`; trigger-arrival reason ⇒ `- triggerArrival: <id>` |
|
|
1396
|
-
| `t.judge.autoevals.closedQA(c).atLeast(n)` | `- grade: { closedQA: c, atLeast: n }` step entry |
|
|
1397
|
-
| `t.skip(reason)` | `skip: <reason>` |
|
|
1398
|
-
| bespoke predicates | **No YAML equivalent — deliberate.** Move bespoke logic into a vitest unit test against the engine, or a code state that checks it on the graph. |
|
|
1399
|
-
| `import { defineEval } from "@archmax-ai/harness/testing"` | Nothing — the subpath is removed; YAML imports nothing |
|
|
1400
|
-
|
|
1401
|
-
Removed package exports: `createDriveTool`, `DriveRequest`, `DriveSession`,
|
|
1402
|
-
and the `@archmax-ai/harness/testing` subpath. New root exports for hosts that parse
|
|
1403
|
-
cases themselves: `parseCaseDocument`, `CaseSchemaError`, the case types
|
|
1404
|
-
(`CaseDocument`, `CaseStep`, `CaseExpectation`, …), and the test-spec
|
|
1405
|
-
versioning surface (`DEFAULT_TEST_SPEC_VERSION`, `SUPPORTED_TEST_SPEC_VERSIONS`,
|
|
1406
|
-
`resolveTestSpecVersion`, `isTestSpecVersionSupported`,
|
|
1407
|
-
`UnsupportedTestSpecVersionError`). `@archmax-ai/harness/sandbox` carries the
|
|
1408
|
-
code-state contract (`defineCodeState`, `CodeStateInput`, `CodeStateArgs`,
|
|
1409
|
-
`CodeStateFn`, `SandboxTools`, `SessionHistory`, `HistoryMessage`, `TrailStep`).
|
|
1410
|
-
|
|
1411
|
-
---
|
|
1412
|
-
|
|
1413
|
-
### Source of truth in code
|
|
1414
|
-
|
|
1415
|
-
| Concern | File in the SDK |
|
|
1416
|
-
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
|
1417
|
-
| Sandbox prelude (the `code` context, `defineCodeState`, the scoped `tools` view) | `src/sandbox/assets/parts/code-state.js`; `src/sandbox/prelude.ts` |
|
|
1418
|
-
| Script source preparation (`@archmax-ai/harness/*` import stripping, `export default`, foreign-import errors) | `src/sandbox/imports.ts` |
|
|
1419
|
-
| Executor, sandbox tools, PTC gateway | `src/sandbox/executor.ts`, `tools.ts`, `ptc-gateway.ts` |
|
|
1420
|
-
| Typed entry point (`defineCodeState`, `CodeStateInput`, `SessionHistory`, `HistoryMessage`, `TrailStep`) | `src/public/sandbox.ts` |
|
|
1421
|
-
| Running an automated state, its outcomes and controls | `src/workflow/automated.ts`, `automated-controls.ts` |
|
|
1422
|
-
| Code-state and grader runs; the history (`messages`, `trail`) | `src/workflow/code-state.ts`, `grader-state.ts`, `history.ts` |
|
|
1423
|
-
| Case engine (host-side) | `src/testing/` — case schema `case-schema.ts`, discovery `discovery.ts`, assertion evaluation `assertions.ts`, grader `grade.ts`, runner `runner.ts` |
|
|
1424
|
-
| Validation diagnostics (missing code-state script, foreign import, YAML case checks) | `src/validate/validate.ts` |
|