@statelyai/agent 2.0.0-alpha.38 → 2.0.0-alpha.40
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +59 -0
- package/dist/{act-CrX0JwWY.cjs → act-B0CVbmo4.cjs} +1 -1
- package/dist/{act-B9hLQls6.mjs → act-Bsz-hUfD.mjs} +1 -1
- package/dist/ai-sdk.cjs +4 -4
- package/dist/ai-sdk.d.cts +2 -3
- package/dist/ai-sdk.d.mts +2 -3
- package/dist/ai-sdk.mjs +4 -4
- package/dist/{decision-m5hGmHpF.mjs → decision-B41YBx5U.mjs} +1 -1
- package/dist/{decision-CMeuNZ1g.cjs → decision-D8bcIGXQ.cjs} +1 -1
- package/dist/{decision-object-DBcLWVvv.d.cts → decision-object-BB8XV5Ec.d.cts} +1 -1
- package/dist/{decision-object-B5zGSPFp.d.mts → decision-object-CH_sE694.d.mts} +1 -1
- package/dist/effects-B9IvUPVh.mjs +23 -0
- package/dist/effects-DrWcUGgc.cjs +34 -0
- package/dist/evals.cjs +810 -0
- package/dist/evals.d.cts +595 -0
- package/dist/evals.d.mts +595 -0
- package/dist/evals.mjs +801 -0
- package/dist/{event-log-DkVlk3Rh.mjs → event-log-CEq_bVXQ.mjs} +8 -4
- package/dist/{event-log-Dkc-yIYu.cjs → event-log-dSOnXHuG.cjs} +7 -15
- package/dist/{event-log-store-Brx9tOHt.mjs → event-log-store-DLAfNM34.mjs} +1 -1
- package/dist/{event-log-store-BBS8so8a.d.mts → event-log-store-DYun526m.d.mts} +15 -14
- package/dist/{event-log-store-Cl-YWDnq.cjs → event-log-store-Dey9ufA4.cjs} +1 -1
- package/dist/{event-log-store-bc4ICGcS.d.cts → event-log-store-aFXK6yZX.d.cts} +15 -14
- package/dist/index.cjs +142 -97
- package/dist/index.d.cts +97 -141
- package/dist/index.d.mts +97 -141
- package/dist/index.mjs +123 -74
- package/dist/internal-BzlpzSux.d.cts +14 -0
- package/dist/internal-BzlpzSux.d.mts +14 -0
- package/dist/log.cjs +2 -3
- package/dist/log.d.cts +2 -2
- package/dist/log.d.mts +2 -2
- package/dist/log.mjs +3 -3
- package/dist/machine-input-0YTPcnCm.mjs +27 -0
- package/dist/machine-input-DU3owLUr.cjs +32 -0
- package/dist/machines.cjs +1 -719
- package/dist/machines.d.cts +2 -424
- package/dist/machines.d.mts +2 -424
- package/dist/machines.mjs +2 -709
- package/dist/{model-source-kjGwqAV8.mjs → model-source-BrsLDvHL.mjs} +1 -1
- package/dist/{model-source-B4WQ8Idp.cjs → model-source-YpxjmWuX.cjs} +1 -1
- package/dist/openai.cjs +3 -3
- package/dist/openai.d.cts +2 -3
- package/dist/openai.d.mts +2 -3
- package/dist/openai.mjs +3 -3
- package/dist/otel.d.cts +1 -1
- package/dist/otel.d.mts +1 -1
- package/dist/policy.cjs +77 -0
- package/dist/policy.d.cts +143 -0
- package/dist/policy.d.mts +143 -0
- package/dist/policy.mjs +75 -0
- package/dist/{run-agent-DlPzBWj-.d.mts → run-agent-BO9RSb7i.d.cts} +33 -40
- package/dist/{run-agent-30mH2moE.d.cts → run-agent-BxkeuAYR.d.mts} +33 -40
- package/dist/{effects-DDseYtJk.cjs → run-agent-COdR9FpM.cjs} +175 -228
- package/dist/{effects-BT_cSY0j.mjs → run-agent-D3gW9wfd.mjs} +176 -211
- package/dist/{setup-agent-BDmNr6PC.mjs → setup-agent-CLm6QVGy.mjs} +3 -3
- package/dist/{setup-agent-BaNyz6Q3.d.cts → setup-agent-Cv8Hk3mX.d.cts} +12 -11
- package/dist/{setup-agent-C1CG_nfX.d.mts → setup-agent-DgFAhaPJ.d.mts} +12 -11
- package/dist/{setup-agent-CmEToB8y.cjs → setup-agent-Dv5VpuHe.cjs} +3 -3
- package/dist/tanstack-ai.cjs +4 -4
- package/dist/tanstack-ai.d.cts +2 -3
- package/dist/tanstack-ai.d.mts +2 -3
- package/dist/tanstack-ai.mjs +4 -4
- package/dist/testing.cjs +122 -996
- package/dist/testing.d.cts +36 -695
- package/dist/testing.d.mts +36 -695
- package/dist/testing.mjs +112 -977
- package/dist/{text-logic-BqxalgMr.d.mts → text-logic-x17uLBwA.d.cts} +238 -2
- package/dist/{text-logic-CSf9bqpP.d.cts → text-logic-x17uLBwA.d.mts} +238 -2
- package/dist/trajectory-BFjXaooB.d.cts +96 -0
- package/dist/trajectory-BFjXaooB.d.mts +96 -0
- package/dist/trajectory-BUXj_1Ue.mjs +152 -0
- package/dist/trajectory-CoZSv4wF.cjs +157 -0
- package/dist/{usage-Bm5nvpow.d.cts → usage-B4B-TQSV.d.mts} +1 -1
- package/dist/{usage-BnikdOca.d.mts → usage-DvGcoK6x.d.cts} +1 -1
- package/dist/{utils-7lSa4H-i.d.cts → utils-B8k-YPYI.d.mts} +23 -17
- package/dist/{utils-CBbKw2fM.d.mts → utils-DAclF-kw.d.cts} +23 -17
- package/dist/validate.cjs +4 -4
- package/dist/validate.d.cts +2 -2
- package/dist/validate.d.mts +2 -2
- package/dist/validate.mjs +4 -4
- package/dist/{verify-DJHfEfm6.d.cts → verify-CIu4mqIZ.d.mts} +56 -6
- package/dist/{verify-i9Hiwc5g.d.mts → verify-CTHlmBbh.d.cts} +56 -6
- package/dist/{verify-ChyX6jq0.mjs → verify-Cacz6XSk.mjs} +170 -46
- package/dist/{verify-CwlombbC.cjs → verify-D8UDqU4l.cjs} +170 -46
- package/dist/{with-model-calls-B_WLmOSl.cjs → with-model-calls-D50Ca6qW.cjs} +2 -2
- package/dist/{with-model-calls-BkOwdroS.mjs → with-model-calls-uY-KKliA.mjs} +2 -2
- package/docs/advanced.md +5 -5
- package/docs/choosing-a-run-mode.md +7 -7
- package/docs/debugging.md +6 -8
- package/docs/decisions.md +5 -4
- package/docs/environment-and-policy.md +1 -1
- package/docs/evals.md +57 -43
- package/docs/event-log.md +13 -15
- package/docs/from-a-loop.md +3 -3
- package/docs/generate-machines.md +4 -4
- package/docs/hosts.md +25 -30
- package/docs/human-in-the-loop.md +5 -8
- package/docs/index.md +4 -4
- package/docs/judgments.md +5 -7
- package/docs/langgraph-comparison.md +6 -6
- package/docs/machines-as-data.md +7 -12
- package/docs/machines-presets.md +17 -120
- package/docs/machines.md +13 -13
- package/docs/models-and-providers.md +19 -25
- package/docs/multi-agent.md +1 -1
- package/docs/observability.md +6 -8
- package/docs/persistence.md +10 -20
- package/docs/quickstart.md +3 -3
- package/docs/statechart-policy-examples.md +1 -1
- package/docs/steps.md +5 -6
- package/docs/strangler-fig.md +105 -33
- package/docs/text-requests.md +10 -12
- package/docs/thinking-in-state-machines.md +8 -8
- package/docs/tools.md +10 -12
- package/docs/upgrading.md +1 -1
- package/docs/usage-and-budgets.md +51 -56
- package/docs/validate-and-repair.md +9 -14
- package/docs/verify.md +54 -7
- package/docs/xstate-4-and-5-apps.md +19 -28
- package/llms.txt +1 -1
- package/package.json +31 -6
- package/readme.md +32 -24
- package/dist/internal-C3Lqu538.d.cts +0 -43
- package/dist/internal-DUcdhk-i.d.mts +0 -43
- package/dist/types-DQUOOaZO.d.cts +0 -238
- package/dist/types-DQUOOaZO.d.mts +0 -238
- package/skills/add-evals/SKILL.md +0 -152
- package/skills/add-traces/SKILL.md +0 -152
- package/skills/build-machine/SKILL.md +0 -84
- package/skills/generate-machine/SKILL.md +0 -335
- package/skills/graph-engineering/SKILL.md +0 -43
- package/skills/improve-machine/SKILL.md +0 -261
- package/skills/improve-machine/reference.md +0 -101
- package/skills/loop-to-machine/SKILL.md +0 -205
- package/skills/loop-to-machine/policy.md +0 -154
- package/skills/loop-to-machine/reference.md +0 -260
|
@@ -1,152 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: add-traces
|
|
3
|
-
description: Make every run of a @statelyai/agent or XState machine leave a replayable, stamped trace — an event log thread plus trace events — so evals and self-improvement can read what happened and what produced it. Use when asked to "add tracing", "add logging", "persist runs", "stamp traces", "record what the agent did", or "set up observability" for an agent machine, or before writing evals or improving a policy from traces.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Add traces to an agent machine
|
|
7
|
-
|
|
8
|
-
Every run should leave two records, both stamped with what produced it:
|
|
9
|
-
|
|
10
|
-
| Record | What it is | Read by |
|
|
11
|
-
| ---------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------- |
|
|
12
|
-
| Event log thread | The external inputs the machine consumed, in order. The source of truth | `replay`, `getDecisionAttempts`, `getUsageFromEvents` |
|
|
13
|
-
| Trace events | Observations: transitions with `from`, decision requests with offered options | `foldTraces`, OpenTelemetry, debugging |
|
|
14
|
-
|
|
15
|
-
```
|
|
16
|
-
version the machines → pick a store → name threads → stamp → collect trace events → prove replay → hand back
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
This is step 2 of 4. Before: `loop-to-machine`. Next: `add-evals`, then `improve-machine`.
|
|
20
|
-
|
|
21
|
-
## Hard rules
|
|
22
|
-
|
|
23
|
-
- **Traces never change behavior.** Recording is an observer. No branch reads a trace. An `onTrace` or `onTransition` handler that throws is reported with `console.error` and skipped. The store is different: it is write-ahead, so a failed write stops the run with `cause: "journal"`. Ask the user whether a run should fail when its log cannot be written (the store) or carry on unrecorded (record from `onEvent` instead).
|
|
24
|
-
- **A trace must name what produced it.** A run with no machine version or policy version cannot be used to compare versions, so it is useless to the improver.
|
|
25
|
-
- **Replay is the test.** A log that does not replay to the run's final state with zero model calls is not a trace yet.
|
|
26
|
-
- **No secrets or personal data in metadata or thread ids.** Use ids, not names or emails. The log itself holds the machine's input and every event verbatim (a ticket's text, a customer's reply), and trace snapshots hold the context: in production, redact or encrypt them per your data rules, and ask the user which applies.
|
|
27
|
-
|
|
28
|
-
## 1. Version the machines
|
|
29
|
-
|
|
30
|
-
Declare `version` on every machine whose runs you record: the environment and, if there is one, the policy machine.
|
|
31
|
-
|
|
32
|
-
```ts
|
|
33
|
-
export const deskMachine = deskSetup.createMachine({ id: "refund-desk", version: "1", ... });
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
Also name the **decider version**: whatever produces the model's answers. It is the policy machine's `version` when there is one. Otherwise it is a label for the model, its settings, and its prompts (`gpt-6-luna.xhigh.p<hash of the prompts>`); this holds for a machine with only text requests too. When the prompts are private to another module (a route file), export a prompt version from that module rather than guessing; a release or commit id is the fallback. In tests it is the stand-in model's name (`careful`). Runs by different deciders must never share a label.
|
|
37
|
-
|
|
38
|
-
**No decider.** A machine with no model decisions (a scheduler, a tool runner whose model only picks tools inside one text request) still stamps a decider version: whatever decides what runs. That is the version of the policy engine or approval rules the guards read (`rules.v4`, a hash of the persisted rules file), plus the request config version when a text request picks tools. Stamp both in `metadata` (`rulesVersion`, `requestVersion`). Thread ids keep the formats below; a non-empty thread resumes with its original stamp, so a run under a changed rule set or request config never resumes an old thread: start a new one (`prod:<domain id>:<rulesVersion>:<requestVersion>`). A version bridge (`docs/persistence.md`) applies only when the machine `version` changes, with the old `events` and a `snapshot`; a changed rule set or request config alone never bridges. A changed rule set must never share a label with the old one. `getDecisionAttempts` is always empty here; record approval verdicts as events so they reach the log.
|
|
39
|
-
|
|
40
|
-
- Every log entry carries `machineVersion`. Without a declared `version` it falls back to a structural hash that skips functions, and in XState v6 the guards and targets live in transition functions. Changing a guard would not change the hash.
|
|
41
|
-
- Bump `version` on every behavior change. Adding recording is not one: it only observes. `improve-machine` bumps the policy's on promotion.
|
|
42
|
-
- A test-only variant built from the same factory (a loosened guard) gets its own `version` (`"1-loose"`), so its runs are never mistaken for the real machine's.
|
|
43
|
-
|
|
44
|
-
## 2. Pick a store
|
|
45
|
-
|
|
46
|
-
- **Tests and local runs:** `createInMemoryEventLogStore()` from `@statelyai/agent/log`, or write each thread's `result.events` to a JSON file after the run.
|
|
47
|
-
- **Production:** implement `AgentEventLogStore` against your database and check it with `assertEventLogStoreConformance`. Implement the optional `listThreads` so a split can be paged by prefix.
|
|
48
|
-
|
|
49
|
-
Pass it with a `threadId`. The store is write-ahead: no model call starts until the entries before it are durable. A runner that calls `runAgent` itself (a ported workflow's) takes these as an `options` argument it passes through; add that argument rather than copying the runner.
|
|
50
|
-
|
|
51
|
-
```ts
|
|
52
|
-
const result = await runAgent(deskMachine, { input, executors, store, threadId, metadata });
|
|
53
|
-
await result.drain(); // late usage entries, before you read the store or exit
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
**A thread that already has entries is resumed, not overwritten.** Give every new run a new thread id; reuse one only to continue that run. Passing different `input` to a thread that already has entries throws `thread-input-mismatch` instead of handing back another run's result.
|
|
57
|
-
|
|
58
|
-
For a production store, use a table of its own (`threadId`, `index`, the entry as JSON, a unique key on both), not a table that already means something else, such as a usage counter.
|
|
59
|
-
|
|
60
|
-
## 3. Name threads
|
|
61
|
-
|
|
62
|
-
A thread id is how runs are found later. Use a prefix scheme and keep it in one function:
|
|
63
|
-
|
|
64
|
-
| Runs | Thread id |
|
|
65
|
-
| ------------------------ | -------------------------------------------- |
|
|
66
|
-
| Scenario runs for evals | `<split>:<deciderVersion>:<scenarioId>:r<k>`, `k` = the repeat index |
|
|
67
|
-
| Production | `prod:<domain id>` (ticket id, session id) |
|
|
68
|
-
| Production, no decider | `prod:<domain id>:<rulesVersion>:<requestVersion>` |
|
|
69
|
-
|
|
70
|
-
With no decider, `<deciderVersion>` in a scenario thread id is `<rulesVersion>.<requestVersion>`.
|
|
71
|
-
|
|
72
|
-
`store.listThreads?.({ prefix: "train:careful:" })` then pages through one split of one decider. `listThreads` is optional on a custom store.
|
|
73
|
-
|
|
74
|
-
## 4. Stamp each run
|
|
75
|
-
|
|
76
|
-
`metadata` lands on the log's first (`@agent.init`) entry. Put there everything needed to tell two runs' producers apart:
|
|
77
|
-
|
|
78
|
-
```ts
|
|
79
|
-
const metadata = {
|
|
80
|
-
environmentVersion: deskMachine.version ?? "unversioned",
|
|
81
|
-
deciderVersion, // see step 1
|
|
82
|
-
models: { decider: "openai/gpt-6-luna" }, // the machine's model refs → what the host resolved them to
|
|
83
|
-
scenario: scenario.id, // eval runs only
|
|
84
|
-
split: scenario.split, // eval runs only
|
|
85
|
-
};
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
Read it back with `entries[0].metadata`. It also holds `executionId` (and `migratedFrom` after a version bridge), which the library adds and reserves, so compare with `toMatchObject`, not `toEqual`. A resumed run keeps the stamp of the run that started the log. Type the stamp as `Record<string, JsonValue>` (`JsonValue` from `@statelyai/agent/log`); an interface without an index signature does not fit.
|
|
89
|
-
|
|
90
|
-
## 5. Collect trace events
|
|
91
|
-
|
|
92
|
-
`onTrace` sees every trace event. Serialize and keep them per thread:
|
|
93
|
-
|
|
94
|
-
```ts
|
|
95
|
-
import { serializeTraceEvent, type JsonSerializableTraceEvent } from "@statelyai/agent";
|
|
96
|
-
|
|
97
|
-
const trace: JsonSerializableTraceEvent[] = [];
|
|
98
|
-
const result = await runAgent(deskMachine, { ..., onTrace: (event) => trace.push(serializeTraceEvent(event)) });
|
|
99
|
-
await result.drain();
|
|
100
|
-
|
|
101
|
-
// For local batches: one file per thread. Encoding keeps colons and path separators out of the name.
|
|
102
|
-
const file = encodeURIComponent(threadId);
|
|
103
|
-
await mkdir("runs", { recursive: true });
|
|
104
|
-
await writeFile(`runs/${file}.events.json`, JSON.stringify(result.events));
|
|
105
|
-
await writeFile(`runs/${file}.trace.json`, JSON.stringify(trace));
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
Decide with the user whether `runs/` is committed or gitignored.
|
|
109
|
-
|
|
110
|
-
- In production, send them to OpenTelemetry: `createOtelTraceHandler({ tracer })` from `@statelyai/agent/otel` is an `onTrace` handler. It needs the optional peer `@opentelemetry/api`; without OpenTelemetry in the app, write the serialized events next to the log instead.
|
|
111
|
-
- **A policy run inside a decider** is a separate run. Pass it the same `onTrace` (and a `store`) if you want per-state stats for the policy too. `createPolicyDecider` gives each decision's policy run its own thread, `policy:<callKey>:<SHA-256 of its input>`; never give it one fixed `threadId`, or the second decision resumes the first.
|
|
112
|
-
- **A plain XState machine** (built with `createMachine` or `setup`, decided with `decideNext`): drive it with `createAgentRuntime` and `runToQuiescence`, which accept plain machines, and pass each `decideNext` result as `{ event, attempts }` so refused choices reach the log. Push `{ type: "request.start", request: result.request }` onto the trace before sending the event, since a host-driven decision emits no trace itself.
|
|
113
|
-
- **A plain `createActor` run you cannot change:** attach `recordLog(actor)` from `@statelyai/agent/log` before `actor.start()`, and pass `inspect: traceTransitions((event) => trace.push(serializeTraceEvent(event)))` to `createActor`. `recordLog` cannot record decision attempts.
|
|
114
|
-
|
|
115
|
-
## 6. Prove it
|
|
116
|
-
|
|
117
|
-
Write these tests, with the stand-in model from `loop-to-machine`:
|
|
118
|
-
|
|
119
|
-
```ts
|
|
120
|
-
import { getDecisionAttempts, replay } from "@statelyai/agent/log";
|
|
121
|
-
import { foldTraces } from "@statelyai/agent/testing";
|
|
122
|
-
|
|
123
|
-
test("a recorded run replays to its final state with no model calls", async () => {
|
|
124
|
-
const result = await runAgent(deskMachine, { input, executors, store, threadId: "test:careful:small-clean:r1", metadata });
|
|
125
|
-
await result.drain();
|
|
126
|
-
const entries = await store.read("test:careful:small-clean:r1");
|
|
127
|
-
expect(entries[0]?.event.type).toBe("@agent.init"); // a log was written at all; a run with no outside input has only this entry
|
|
128
|
-
const { snapshot } = replay(deskMachine, entries, { verify: "strict" }); // a pure fold: nothing runs
|
|
129
|
-
// Compare more than the state value: context and output catch a field the log does not keep.
|
|
130
|
-
expect({ status: snapshot.status, value: snapshot.value, context: snapshot.context, output: snapshot.output }).toEqual({
|
|
131
|
-
status: result.snapshot.status,
|
|
132
|
-
value: result.snapshot.value,
|
|
133
|
-
context: result.snapshot.context,
|
|
134
|
-
output: result.snapshot.output,
|
|
135
|
-
});
|
|
136
|
-
expect(entries[0]!.metadata).toMatchObject({ environmentVersion: "1" });
|
|
137
|
-
});
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
`verify: "strict"` checks that every entry carries a hash and that each matches the fold. It does not show the log is complete (a dropped last entry leaves the rest intact), nor that the fold matches the live run, which is why the test compares the snapshots too. A journaled `Error` replays as an `Error` with its `name`, `message`, and a string or number `code`; any other property of the thrown value is not kept. A thrown plain object replays as a plain object.
|
|
141
|
-
|
|
142
|
-
- `foldTraces(traces, { failed })` over a batch returns a row for every state a run rested in. A `choice` state the run passes through in the same step gets no row. `failed` judges one whole trace; start with "the last event is not a `run.end` with status `done`, or the run ended in a failure or sentinel state", and ask the user whether a wrong outcome counts too. A serialized `run.end` is `{ type: "run.end", status, snapshot: { value, context, … } }`: `failed: (t) => { const end = t.at(-1); return end?.type !== "run.end" || end.status !== "done" || ["failed", "invariantBroken"].includes(String(end.snapshot?.value)); }`. `withheld` is above 0 only for a closed choice (`payloads`), where the guards refused an option.
|
|
143
|
-
- `getDecisionAttempts(entries)` lists the refused choices of a run; for a machine with no decisions (`agent.decide` or `decideNext`) it is always empty, so skip this test. A stand-in model that picks from the offered options can never propose an illegal one, so write a raw decide executor for this test that proposes a refused event first: `decide: async (request) => ({ event: request.attempts.length ? request.options![0]!.event : { type: "REFUND", amount: 1000 } })`.
|
|
144
|
-
- Running a scenario with and without recording leaves the same world, the same model calls, and the same final state, and the recorded run's log is not empty.
|
|
145
|
-
|
|
146
|
-
## 7. Hand back
|
|
147
|
-
|
|
148
|
-
1. Where runs are stored and how thread ids are built.
|
|
149
|
-
2. The stamp fields.
|
|
150
|
-
3. Where trace events go.
|
|
151
|
-
4. The replay test passing.
|
|
152
|
-
5. Next step: `add-evals`.
|
|
@@ -1,84 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: build-machine
|
|
3
|
-
description: Build a new @statelyai/agent machine in TypeScript from a product spec and prove its invariants. Use when asked to build an agent from scratch whose rules need thresholds, counters, caps, spawned fan-out, timers, or any-time events (cancel, takeover, interrupt) — anything the JSON config's truthy-only guards cannot express — and when asked to prove a spec's invariants with canReach or explorePaths.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Build an agent machine
|
|
7
|
-
|
|
8
|
-
The spec's invariants become structure, then proofs. A rule that lives in a prompt is not an invariant.
|
|
9
|
-
|
|
10
|
-
```
|
|
11
|
-
invariants → machine → scenario tests → proofs → crash/resume → inspector → hand back
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
Use `generate-machine` instead when the whole spec fits a JSON `AgentWorkflowConfig`: truthy guards, no counters, no spawn.
|
|
15
|
-
|
|
16
|
-
## 1. List the invariants
|
|
17
|
-
|
|
18
|
-
Write each spec rule as one sentence a search can falsify: "`book_table` never runs without a `CONFIRM` for its key", "at most 8 model turns". Ask the user for thresholds and caps you cannot read from the spec; decide the mechanics yourself.
|
|
19
|
-
|
|
20
|
-
## 2. Make each one structural
|
|
21
|
-
|
|
22
|
-
| Rule shape | Structure |
|
|
23
|
-
| --- | --- |
|
|
24
|
-
| Needs approval first | A waiting state the effect state is entered only from, on the approval event |
|
|
25
|
-
| A cap (turns, retries, rejections) | A counter in `context`, checked inline in the transition that would exceed it |
|
|
26
|
-
| A threshold (confidence ≥ 0.8) | An inline guard in the transition function |
|
|
27
|
-
| Any time (cancel, takeover, interrupt) | A root-level `on` handler targeting a final state, or a state-scoped `on` in every state that must accept it |
|
|
28
|
-
| Wait for all parallel work | Children (`enq.spawn` or parallel invokes) plus a join that counts settled results |
|
|
29
|
-
| Timeout / SLA | An `after` transition |
|
|
30
|
-
| Token budget | A root `on: { '@agent.usage': … }` fold plus a guard |
|
|
31
|
-
|
|
32
|
-
- Write guards inline and keep targets in the transition function, so `describeMachine` can read them.
|
|
33
|
-
- Pass `events` to `setupAgent`, even `events: {}`, so built-in events (`@agent.usage`, `xstate.error.actor`) are typed.
|
|
34
|
-
- A state reachable only when a rule breaks (a sentinel) makes the proof one line. A `canReach` predicate works without one.
|
|
35
|
-
|
|
36
|
-
## 3. Scenario tests
|
|
37
|
-
|
|
38
|
-
At least one test per rule: the happy path, then each failure mode. Fake models with scripted executors or `MockLanguageModelV3` from `ai/test`. Test runtime behavior a search cannot see: an aborted signal reaching the executor, a late model answer after a cancel.
|
|
39
|
-
|
|
40
|
-
## 4. Prove
|
|
41
|
-
|
|
42
|
-
```ts
|
|
43
|
-
import { canReach, explorePaths } from "@statelyai/agent/testing";
|
|
44
|
-
|
|
45
|
-
const result = await canReach(machine, (s) => s.matches("writing") && s.context.succeeded < 2, {
|
|
46
|
-
outputs: { plan: [twoQueries, fiveQueries], search: [hits, []] },
|
|
47
|
-
errors: { search: [new Error("down")] },
|
|
48
|
-
interleave: "pending",
|
|
49
|
-
});
|
|
50
|
-
expect(result).toMatchObject({ reachable: false, complete: true });
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
- **Fork everything.** List at least two outputs per src, plus `errors`. A src with one output resolves without a fork and never appears in a path.
|
|
54
|
-
- **Any-time rules need `interleave: "pending"`**, so events arrive while work is in flight.
|
|
55
|
-
- **Rules across children** need `children: "coupled"`.
|
|
56
|
-
- **Timers fire** as events, in any order (`timers: "ignore"` turns that off and marks the result incomplete). **Usage** is offered only for root text requests listed in `usage: { src: [{ totalTokens: 500 }] }`; usage inside child machines is test-only.
|
|
57
|
-
- **A machine that reads `output.messages`** needs `textOutput(result, { messages })` from `@statelyai/agent/testing` as the canned output.
|
|
58
|
-
- **`complete: true` or it is not a proof.** Read `unexplored` and `unexpandedEvents` when it is false. Add `payloads` for events with fields, and `dedupeBy` over the context fields guards read when the search explodes.
|
|
59
|
-
- **Long-lived machines never complete.** Every path ends at `maxDepth`. Report a bounded proof: `reachable: false` with only max-depth notes, at a stated depth.
|
|
60
|
-
- **Mutation check.** Weaken each guard in a throwaway copy and confirm the proof finds a witness. A proof that cannot fail proves nothing.
|
|
61
|
-
- Record each rule you could not prove and why. It stays covered by tests only.
|
|
62
|
-
|
|
63
|
-
## 5. Crash and resume
|
|
64
|
-
|
|
65
|
-
Call `checkCrashRecovery(machine, { input, events, executors })` from `@statelyai/agent/testing` for each scenario, and assert `report.failures` is `[]`. It kills the run after every log append and while every effect is in flight, resumes each from the log, and checks the resume against an uninterrupted run. See `docs/persistence.md#crash-testing`.
|
|
66
|
-
|
|
67
|
-
- Executors and fake tools must answer from the request, not call order. A fake that keeps state passes `reset`.
|
|
68
|
-
- A failure names the crash point and the problem (`reran`, `skipped`, `outcome`, `stuck`, …). Fix the machine or the effect, never the check.
|
|
69
|
-
|
|
70
|
-
- A completed model call never runs again. An in-flight one re-runs with the same `info.callKey`.
|
|
71
|
-
- A plain actor (`createAsyncLogic`) in flight at the crash re-runs. It gets no call key, so derive an idempotency key from the domain (the ticket id, the booking key) and dedupe in the effect.
|
|
72
|
-
- Assert the side-effect ledger, not just the final state: each effect once, same output as an uninterrupted run.
|
|
73
|
-
|
|
74
|
-
## 6. Inspect
|
|
75
|
-
|
|
76
|
-
Run one scenario with `inspect: inspector.inspect` from `@statelyai/sdk`. Each actor registers once and the root's states arrive in order, across `onIdle` passes too. The SDK's `node-relay` needs `ws` installed.
|
|
77
|
-
|
|
78
|
-
## 7. Hand back
|
|
79
|
-
|
|
80
|
-
- The invariant list, each with proved / bounded / test-only and the method.
|
|
81
|
-
- The commands that rerun tests, proofs, and the crash test.
|
|
82
|
-
- What you could not prove and why.
|
|
83
|
-
|
|
84
|
-
Then route to `add-traces` (step 2 of `graph-engineering`).
|
|
@@ -1,335 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: generate-machine
|
|
3
|
-
description: Author a @statelyai/agent workflow as JSON and check it before it runs. Use when asked to generate or author an agent machine, create a workflow config, write an AgentWorkflowConfig, or produce an LLM-authored state machine for an agent — and when a generated config fails verifyAgentConfig, Ajv validation, or agent lint and needs repair.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Generate an agent machine
|
|
7
|
-
|
|
8
|
-
An agent machine is data. You author a JSON `AgentWorkflowConfig`, then run it through `verifyAgentConfig`, whose gates all work with no API key: validate → vocabulary → lower → lint → explore. Do not hand back a config that has not passed every gate.
|
|
9
|
-
|
|
10
|
-
```
|
|
11
|
-
split environment/policy → author (description = goal) → verifyAgentConfig → repair (≤ 3) → persist traces → hand back
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
To improve an existing policy from its traces, use the `improve-machine` skill instead.
|
|
15
|
-
|
|
16
|
-
## 0. Split environment and policy
|
|
17
|
-
|
|
18
|
-
Decide what is fixed before you write any JSON.
|
|
19
|
-
|
|
20
|
-
- **Environment** (TypeScript, fixed): the rules of the world and the business invariants, as states and guards. People author it and verify it with `canReach` / `explorePaths` from `@statelyai/agent/testing`. You never generate it, and no generated config may change it.
|
|
21
|
-
- **Policy** (JSON, generated): how the agent works toward the goal: which step next, which tool, when to ask a person, cheap judgment vs model call. This is the `AgentWorkflowConfig` you author.
|
|
22
|
-
- The policy plugs into the environment at its decision points and outputs one event; the environment's guards decide whether to accept it.
|
|
23
|
-
- No split needed for a single judgment inside a fixed workflow, or for a workflow where every step is dictated by rules. Author one machine.
|
|
24
|
-
|
|
25
|
-
If the request asks you to generate a business rule (an approval threshold, a payout limit), stop: that belongs in the environment, in TypeScript, reviewed by a person. Say so.
|
|
26
|
-
|
|
27
|
-
Before editing an existing machine, read it as text:
|
|
28
|
-
|
|
29
|
-
```ts
|
|
30
|
-
import { describeMachine } from "@statelyai/agent";
|
|
31
|
-
const text = describeMachine(machine); // "Machine <id>", then "Goal: …", states, transitions, events
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
Pass `{ snapshot }` to mark the active states with `*`. Concept page: [Environment and policy](../../docs/environment-and-policy.md).
|
|
35
|
-
|
|
36
|
-
## 1. Read the schema
|
|
37
|
-
|
|
38
|
-
The config shape is fixed by a shipped JSON Schema. Read it first; it is the contract.
|
|
39
|
-
|
|
40
|
-
- Consuming the package: `node_modules/@statelyai/agent/schemas/agent-workflow.json` (importable as `@statelyai/agent/agent-workflow.json`)
|
|
41
|
-
- In the `statelyai/agent` repo: `schemas/agent-workflow.json`
|
|
42
|
-
|
|
43
|
-
## 2. Author the config
|
|
44
|
-
|
|
45
|
-
Start with the root `description`: it is the machine's **goal**, the overall objective plus measurable success criteria. `describeMachine` renders it first as `Goal: …`, and an improver later climbs toward it. Write criteria a scorer can compute from outcomes (terminal states, effects in the world, cost), not "be helpful":
|
|
46
|
-
|
|
47
|
-
```json
|
|
48
|
-
"description": "Resolve each support ticket in one session. Success: escalations only for tickets that need a human; every reply approved before sending; at most 2 model calls per ticket."
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
Rules the runtime enforces. Each one maps to a build error or a lint diagnostic:
|
|
52
|
-
|
|
53
|
-
- Every value is a JSON literal or a whole-string `"{{ }}"` expression reading a dot path on `input`, `context`, or `event`. No JavaScript.
|
|
54
|
-
- Guard expressions are truthy-only: no comparisons, operators, or negation.
|
|
55
|
-
- A model call is either a named `requests` entry invoked with `src: "<requestName>"`, or an inline invoke with `src: "agent.decide"` plus an `allowedEvents` list. A `requests` entry's templates read only its own `input` (`"{{ input.ticket }}"`); the invoke's `input` maps context into it.
|
|
56
|
-
- A state invoking `agent.decide` MUST handle every allowed event in its `on`.
|
|
57
|
-
- An `agent.decide` invoke has no `onDone` (a decision produces no output). Use `onError` for the retries-exhausted path.
|
|
58
|
-
- A request invoke resolves with `{ result, messages }`, so `onDone.assign` reads its result from `"{{ event.output.result.<field> }}"` (`"{{ event.output.result }}"` for a plain-text request). A non-request `actors` invoke reads `"{{ event.output.<field> }}"` directly.
|
|
59
|
-
- Every path must reach a `"type": "final"` state, and each final state needs an `output` when `schemas.output` is declared.
|
|
60
|
-
- Do not invent guard or action names. Only `"{{ }}"` guards, `assign`, and `emit` exist unless the host tells you which named guards/actions it implements.
|
|
61
|
-
- Model refs are strings (`"openai/gpt-6-luna"`); the host resolves them.
|
|
62
|
-
- To make a decision a closed choice, add `payloads` to the `agent.decide` input: one payload list per event type, without `type` (`"payloads": { "REFUND": [{ "amount": 20 }, { "amount": 50 }], "ESCALATE": [] }`). The model then picks one listed event instead of writing a payload. A type with no key is offered as `{ type }`; `[]` withholds it.
|
|
63
|
-
- An idle human step can carry `meta.interaction` for the host to render with `getInteraction`: a `label`, `events` (one button per event type), `choices` (one button per full event, `{ "label": "Refund $20", "event": { "type": "REFUND", "amount": 20 } }`, or a context path string holding such a list), and `textEvent` for free text. A bare `"textEvent": "REJECT"` puts the text in a `text` field; `{ "type": "REJECT", "field": "reason" }` names the field. `meta` is not template-evaluated.
|
|
64
|
-
|
|
65
|
-
Reference config — decision, text request, idle human step, one final state:
|
|
66
|
-
|
|
67
|
-
```json
|
|
68
|
-
{
|
|
69
|
-
"id": "support-ticket",
|
|
70
|
-
"description": "Resolve each support ticket in one session. Success: escalations only for tickets that need a human; every reply approved before sending; at most 2 model calls per ticket.",
|
|
71
|
-
"schemas": {
|
|
72
|
-
"input": {
|
|
73
|
-
"type": "object",
|
|
74
|
-
"properties": { "ticket": { "type": "string" } },
|
|
75
|
-
"required": ["ticket"]
|
|
76
|
-
},
|
|
77
|
-
"context": {
|
|
78
|
-
"type": "object",
|
|
79
|
-
"properties": {
|
|
80
|
-
"ticket": { "type": "string" },
|
|
81
|
-
"reply": { "type": "string" },
|
|
82
|
-
"resolution": { "type": "string" }
|
|
83
|
-
},
|
|
84
|
-
"required": ["ticket"]
|
|
85
|
-
},
|
|
86
|
-
"events": {
|
|
87
|
-
"ESCALATE": {
|
|
88
|
-
"type": "object",
|
|
89
|
-
"properties": { "reason": { "type": "string" } },
|
|
90
|
-
"required": ["reason"]
|
|
91
|
-
},
|
|
92
|
-
"REPLY": { "type": "object", "properties": {} },
|
|
93
|
-
"APPROVE": { "type": "object", "properties": {} },
|
|
94
|
-
"REJECT": { "type": "object", "properties": {} }
|
|
95
|
-
},
|
|
96
|
-
"output": {
|
|
97
|
-
"type": "object",
|
|
98
|
-
"properties": { "resolution": { "type": "string" }, "reply": { "type": "string" } },
|
|
99
|
-
"required": ["resolution"]
|
|
100
|
-
}
|
|
101
|
-
},
|
|
102
|
-
"context": { "ticket": "{{ input.ticket }}" },
|
|
103
|
-
"requests": {
|
|
104
|
-
"draftReply": {
|
|
105
|
-
"model": "openai/gpt-6-luna",
|
|
106
|
-
"system": "Draft a short, courteous support reply.",
|
|
107
|
-
"prompt": "{{ input.ticket }}",
|
|
108
|
-
"input": {
|
|
109
|
-
"type": "object",
|
|
110
|
-
"properties": { "ticket": { "type": "string" } },
|
|
111
|
-
"required": ["ticket"]
|
|
112
|
-
},
|
|
113
|
-
"output": {
|
|
114
|
-
"type": "object",
|
|
115
|
-
"properties": { "reply": { "type": "string" } },
|
|
116
|
-
"required": ["reply"]
|
|
117
|
-
}
|
|
118
|
-
}
|
|
119
|
-
},
|
|
120
|
-
"initial": "triaging",
|
|
121
|
-
"states": {
|
|
122
|
-
"triaging": {
|
|
123
|
-
"invoke": {
|
|
124
|
-
"id": "triageDecision",
|
|
125
|
-
"src": "agent.decide",
|
|
126
|
-
"input": {
|
|
127
|
-
"model": "openai/gpt-6-luna",
|
|
128
|
-
"system": "Decide whether this ticket needs escalation or a drafted reply.",
|
|
129
|
-
"prompt": "{{ context.ticket }}",
|
|
130
|
-
"allowedEvents": ["ESCALATE", "REPLY"],
|
|
131
|
-
"maxRetries": 2
|
|
132
|
-
},
|
|
133
|
-
"onError": { "target": "resolved", "assign": { "resolution": "escalated" } }
|
|
134
|
-
},
|
|
135
|
-
"on": {
|
|
136
|
-
"ESCALATE": { "target": "resolved", "assign": { "resolution": "escalated" } },
|
|
137
|
-
"REPLY": { "target": "drafting" }
|
|
138
|
-
}
|
|
139
|
-
},
|
|
140
|
-
"drafting": {
|
|
141
|
-
"invoke": {
|
|
142
|
-
"id": "draft",
|
|
143
|
-
"src": "draftReply",
|
|
144
|
-
"input": { "ticket": "{{ context.ticket }}" },
|
|
145
|
-
"onDone": {
|
|
146
|
-
"target": "awaitingApproval",
|
|
147
|
-
"assign": { "reply": "{{ event.output.result.reply }}" }
|
|
148
|
-
}
|
|
149
|
-
}
|
|
150
|
-
},
|
|
151
|
-
"awaitingApproval": {
|
|
152
|
-
"description": "Idle: nothing to do until a human approves or rejects the draft.",
|
|
153
|
-
"on": {
|
|
154
|
-
"APPROVE": { "target": "resolved", "assign": { "resolution": "replied" } },
|
|
155
|
-
"REJECT": { "target": "resolved", "assign": { "resolution": "escalated" } }
|
|
156
|
-
}
|
|
157
|
-
},
|
|
158
|
-
"resolved": {
|
|
159
|
-
"type": "final",
|
|
160
|
-
"output": { "resolution": "{{ context.resolution }}", "reply": "{{ context.reply }}" }
|
|
161
|
-
}
|
|
162
|
-
}
|
|
163
|
-
}
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
## 3. Verify with `verifyAgentConfig`
|
|
167
|
-
|
|
168
|
-
`verifyAgentConfig(config, options)` from `@statelyai/agent/validate` runs every pre-run gate and stops at the first failure. It never throws on a bad config; each gate is a record `{ name, ok, detail? }`. It needs `ajv` installed (an optional peer of the package).
|
|
169
|
-
|
|
170
|
-
| Gate | Checks with | Fails on |
|
|
171
|
-
| ------------ | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
172
|
-
| `validate` | `validateAgentConfig` (shipped schema) | malformed config shape |
|
|
173
|
-
| `vocabulary` | `allow` lists | an invoke src, named guard, named action, `agent.decide` event, or model ref not listed |
|
|
174
|
-
| `lower` | `setupAgent.fromConfig` | unresolved named guard/action, `onDone` on an `agent.decide` invoke |
|
|
175
|
-
| `lint` | `lintAgentMachine` | error findings (warnings too with `lint: { warnings: true }`) |
|
|
176
|
-
| `explore` | `explorePaths`, once per input | a path ending outside `explore.terminals` (default `['done']`), a loop past `maxDepth`, a `maxPaths`/`maxNodes` cap, untried payloads (`unexpandedEvents`) |
|
|
177
|
-
|
|
178
|
-
`compileSchema` turns the JSON Schemas _inside_ the config into Standard Schema validators for `fromConfig`. The library bundles no JSON Schema engine:
|
|
179
|
-
|
|
180
|
-
```ts
|
|
181
|
-
import Ajv from "ajv";
|
|
182
|
-
import type { SchemaCompiler, StandardSchemaV1 } from "@statelyai/agent";
|
|
183
|
-
|
|
184
|
-
const ajv = new Ajv({ strict: false });
|
|
185
|
-
|
|
186
|
-
export const compileSchema: SchemaCompiler = (jsonSchema, name): StandardSchemaV1 => {
|
|
187
|
-
const validate = ajv.compile(jsonSchema);
|
|
188
|
-
return {
|
|
189
|
-
"~standard": {
|
|
190
|
-
version: 1,
|
|
191
|
-
vendor: "ajv",
|
|
192
|
-
validate: (value) =>
|
|
193
|
-
validate(value)
|
|
194
|
-
? { value }
|
|
195
|
-
: {
|
|
196
|
-
issues: (validate.errors ?? []).map((e) => ({
|
|
197
|
-
message: `${name}${e.instancePath} ${e.message}`,
|
|
198
|
-
})),
|
|
199
|
-
},
|
|
200
|
-
// Expose the source JSON Schema so lint and describeMachine can read the shape.
|
|
201
|
-
jsonSchema: { input: () => jsonSchema },
|
|
202
|
-
},
|
|
203
|
-
};
|
|
204
|
-
};
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
Run the gates. List the host's real names in `allow`; an omitted key is unrestricted:
|
|
208
|
-
|
|
209
|
-
```ts
|
|
210
|
-
import { verifyAgentConfig } from "@statelyai/agent/validate";
|
|
211
|
-
|
|
212
|
-
const result = await verifyAgentConfig(candidate, {
|
|
213
|
-
compileSchema,
|
|
214
|
-
allow: {
|
|
215
|
-
actors: [],
|
|
216
|
-
requests: ["draftReply"],
|
|
217
|
-
guards: [],
|
|
218
|
-
actions: [],
|
|
219
|
-
events: ["ESCALATE", "REPLY", "APPROVE", "REJECT"],
|
|
220
|
-
models: ["openai/gpt-6-luna"],
|
|
221
|
-
},
|
|
222
|
-
explore: {
|
|
223
|
-
inputs: [{ ticket: "Export downloads a 0-byte CSV on Safari." }],
|
|
224
|
-
outputs: { draftReply: [{ reply: "" }] },
|
|
225
|
-
// ESCALATE's `reason` is a free-form string: give exploration one to try.
|
|
226
|
-
payloads: () => ({ ESCALATE: [{ reason: "needs a human" }] }),
|
|
227
|
-
},
|
|
228
|
-
});
|
|
229
|
-
// result.ok, result.gates, result.config (after validate), result.machine (after lower)
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
- Exploration forks past idle waits on every event the waiting state accepts, so a human step still ends `done`. Add `terminals: ["done", "idle"]` only when a path may legitimately stop at a wait with nothing to accept.
|
|
233
|
-
- `guards` / `actions`: named implementations, passed to `fromConfig`. `actors`: bound with `machine.provide({ actors })` after lowering.
|
|
234
|
-
- `explore` keys other than `inputs` and `terminals` pass through to `explorePaths`: `outputs` (output lists per request or actor src: one element is a canned output, two or more fork one branch each), `errors` (fork a failing branch per src), `payloads` (fork one branch per payload at decisions and idle waits), `dedupeBy` (explore each snapshot key once; the key must cover every field a guard reads), `maxDepth` (default 8), `maxPaths` (default 200), `maxNodes` (default 100,000).
|
|
235
|
-
- Stub each request's output from its declared `output` schema (`""` for string, `0` for number, `[]` for array, recurse on `properties`). Add an `errors` entry for every src whose `onError` path matters.
|
|
236
|
-
- Pick `explore.inputs` that reach every branch the goal cares about. Exploration covers only the inputs you give it.
|
|
237
|
-
- A passing `lint` gate carries its warnings in `detail`. Report them.
|
|
238
|
-
|
|
239
|
-
## 4. Repair loop
|
|
240
|
-
|
|
241
|
-
A failed gate's `detail` names the offending path, state, or name, one problem per line. Feed it back verbatim with the rejected config, and run the gates again:
|
|
242
|
-
|
|
243
|
-
```ts
|
|
244
|
-
let prompt = originalTask;
|
|
245
|
-
for (let attempt = 0; attempt < 3; attempt++) {
|
|
246
|
-
const candidate = await author({ system, prompt });
|
|
247
|
-
const result = await verifyAgentConfig(candidate, verifyOptions);
|
|
248
|
-
if (result.ok) return result;
|
|
249
|
-
const failed = result.gates.find((gate) => !gate.ok)!;
|
|
250
|
-
prompt = `${originalTask}\n\nYour previous config:\n${JSON.stringify(candidate)}\n\nGate '${failed.name}' failed:\n${failed.detail}\nReturn a corrected config.`;
|
|
251
|
-
}
|
|
252
|
-
```
|
|
253
|
-
|
|
254
|
-
Keep the cap at 3. A config that fails three repairs is usually asking for something the data form cannot express: author it in TypeScript with `setupAgent` instead, or report that the vocabulary is missing something.
|
|
255
|
-
|
|
256
|
-
## 5. Manual fallback
|
|
257
|
-
|
|
258
|
-
Without `verifyAgentConfig` (older package, or a gate you need to run alone), run the same gates by hand:
|
|
259
|
-
|
|
260
|
-
```ts
|
|
261
|
-
import { setupAgent, type AgentWorkflowConfig } from "@statelyai/agent";
|
|
262
|
-
import { validateAgentConfig } from "@statelyai/agent/validate";
|
|
263
|
-
import { explorePaths, lintAgentMachine, simulateAgent } from "@statelyai/agent/testing";
|
|
264
|
-
|
|
265
|
-
const validation = validateAgentConfig(candidate); // { valid, errors: [{ path, message }] }, path is a JSON Pointer
|
|
266
|
-
if (!validation.valid) {
|
|
267
|
-
// Stop here: feed the errors back as the repair prompt. Never lower an invalid config.
|
|
268
|
-
throw new Error(validation.errors.map((e) => `${e.path} ${e.message}`).join("\n"));
|
|
269
|
-
}
|
|
270
|
-
const { machine } = setupAgent.fromConfig(candidate as AgentWorkflowConfig, { compileSchema }); // throws on unresolved names
|
|
271
|
-
lintAgentMachine(machine, { throw: true }); // throws AgentLintError on error findings
|
|
272
|
-
const dryRun = await simulateAgent(machine, {
|
|
273
|
-
input: { ticket: "" },
|
|
274
|
-
script: {
|
|
275
|
-
decisions: { "agent.decide": [{ type: "REPLY" }] },
|
|
276
|
-
text: { draftReply: [{ reply: "" }] },
|
|
277
|
-
},
|
|
278
|
-
}); // dryRun.status: 'done' | 'idle' | 'exhausted' ('exhausted' = loops, reject)
|
|
279
|
-
const report = await explorePaths(machine, {
|
|
280
|
-
input: { ticket: "" },
|
|
281
|
-
outputs: { draftReply: [{ reply: "" }] },
|
|
282
|
-
payloads: () => ({ ESCALATE: [{ reason: "needs a human" }] }),
|
|
283
|
-
}); // report.unexpandedEvents lists types that still need a payloads entry
|
|
284
|
-
```
|
|
285
|
-
|
|
286
|
-
The lint check that bites most often is `decide-without-events`: an `allowedEvents` list with no matching `on` transitions. Do not disable checks.
|
|
287
|
-
|
|
288
|
-
## 6. Persist traces from day one
|
|
289
|
-
|
|
290
|
-
An improver needs data. Wire an event log store and a thread id per run before the first real run, so every run is a trace:
|
|
291
|
-
|
|
292
|
-
```ts
|
|
293
|
-
import { runAgent } from "@statelyai/agent";
|
|
294
|
-
import { createInMemoryEventLogStore } from "@statelyai/agent/log";
|
|
295
|
-
|
|
296
|
-
const store = createInMemoryEventLogStore(); // tests only; implement AgentEventLogStore on your database
|
|
297
|
-
const result = await runAgent(machine, {
|
|
298
|
-
input,
|
|
299
|
-
executors,
|
|
300
|
-
store,
|
|
301
|
-
threadId: `tickets:${ticketId}`,
|
|
302
|
-
});
|
|
303
|
-
```
|
|
304
|
-
|
|
305
|
-
- `threadId` is required with `store`. One thread per run.
|
|
306
|
-
- Give thread ids a prefix an improver can list by, such as `tickets:<ticketId>`. `store.listThreads({ prefix, limit, cursor })` pages through them. It is optional on a custom `AgentEventLogStore`, so with a store that lacks it, record each thread id with its scenario or ticket.
|
|
307
|
-
- Failed decision attempts are journaled on the entry they explain; `getDecisionAttempts(entries)` from `@statelyai/agent/log` reads them back.
|
|
308
|
-
- `replay(machine, await store.read(threadId))` from `@statelyai/agent/log` rebuilds the final state with no model calls. Pass the machine that recorded the thread: with an environment and a policy, the environment; `getUsageFromEvents(entries)` sums token usage.
|
|
309
|
-
|
|
310
|
-
## 7. Hand back
|
|
311
|
-
|
|
312
|
-
Deliver the config JSON plus how to run it:
|
|
313
|
-
|
|
314
|
-
```ts
|
|
315
|
-
import { runAgent } from "@statelyai/agent";
|
|
316
|
-
import { createAiSdkExecutors } from "@statelyai/agent/ai-sdk";
|
|
317
|
-
|
|
318
|
-
const run = await runAgent(result.machine!, {
|
|
319
|
-
input: { ticket: "Export downloads a 0-byte CSV on Safari." },
|
|
320
|
-
executors: createAiSdkExecutors({ resolveModel }),
|
|
321
|
-
store,
|
|
322
|
-
threadId,
|
|
323
|
-
});
|
|
324
|
-
```
|
|
325
|
-
|
|
326
|
-
Say which gates passed (list `result.gates`), list any lint warnings, state the goal and its success criteria, and say where traces are written. Model refs are strings, so the host supplies `resolveModel`.
|
|
327
|
-
|
|
328
|
-
## Known limits
|
|
329
|
-
|
|
330
|
-
- **Named guards and actions are host-resolved.** A config carries no functions. `guard: "isReady"` only works if the host passes an `isReady` implementation to `fromConfig`; an unresolved name is a build-time throw. Either list the host's names in the prompt or forbid named guards/actions entirely.
|
|
331
|
-
- **Expressions are dot-path templates only.** `"{{ context.a.b }}"` — no comparisons, no arithmetic, no method calls, no partial interpolation inside a larger string.
|
|
332
|
-
- **Ajv validity is not semantic validity.** A config can validate and still invoke a request that does not exist or route to a state that solves nothing. That is what the `vocabulary`, `lint`, and `explore` gates are for.
|
|
333
|
-
- **Exploration covers the inputs you give it.** A branch no `explore.inputs` entry reaches is unverified.
|
|
334
|
-
- **Model refs are checked only with `allow.models`.** Tool names stay unchecked strings that resolve at run time, in the host.
|
|
335
|
-
- **The gates check structure, not quality.** Whether a config is better than another is measured by replaying scenarios against an objective. See the `improve-machine` skill.
|
|
@@ -1,43 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: graph-engineering
|
|
3
|
-
description: Route an agent-reliability request to the right @statelyai/agent skill and keep the four steps in order — make it a state machine, add traces, add evals, self-improve. Use when asked to "make my agent reliable", "graph engineering", "turn my agent into a state machine and improve it", "where do I start with this agent", or any request that spans more than one of those steps.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Graph engineering
|
|
7
|
-
|
|
8
|
-
Four steps, in order. Each one needs the one before it.
|
|
9
|
-
|
|
10
|
-
| Step | Skill | Done when |
|
|
11
|
-
| --- | --- | --- |
|
|
12
|
-
| 1. Make it a state machine | `loop-to-machine` (existing code), `build-machine` (from scratch, TypeScript), or `generate-machine` (from scratch, when the spec fits JSON) | Rules are guards and the sentinel is unreachable (`canReach`). From existing code: the parity report has no unreviewed regressions. From scratch: every invariant is proved complete or listed as not proved, or every `verifyAgentConfig` gate passes |
|
|
13
|
-
| 2. Add traces | `add-traces` | Every run leaves a stamped log thread that replays to its final state |
|
|
14
|
-
| 3. Add evals | `add-evals` | Evals per state at the cheapest level, an objective, baseline numbers per split |
|
|
15
|
-
| 4. Self-improve (experimental) | `improve-machine` | A pull request with one policy change and its held-out scorecard, or a report that none helped |
|
|
16
|
-
|
|
17
|
-
## Find where the project is
|
|
18
|
-
|
|
19
|
-
Check in this order and start at the first step that is not done:
|
|
20
|
-
|
|
21
|
-
1. Does the code path production actually runs go through a machine (`setupAgent`, `createMachine`, an `AgentWorkflowConfig`)? Follow the route or entry point to the model call; a machine that nothing calls does not count. If the path is a loop or ad-hoc code, step 1.
|
|
22
|
-
2. Do runs persist to an event log store with a declared machine `version` and a `metadata` stamp? If not, step 2.
|
|
23
|
-
3. Is there an objective over ground truth, and evals that name the states they cover? If not, step 3.
|
|
24
|
-
4. Otherwise, step 4, which is experimental: say so when you route there. A machine with no decision point (a tool loop, a scheduler: the model picks tools inside one text request) still goes to step 4: `improve-machine`'s "No decision points" branch, which turns traces into approval-rule, cap, and request-wording proposals.
|
|
25
|
-
|
|
26
|
-
From scratch, pick `build-machine` when a rule needs a threshold, a counter, spawned fan-out, a timer, or an any-time event: the JSON config's guards are truthy-only. Pick `generate-machine` when every rule is a truthy check.
|
|
27
|
-
|
|
28
|
-
Tell the user which step you are starting at and why, in one line, then load that skill.
|
|
29
|
-
|
|
30
|
-
## Across all steps
|
|
31
|
-
|
|
32
|
-
- **XState v6, not v5.** Most training data is v5; v6 differs:
|
|
33
|
-
- A transition is a plain function: `({ context, event }) => ({ target, context })`. There is no `assign`; return the new context.
|
|
34
|
-
- Guards are inline JS in that function. Named guards are optional, not required.
|
|
35
|
-
- Targets are objects (`{ target: "b" }`), never bare strings.
|
|
36
|
-
- Actors: `createAsyncLogic`, not `fromPromise`. Done/error events are `{ type: "xstate.done.actor", actorId, output }` and `{ type: "xstate.error.actor", actorId, error }`.
|
|
37
|
-
- **Keep targets in the transition function.** `describeMachine` parses each function and lists every target it can return. A target from a helper (`=> resolveRound(context)`) or a variable reads as `→ ?`: have the helper return data and pick the target inline. Pass `{ source: true }` to read each function under its line, and `helpers: { name: fn }` for the functions they call by name.
|
|
38
|
-
- **To see what one event does, call `transition`.** It is pure and runs nothing; see [loop-to-machine](../loop-to-machine/SKILL.md#inspect-one-event).
|
|
39
|
-
- **The environment is people's code.** Rules, guards, invariants, and the sentinel change only by human review. No step rewrites them on its own.
|
|
40
|
-
- **No decision point is not a dead end.** Steps 2–4 still apply: traces stamp the request config or approval-rules version, evals are rule checks and per-state stats, and improvement proposes changes for a person to review.
|
|
41
|
-
- **Ground truth scores everything.** Outcomes and effects in the world, labeled by people. A model never grades its own runs.
|
|
42
|
-
- **Ask for labels and rules; decide the mechanics.** Expected outcomes, thresholds, the objective, and the train/held-out split are the user's. File layout, test names, and helper code are yours.
|
|
43
|
-
- **Hand back between steps.** Finish one step's hand-back before starting the next, unless the user asked for several in one go.
|