@statelyai/agent 2.0.0-alpha.39 → 2.0.0-alpha.41
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +35 -0
- package/dist/evals.cjs +1 -1
- package/dist/evals.d.cts +1 -1
- package/dist/evals.d.mts +1 -1
- package/dist/evals.mjs +1 -1
- package/dist/index.cjs +17 -10
- package/dist/index.d.cts +29 -12
- package/dist/index.d.mts +29 -12
- package/dist/index.mjs +17 -10
- package/dist/machine-input-0YTPcnCm.mjs +27 -0
- package/dist/machine-input-DU3owLUr.cjs +32 -0
- package/dist/otel.d.cts +1 -1
- package/dist/otel.d.mts +1 -1
- package/dist/policy.cjs +1 -1
- package/dist/policy.d.cts +1 -1
- package/dist/policy.d.mts +1 -1
- package/dist/policy.mjs +1 -1
- package/dist/{run-agent-Dq0oSFRm.d.cts → run-agent-BO9RSb7i.d.cts} +16 -7
- package/dist/{run-agent-DHlq87G7.d.mts → run-agent-BxkeuAYR.d.mts} +16 -7
- package/dist/{run-agent-De3xVFGw.cjs → run-agent-COdR9FpM.cjs} +26 -27
- package/dist/{run-agent-BE7DOWD6.mjs → run-agent-D3gW9wfd.mjs} +27 -28
- package/dist/testing.cjs +377 -18
- package/dist/testing.d.cts +181 -5
- package/dist/testing.d.mts +181 -5
- package/dist/testing.mjs +377 -20
- package/dist/{utils-DAclF-kw.d.cts → utils-BMxUcjay.d.cts} +57 -1
- package/dist/{utils-B8k-YPYI.d.mts → utils-BV7pxhD-.d.mts} +57 -1
- package/dist/validate.cjs +1 -1
- package/dist/validate.d.cts +1 -1
- package/dist/validate.d.mts +1 -1
- package/dist/validate.mjs +1 -1
- package/dist/{verify-H3Pr2gNw.d.mts → verify-C1UU84eq.d.mts} +22 -2
- package/dist/{verify-BM0sGmsf.d.cts → verify-DlN6tK8_.d.cts} +22 -2
- package/dist/{verify-Dek6gHCS.cjs → verify-IVYsI026.cjs} +84 -17
- package/dist/{verify-CWlg3j7B.mjs → verify-lKyu3buS.mjs} +84 -17
- package/docs/evals.md +1 -0
- package/docs/hosts.md +5 -3
- package/docs/index.md +2 -2
- package/docs/meta.json +1 -0
- package/docs/strangler-fig.md +36 -29
- package/docs/verify-in-place.md +326 -0
- package/docs/verify.md +70 -4
- package/llms.txt +1 -0
- package/package.json +1 -1
- package/readme.md +1 -0
package/docs/verify.md
CHANGED
|
@@ -45,7 +45,7 @@ assertAgentMachine(machine, { warnings: true });
|
|
|
45
45
|
| `decide-without-events` | error | A state invokes `agent.decide` but neither it nor any ancestor handles any event, so the chosen event can never be delivered. |
|
|
46
46
|
| `decide-allows-human-event` | error | A decision's literal `allowedEvents` names an event the state or an ancestor handles through a transition tagged `meta: { actor: "human" }`, so the model is never offered it. See [Events a person sends](decisions.md#events-a-person-sends). |
|
|
47
47
|
| `invoke-without-on-error` | warning | An invoke declares no `onError`, and neither its state nor any ancestor handles an actor error, so a rejected request or actor lands the machine in an error state with no modeled recovery. Add `onError` targeting a `failed` final state or a bounded retry. |
|
|
48
|
-
| `direct-object-src` | warning | An invoke `src` is a direct object or machine value that `runAgent` cannot rebind, so it inherits no host executors.
|
|
48
|
+
| `direct-object-src` | warning | An invoke `src` is a direct object or machine value that `runAgent` cannot rebind, so it inherits no host executors. |
|
|
49
49
|
|
|
50
50
|
## Test assertions
|
|
51
51
|
|
|
@@ -173,6 +173,7 @@ settled?.resolvedRequest; // { kind: 'text', src: 'parse', id: 'parse', outcome:
|
|
|
173
173
|
|
|
174
174
|
`explorePaths(machine, { input, maxDepth?, maxPaths?, maxNodes?, budget?, outputs?, errors?, payloads?, dedupeBy?, interleave?, children?, timers?, usage?, steps? })` enumerates decision and external-event branches without a model, and reports coverage.
|
|
175
175
|
|
|
176
|
+
- `input` is validated against the machine's input schema, as `runAgent` does. Input the schema rejects throws `AgentError` with code `invalid-machine-input`, so a misspelled field fails loudly instead of proving a rule over a start state no run can reach. The parsed value, with defaults filled, seeds the search. `canReach` and `simulateAgent` validate `input` the same way.
|
|
176
177
|
- At each decision, it forks one branch per candidate event. Guard-rejected candidates count in `prunedByGuard` and are not explored.
|
|
177
178
|
- A decision with no candidate the snapshot takes, because its guards refuse every one or the closed choice offers nothing, exhausts as it does in a live run. That branch records `{ type: 'xstate.error.actor', actorId }` and takes the invoke's `onError`. With no `onError`, it ends in an `error` terminal.
|
|
178
179
|
- At an idle wait, it forks one branch per externally accepted event and per armed timer. See [Timers](#timers).
|
|
@@ -258,6 +259,7 @@ By default, events are applied as `{ type }`. A guard that reads the payload, su
|
|
|
258
259
|
- A text request's canned value is its `result`, with `messages: []`. Wrap it in `textOutput(result, { messages })` from `@statelyai/agent/testing` when the machine reads `output.messages`.
|
|
259
260
|
- `dedupeBy` projects a settled snapshot to a key. Each key is explored once, or again when reached at a shallower depth. The key must include every field a guard, choice, or `onDone` reads, or the search misses paths. Include `snapshot.timers` when a delayed `enq.raise` outlives the state that scheduled it.
|
|
260
261
|
- Under `dedupeBy`, `terminals` are representative paths, not every path: a revisited key is not explored again, so the paths below it appear once.
|
|
262
|
+
- `canReach` without `dedupeBy` uses a default key: the state value, context, status, history, pending requests and their invoke inputs, children and their inputs, armed timers, and coupled children's states (their timers included). A step whose context or inputs hold anything but plain data (a `Map`, a class instance, a function) is not deduplicated. The key holds structure, not identity: a guard that compares context with an object held outside the snapshot, such as a module-level token, needs its own `dedupeBy`. `explorePaths` without `dedupeBy` explores every path in full.
|
|
261
263
|
|
|
262
264
|
```ts
|
|
263
265
|
const amounts = [0, 1, 100, 100.01, 500];
|
|
@@ -386,7 +388,7 @@ const report = await explorePaths(parentMachine, {
|
|
|
386
388
|
- What a child's move sends its own children, and their replies, happen in the same move. A child's send to a sibling is delivered to that sibling, entry sends included. A child's sends reach their targets in the order it made them, each run to quiescence before the next, as a live run does. A send to an actor the search does not model lists the event type in `unexpandedEvents`, so the report is not complete.
|
|
387
389
|
- Child moves count against `maxDepth` like the parent's own branches. On a path they carry `actor` with the child's id (a grandchild's as `child.grandchild`): the host events it received, its choices, its settled work. The events it sends the parent appear as the parent receives them.
|
|
388
390
|
- A target predicate and `dedupeBy` see each coupled child's explored state: `snapshot.children[id].getSnapshot()` returns the state the search stepped the child to, its own coupled children likewise. A parent's send and the child's receipt are one state, so a cross-actor invariant such as "every chunk the parent forwarded, a child received" can be a target.
|
|
389
|
-
- With `dedupeBy`, each child's state joins the key.
|
|
391
|
+
- With `dedupeBy`, and in `canReach`'s default key, each child's state joins the key. `explorePaths` without `dedupeBy` explores every interleaving in full.
|
|
390
392
|
- A coupled child's text requests deliver `@agent.usage` to the root, as a run does. See [Usage events](#usage-events).
|
|
391
393
|
- A child that stops on `needs-output` or `max-depth` ends the parent branch the same way, so the parent report is not complete.
|
|
392
394
|
|
|
@@ -463,7 +465,7 @@ When `complete` is false, `unexplored` lists why, in the wording of the `explore
|
|
|
463
465
|
|
|
464
466
|
Raise the caps, add `dedupeBy`, supply the missing outputs, or add `payloads` entries until it is true.
|
|
465
467
|
|
|
466
|
-
A long-lived machine that never finishes, such as a chat loop, ends every path at `maxDepth
|
|
468
|
+
A long-lived machine that never finishes, such as a chat loop, ends every path at `maxDepth` unless its states repeat, so `complete` is false when its context keeps growing. Its proofs are bounded:
|
|
467
469
|
|
|
468
470
|
- `reachable: false` whose only `unexplored` notes are `max-depth` notes means no path of up to `maxDepth` branch points reaches the target.
|
|
469
471
|
- Any other note, such as `needs-output` or `unexpanded`, is a gap within that bound too.
|
|
@@ -505,7 +507,7 @@ const turnAfterStop = await canReach(
|
|
|
505
507
|
```
|
|
506
508
|
|
|
507
509
|
- `after` and `reach` match [steps](#steps). `after` is an event type or a step predicate.
|
|
508
|
-
- `reach` is checked on
|
|
510
|
+
- `reach` is checked on the first step that matched `after` and every step after it. On the trigger's own step, `started`, `settled` and `state` are what the trigger's transition did, so a turn it starts is a violation. That step's `event` is the trigger itself: match its effects, not its event.
|
|
509
511
|
- `after` and `reach` may read an event's payload, so an event with no schema is unexpanded until `payloads` lists its variants, even when no transition reads it. A timer's step carries the event the timer delivers.
|
|
510
512
|
- The witness is the event sequence to the first violation.
|
|
511
513
|
- A turn already running when STOP arrives started before it, so it is not a violation. Check that turn with `settled` if the rule covers it.
|
|
@@ -513,6 +515,70 @@ const turnAfterStop = await canReach(
|
|
|
513
515
|
- Test the rule with a mutant that breaks it (a loop that starts another turn after STOP), as with a loosened guard.
|
|
514
516
|
- Don't match strings over `terminal.path` for an ordering rule: it lists branch points only and names actors by generated id.
|
|
515
517
|
|
|
518
|
+
## Recorded runs against a model
|
|
519
|
+
|
|
520
|
+
`checkConformance` from `@statelyai/agent/testing` checks a recorded run of existing code against a model of that code. [Verify in place](verify-in-place.md) walks through the whole track with a worked example. The code is not changed and does not run. Use it to verify a workflow in place before, or instead of, migrating it to a machine:
|
|
521
|
+
|
|
522
|
+
1. Write the model: a machine whose actors stand for the code's work, with placeholder implementations.
|
|
523
|
+
2. Prove the model's rules with `canReach` and `explorePaths`, as ordinary tests.
|
|
524
|
+
3. Record runs of the code with `createRecorder`, and check each one against the model.
|
|
525
|
+
|
|
526
|
+
```ts no-check
|
|
527
|
+
import { checkConformance, createRecorder } from "@statelyai/agent/testing";
|
|
528
|
+
import { traceCoverage } from "@statelyai/agent/evals";
|
|
529
|
+
|
|
530
|
+
const recorder = createRecorder();
|
|
531
|
+
await supportLoop({
|
|
532
|
+
assist: recorder.fn("assist", assist),
|
|
533
|
+
approve: recorder.fn("approve", approve),
|
|
534
|
+
send: recorder.fn("send", send),
|
|
535
|
+
});
|
|
536
|
+
|
|
537
|
+
const report = await checkConformance(supportModel, {
|
|
538
|
+
input: { maxSteps: 3 },
|
|
539
|
+
// One observation per recorded call the model can see.
|
|
540
|
+
observations: recording.effects.flatMap(toObservation),
|
|
541
|
+
actors: {
|
|
542
|
+
assist: {
|
|
543
|
+
observes: "assist",
|
|
544
|
+
matches: ({ input, observation }) => input.step === observation.step,
|
|
545
|
+
output: ({ observation }) => ({ done: observation.done, email: observation.email }),
|
|
546
|
+
},
|
|
547
|
+
send: { observes: "send" },
|
|
548
|
+
},
|
|
549
|
+
// Observations the model receives as events while it waits.
|
|
550
|
+
events: { approval: (o) => ({ type: o.approved ? "APPROVE" : "REJECT" }) },
|
|
551
|
+
});
|
|
552
|
+
|
|
553
|
+
report.conforms; // false: the code did something the model does not allow
|
|
554
|
+
report.deviation; // { reason: "extra-observations", expected: "idle, waiting for an event", observed: { kind: "send" }, state: "awaitingApproval" }
|
|
555
|
+
traceCoverage(supportModel, reports.map((r) => r.trace)); // states and handlers no recorded run reached
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
- The model runs with `runAgent`. Each actor in `actors` is answered from the observations instead of running. An actor with `observes` consumes the next observation, which must be of that kind.
|
|
559
|
+
- `output` builds the actor's output from the observation. When the output carries state the recording does not show, such as a queue the code keeps internally, build it from the model's own `input` and the observation. Rebuilding it from the code's logic would make the check prove nothing.
|
|
560
|
+
- `matches` returns `false` when the actor's input does not fit the observation. `error` returns a value to reject the actor, for work that failed in the recording.
|
|
561
|
+
- An actor without `observes` consumes nothing. It is listed in `report.unobserved`, because the recording cannot check what it did. A run whose history or streaming was off cannot see those effects, so a bug in them goes unnoticed.
|
|
562
|
+
- `events` maps observation kinds to events. When the model settles idle and the next observation has one of those kinds, it is sent and the run resumes. An entry `{ event, matches }` also compares the observation with the model's `context` and `state` before sending it, such as the draft a person approved against the draft the model holds.
|
|
563
|
+
- `output: { observes, matches }` checks the code's return value. Map the recording's `output` effect to an observation of kind `observes`. The model must finish there, and `matches` compares its output with that observation. Without `output`, a recorded return value is an `extra-observations` deviation. For a run where the code threw, record the throw as the outcome and model the failure as a final state. Without `output`, nothing consumes a recorded throw at the end, so a run that otherwise conforms reports it as `extra-observations`. A run whose throw is left out of the observations has its outcome unchecked.
|
|
564
|
+
- The first deviation stops the run and is reported with its `reason`:
|
|
565
|
+
- `unexpected-observation`: the next observation is of another kind.
|
|
566
|
+
- `missing-observation`: the recording had ended, or the model finished with `output` set and the code never returned.
|
|
567
|
+
- `input-mismatch`: an actor's or event's `matches` returned `false`.
|
|
568
|
+
- `output-mismatch`: the model finished, and `output.matches` returned `false`.
|
|
569
|
+
- `extra-observations`: the model finished or settled idle, and the recording continues.
|
|
570
|
+
- `ignored-event`: the model was idle, and it does not handle the event the next observation maps to.
|
|
571
|
+
- `unmapped-actor`: the model invoked a src with no entry in `actors`.
|
|
572
|
+
|
|
573
|
+
`expected` is what the model did next, `observed` is the recording's next observation, and `state` is the model's state value.
|
|
574
|
+
- An `actors` key the model does not declare throws `AgentError` with code `unknown-actor-source`.
|
|
575
|
+
- Every actor the model invokes must be declared by name in `setupAgent({ actors })` and invoked by `src`. An invoke that names its logic inline would run that logic for real, so `checkConformance` throws `inline-actor-source` before the run starts.
|
|
576
|
+
- Code that builds its AI SDK model inline or passes module-level tools to `generateText` is recorded with `recordAiSdk`. See [Verify in place](verify-in-place.md#ai-sdk-code).
|
|
577
|
+
- A callback in `actors`, `events`, or `output` that throws ends the check with `AgentError` code `conformance-callback-error`. It is not counted as the actor failing, so a bug in the mapping cannot pass as a conforming run.
|
|
578
|
+
- Concurrent actors consume observations in the order they start, so a model that runs observed actors in parallel needs observations in that order.
|
|
579
|
+
|
|
580
|
+
What a conforming run shows: the code behaved as the model allows on that run, and the model's rules are proved. It does not show that the code can never break a rule; a run you did not record still can. After a migration with [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) and `checkParity`, the machine enforces the rules at runtime, and the same model and proofs carry over.
|
|
581
|
+
|
|
516
582
|
## CI checks
|
|
517
583
|
|
|
518
584
|
Everything on this page runs without an API key, so a small script is enough for CI or a generation loop.
|
package/llms.txt
CHANGED
|
@@ -41,6 +41,7 @@ Not v5: transitions are plain functions returning `{ target, context }` (no `ass
|
|
|
41
41
|
- [Testing and verification](docs/verify.md): Statically lint, simulate, and explore agent machines without any API keys or model calls, from `@statelyai/agent/testing`. Exploration branches on payloads and invoke outputs, and `canReach` returns a shortest witness or a complete unreachability result.
|
|
42
42
|
- [Evals](docs/evals.md): Score agent runs on output, trajectory, and token budget using the seams runAgent already returns, with plain vitest or a vendor harness like Braintrust.
|
|
43
43
|
- [Coming from LangGraph](docs/langgraph-comparison.md): A term-by-term translation of LangGraph concepts, the same agent built both ways, an honest dimension-by-dimension comparison, and the ported examples.
|
|
44
|
+
- [Verify in place](docs/verify-in-place.md): Model an existing agent workflow, prove the model's rules with canReach, and check recorded runs of the unchanged code against it with checkConformance; migration is optional.
|
|
44
45
|
- [Migrating from a hand-rolled loop](docs/from-a-loop.md): Convert a while-loop agent into an agent machine one step at a time, then, experimentally, improve a software-factory loop's triage from its traces without loosening its rules.
|
|
45
46
|
- [Agent patterns](docs/patterns.md): Common agent patterns (ReAct, reflection, plan-and-execute, RAG, supervisor, and more) as copy-paste XState machines, one runnable file each.
|
|
46
47
|
- [Statechart policy examples](docs/statechart-policy-examples.md): Three examples that put consequential decisions in visible machine states: consensus review, booking compensation, and deadline escalation.
|
package/package.json
CHANGED
package/readme.md
CHANGED
|
@@ -31,6 +31,7 @@ Stately Agent 2 is in alpha. APIs may change before the stable release.
|
|
|
31
31
|
|
|
32
32
|
### Brownfield: an existing agent
|
|
33
33
|
|
|
34
|
+
- **Verify an existing agent in place.** Write a model of the code's control flow, prove its rules with `canReach`, and check recorded runs of the unchanged code against it with `checkConformance`. Migrating the code to a machine stays optional. See [Verify in place](https://github.com/statelyai/agent/blob/next/docs/verify-in-place.md).
|
|
34
35
|
- **Wrap an existing workflow, then carve it.** `wrapWorkflow` runs an async workflow unchanged inside a machine. In minutes and with no rewrite, you get inspection, traces, the event log, and a recorded baseline. Reliability proofs such as `canReach` arrive per phase, as each phase moves into states, with `checkParity` against the original after each. See [Strangler-fig retrofit](https://github.com/statelyai/agent/blob/next/docs/strangler-fig.md).
|
|
35
36
|
- **Retrofit an existing agent.** Turn a `while` loop into a machine: your SDK calls, tools, and retry code become the executors; the machine replaces only the control flow. See [Migrating from a hand-rolled loop](https://github.com/statelyai/agent/blob/next/docs/from-a-loop.md). A coding agent does it with the `loop-to-machine` skill (see [Agent skills](#agent-skills)): "Use the loop-to-machine skill to convert `src/support-agent.ts` to a machine."
|
|
36
37
|
- **Let an agent improve its own logic (experimental).** Keep the rules in a fixed TypeScript environment and let a model rewrite the JSON policy inside it, gated by `verifyAgentConfig` and promoted only when held-out scenarios improve. See [Environment and policy](https://github.com/statelyai/agent/blob/next/docs/environment-and-policy.md) and the [`improve-machine` skill](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/improve-machine/SKILL.md).
|