@statelyai/agent 2.0.0-alpha.40 → 2.0.0-alpha.41
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/dist/evals.cjs +1 -1
- package/dist/evals.d.cts +1 -1
- package/dist/evals.d.mts +1 -1
- package/dist/evals.mjs +1 -1
- package/dist/index.d.cts +1 -1
- package/dist/index.d.mts +1 -1
- package/dist/testing.cjs +376 -17
- package/dist/testing.d.cts +181 -5
- package/dist/testing.d.mts +181 -5
- package/dist/testing.mjs +377 -20
- package/dist/{utils-DAclF-kw.d.cts → utils-BMxUcjay.d.cts} +57 -1
- package/dist/{utils-B8k-YPYI.d.mts → utils-BV7pxhD-.d.mts} +57 -1
- package/dist/validate.cjs +1 -1
- package/dist/validate.d.cts +1 -1
- package/dist/validate.d.mts +1 -1
- package/dist/validate.mjs +1 -1
- package/dist/{verify-CIu4mqIZ.d.mts → verify-C1UU84eq.d.mts} +7 -1
- package/dist/{verify-CTHlmBbh.d.cts → verify-DlN6tK8_.d.cts} +7 -1
- package/dist/{verify-D8UDqU4l.cjs → verify-IVYsI026.cjs} +65 -14
- package/dist/{verify-Cacz6XSk.mjs → verify-lKyu3buS.mjs} +65 -14
- package/docs/evals.md +1 -0
- package/docs/index.md +2 -2
- package/docs/meta.json +1 -0
- package/docs/verify-in-place.md +326 -0
- package/docs/verify.md +67 -2
- package/llms.txt +1 -0
- package/package.json +1 -1
- package/readme.md +1 -0
package/docs/verify.md
CHANGED
|
@@ -259,6 +259,7 @@ By default, events are applied as `{ type }`. A guard that reads the payload, su
|
|
|
259
259
|
- A text request's canned value is its `result`, with `messages: []`. Wrap it in `textOutput(result, { messages })` from `@statelyai/agent/testing` when the machine reads `output.messages`.
|
|
260
260
|
- `dedupeBy` projects a settled snapshot to a key. Each key is explored once, or again when reached at a shallower depth. The key must include every field a guard, choice, or `onDone` reads, or the search misses paths. Include `snapshot.timers` when a delayed `enq.raise` outlives the state that scheduled it.
|
|
261
261
|
- Under `dedupeBy`, `terminals` are representative paths, not every path: a revisited key is not explored again, so the paths below it appear once.
|
|
262
|
+
- `canReach` without `dedupeBy` uses a default key: the state value, context, status, history, pending requests and their invoke inputs, children and their inputs, armed timers, and coupled children's states (their timers included). A step whose context or inputs hold anything but plain data (a `Map`, a class instance, a function) is not deduplicated. The key holds structure, not identity: a guard that compares context with an object held outside the snapshot, such as a module-level token, needs its own `dedupeBy`. `explorePaths` without `dedupeBy` explores every path in full.
|
|
262
263
|
|
|
263
264
|
```ts
|
|
264
265
|
const amounts = [0, 1, 100, 100.01, 500];
|
|
@@ -387,7 +388,7 @@ const report = await explorePaths(parentMachine, {
|
|
|
387
388
|
- What a child's move sends its own children, and their replies, happen in the same move. A child's send to a sibling is delivered to that sibling, entry sends included. A child's sends reach their targets in the order it made them, each run to quiescence before the next, as a live run does. A send to an actor the search does not model lists the event type in `unexpandedEvents`, so the report is not complete.
|
|
388
389
|
- Child moves count against `maxDepth` like the parent's own branches. On a path they carry `actor` with the child's id (a grandchild's as `child.grandchild`): the host events it received, its choices, its settled work. The events it sends the parent appear as the parent receives them.
|
|
389
390
|
- A target predicate and `dedupeBy` see each coupled child's explored state: `snapshot.children[id].getSnapshot()` returns the state the search stepped the child to, its own coupled children likewise. A parent's send and the child's receipt are one state, so a cross-actor invariant such as "every chunk the parent forwarded, a child received" can be a target.
|
|
390
|
-
- With `dedupeBy`, each child's state joins the key.
|
|
391
|
+
- With `dedupeBy`, and in `canReach`'s default key, each child's state joins the key. `explorePaths` without `dedupeBy` explores every interleaving in full.
|
|
391
392
|
- A coupled child's text requests deliver `@agent.usage` to the root, as a run does. See [Usage events](#usage-events).
|
|
392
393
|
- A child that stops on `needs-output` or `max-depth` ends the parent branch the same way, so the parent report is not complete.
|
|
393
394
|
|
|
@@ -464,7 +465,7 @@ When `complete` is false, `unexplored` lists why, in the wording of the `explore
|
|
|
464
465
|
|
|
465
466
|
Raise the caps, add `dedupeBy`, supply the missing outputs, or add `payloads` entries until it is true.
|
|
466
467
|
|
|
467
|
-
A long-lived machine that never finishes, such as a chat loop, ends every path at `maxDepth
|
|
468
|
+
A long-lived machine that never finishes, such as a chat loop, ends every path at `maxDepth` unless its states repeat, so `complete` is false when its context keeps growing. Its proofs are bounded:
|
|
468
469
|
|
|
469
470
|
- `reachable: false` whose only `unexplored` notes are `max-depth` notes means no path of up to `maxDepth` branch points reaches the target.
|
|
470
471
|
- Any other note, such as `needs-output` or `unexpanded`, is a gap within that bound too.
|
|
@@ -514,6 +515,70 @@ const turnAfterStop = await canReach(
|
|
|
514
515
|
- Test the rule with a mutant that breaks it (a loop that starts another turn after STOP), as with a loosened guard.
|
|
515
516
|
- Don't match strings over `terminal.path` for an ordering rule: it lists branch points only and names actors by generated id.
|
|
516
517
|
|
|
518
|
+
## Recorded runs against a model
|
|
519
|
+
|
|
520
|
+
`checkConformance` from `@statelyai/agent/testing` checks a recorded run of existing code against a model of that code. [Verify in place](verify-in-place.md) walks through the whole track with a worked example. The code is not changed and does not run. Use it to verify a workflow in place before, or instead of, migrating it to a machine:
|
|
521
|
+
|
|
522
|
+
1. Write the model: a machine whose actors stand for the code's work, with placeholder implementations.
|
|
523
|
+
2. Prove the model's rules with `canReach` and `explorePaths`, as ordinary tests.
|
|
524
|
+
3. Record runs of the code with `createRecorder`, and check each one against the model.
|
|
525
|
+
|
|
526
|
+
```ts no-check
|
|
527
|
+
import { checkConformance, createRecorder } from "@statelyai/agent/testing";
|
|
528
|
+
import { traceCoverage } from "@statelyai/agent/evals";
|
|
529
|
+
|
|
530
|
+
const recorder = createRecorder();
|
|
531
|
+
await supportLoop({
|
|
532
|
+
assist: recorder.fn("assist", assist),
|
|
533
|
+
approve: recorder.fn("approve", approve),
|
|
534
|
+
send: recorder.fn("send", send),
|
|
535
|
+
});
|
|
536
|
+
|
|
537
|
+
const report = await checkConformance(supportModel, {
|
|
538
|
+
input: { maxSteps: 3 },
|
|
539
|
+
// One observation per recorded call the model can see.
|
|
540
|
+
observations: recording.effects.flatMap(toObservation),
|
|
541
|
+
actors: {
|
|
542
|
+
assist: {
|
|
543
|
+
observes: "assist",
|
|
544
|
+
matches: ({ input, observation }) => input.step === observation.step,
|
|
545
|
+
output: ({ observation }) => ({ done: observation.done, email: observation.email }),
|
|
546
|
+
},
|
|
547
|
+
send: { observes: "send" },
|
|
548
|
+
},
|
|
549
|
+
// Observations the model receives as events while it waits.
|
|
550
|
+
events: { approval: (o) => ({ type: o.approved ? "APPROVE" : "REJECT" }) },
|
|
551
|
+
});
|
|
552
|
+
|
|
553
|
+
report.conforms; // false: the code did something the model does not allow
|
|
554
|
+
report.deviation; // { reason: "extra-observations", expected: "idle, waiting for an event", observed: { kind: "send" }, state: "awaitingApproval" }
|
|
555
|
+
traceCoverage(supportModel, reports.map((r) => r.trace)); // states and handlers no recorded run reached
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
- The model runs with `runAgent`. Each actor in `actors` is answered from the observations instead of running. An actor with `observes` consumes the next observation, which must be of that kind.
|
|
559
|
+
- `output` builds the actor's output from the observation. When the output carries state the recording does not show, such as a queue the code keeps internally, build it from the model's own `input` and the observation. Rebuilding it from the code's logic would make the check prove nothing.
|
|
560
|
+
- `matches` returns `false` when the actor's input does not fit the observation. `error` returns a value to reject the actor, for work that failed in the recording.
|
|
561
|
+
- An actor without `observes` consumes nothing. It is listed in `report.unobserved`, because the recording cannot check what it did. A run whose history or streaming was off cannot see those effects, so a bug in them goes unnoticed.
|
|
562
|
+
- `events` maps observation kinds to events. When the model settles idle and the next observation has one of those kinds, it is sent and the run resumes. An entry `{ event, matches }` also compares the observation with the model's `context` and `state` before sending it, such as the draft a person approved against the draft the model holds.
|
|
563
|
+
- `output: { observes, matches }` checks the code's return value. Map the recording's `output` effect to an observation of kind `observes`. The model must finish there, and `matches` compares its output with that observation. Without `output`, a recorded return value is an `extra-observations` deviation. For a run where the code threw, record the throw as the outcome and model the failure as a final state. Without `output`, nothing consumes a recorded throw at the end, so a run that otherwise conforms reports it as `extra-observations`. A run whose throw is left out of the observations has its outcome unchecked.
|
|
564
|
+
- The first deviation stops the run and is reported with its `reason`:
|
|
565
|
+
- `unexpected-observation`: the next observation is of another kind.
|
|
566
|
+
- `missing-observation`: the recording had ended, or the model finished with `output` set and the code never returned.
|
|
567
|
+
- `input-mismatch`: an actor's or event's `matches` returned `false`.
|
|
568
|
+
- `output-mismatch`: the model finished, and `output.matches` returned `false`.
|
|
569
|
+
- `extra-observations`: the model finished or settled idle, and the recording continues.
|
|
570
|
+
- `ignored-event`: the model was idle, and it does not handle the event the next observation maps to.
|
|
571
|
+
- `unmapped-actor`: the model invoked a src with no entry in `actors`.
|
|
572
|
+
|
|
573
|
+
`expected` is what the model did next, `observed` is the recording's next observation, and `state` is the model's state value.
|
|
574
|
+
- An `actors` key the model does not declare throws `AgentError` with code `unknown-actor-source`.
|
|
575
|
+
- Every actor the model invokes must be declared by name in `setupAgent({ actors })` and invoked by `src`. An invoke that names its logic inline would run that logic for real, so `checkConformance` throws `inline-actor-source` before the run starts.
|
|
576
|
+
- Code that builds its AI SDK model inline or passes module-level tools to `generateText` is recorded with `recordAiSdk`. See [Verify in place](verify-in-place.md#ai-sdk-code).
|
|
577
|
+
- A callback in `actors`, `events`, or `output` that throws ends the check with `AgentError` code `conformance-callback-error`. It is not counted as the actor failing, so a bug in the mapping cannot pass as a conforming run.
|
|
578
|
+
- Concurrent actors consume observations in the order they start, so a model that runs observed actors in parallel needs observations in that order.
|
|
579
|
+
|
|
580
|
+
What a conforming run shows: the code behaved as the model allows on that run, and the model's rules are proved. It does not show that the code can never break a rule; a run you did not record still can. After a migration with [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) and `checkParity`, the machine enforces the rules at runtime, and the same model and proofs carry over.
|
|
581
|
+
|
|
517
582
|
## CI checks
|
|
518
583
|
|
|
519
584
|
Everything on this page runs without an API key, so a small script is enough for CI or a generation loop.
|
package/llms.txt
CHANGED
|
@@ -41,6 +41,7 @@ Not v5: transitions are plain functions returning `{ target, context }` (no `ass
|
|
|
41
41
|
- [Testing and verification](docs/verify.md): Statically lint, simulate, and explore agent machines without any API keys or model calls, from `@statelyai/agent/testing`. Exploration branches on payloads and invoke outputs, and `canReach` returns a shortest witness or a complete unreachability result.
|
|
42
42
|
- [Evals](docs/evals.md): Score agent runs on output, trajectory, and token budget using the seams runAgent already returns, with plain vitest or a vendor harness like Braintrust.
|
|
43
43
|
- [Coming from LangGraph](docs/langgraph-comparison.md): A term-by-term translation of LangGraph concepts, the same agent built both ways, an honest dimension-by-dimension comparison, and the ported examples.
|
|
44
|
+
- [Verify in place](docs/verify-in-place.md): Model an existing agent workflow, prove the model's rules with canReach, and check recorded runs of the unchanged code against it with checkConformance; migration is optional.
|
|
44
45
|
- [Migrating from a hand-rolled loop](docs/from-a-loop.md): Convert a while-loop agent into an agent machine one step at a time, then, experimentally, improve a software-factory loop's triage from its traces without loosening its rules.
|
|
45
46
|
- [Agent patterns](docs/patterns.md): Common agent patterns (ReAct, reflection, plan-and-execute, RAG, supervisor, and more) as copy-paste XState machines, one runnable file each.
|
|
46
47
|
- [Statechart policy examples](docs/statechart-policy-examples.md): Three examples that put consequential decisions in visible machine states: consensus review, booking compensation, and deadline escalation.
|
package/package.json
CHANGED
package/readme.md
CHANGED
|
@@ -31,6 +31,7 @@ Stately Agent 2 is in alpha. APIs may change before the stable release.
|
|
|
31
31
|
|
|
32
32
|
### Brownfield: an existing agent
|
|
33
33
|
|
|
34
|
+
- **Verify an existing agent in place.** Write a model of the code's control flow, prove its rules with `canReach`, and check recorded runs of the unchanged code against it with `checkConformance`. Migrating the code to a machine stays optional. See [Verify in place](https://github.com/statelyai/agent/blob/next/docs/verify-in-place.md).
|
|
34
35
|
- **Wrap an existing workflow, then carve it.** `wrapWorkflow` runs an async workflow unchanged inside a machine. In minutes and with no rewrite, you get inspection, traces, the event log, and a recorded baseline. Reliability proofs such as `canReach` arrive per phase, as each phase moves into states, with `checkParity` against the original after each. See [Strangler-fig retrofit](https://github.com/statelyai/agent/blob/next/docs/strangler-fig.md).
|
|
35
36
|
- **Retrofit an existing agent.** Turn a `while` loop into a machine: your SDK calls, tools, and retry code become the executors; the machine replaces only the control flow. See [Migrating from a hand-rolled loop](https://github.com/statelyai/agent/blob/next/docs/from-a-loop.md). A coding agent does it with the `loop-to-machine` skill (see [Agent skills](#agent-skills)): "Use the loop-to-machine skill to convert `src/support-agent.ts` to a machine."
|
|
36
37
|
- **Let an agent improve its own logic (experimental).** Keep the rules in a fixed TypeScript environment and let a model rewrite the JSON policy inside it, gated by `verifyAgentConfig` and promoted only when held-out scenarios improve. See [Environment and policy](https://github.com/statelyai/agent/blob/next/docs/environment-and-policy.md) and the [`improve-machine` skill](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/improve-machine/SKILL.md).
|