norn-cli 2.9.7 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,29 @@
2
2
 
3
3
  All notable changes to the "Norn" extension will be documented in this file.
4
4
 
5
+ ## [3.0.0] - 2026-08-11
6
+
7
+ ### Added
8
+ - **`.nornagent` files — the Agent Workbench** - a new file type that declares AI agents alongside your API tests. An imported `.nornagent` sidecar declares the models and agents; a `.norn` sequence owns execution order and calls them with `var reply = run <Agent> "…"`. Agents can be given MCP-backed tools, `accepts` / `returns` JSON Schema contracts that are validated on every boundary, and other agents they are allowed to call.
9
+ - **Agent graph canvas** - a read-only VS Code canvas for a `.nornagent` file showing who calls whom, the contract on each boundary, and how each hop turned out. Hover for models, timings, usage, prompts, payloads, and contract issues.
10
+ - **Run recording, replay, and stepping** - runs are recorded by default under `.norn-cache/runs/` (last 20 kept). Replay a recording to step it in the existing `norn` debugger, or play it back hop by hop on the canvas — nothing re-executes and no model is called. The graph toolbar labels each run with the `.norn` file it came from, and **This file — current view** returns you to the open file.
11
+ - **`judge` statements — AI evals in a sequence** - `judge <value> with <Agent> expects "…"` scores free text that an assertion cannot. The judge rules on each expectation independently and quotes its evidence; Norn computes the pass or fail, so the verdict never comes from the model. A long checklist lives in a file with `expects file <path>`, where an ordinary Markdown list works as-is. Every ruling carries a reason, and a quote that cannot be found in the judged text is flagged.
12
+ - **Judge results everywhere a run is reported** - verdicts appear in the response panel, on the agent graph, in the CLI summary, and in per-statement JUnit output, and travel with the run's trace and recording. A failed verdict fails the sequence and exits the CLI non-zero.
13
+ - **Request body variable highlighting** - a variable used as a request body is now coloured as a variable rather than as a header group reference.
14
+
15
+ ### Changed
16
+ - **`model` blocks in `.nornagent`** - a model is declared as a block naming every input it needs:
17
+
18
+ ```nornagent
19
+ model Workbench
20
+ provider openai
21
+ name gpt-4o
22
+ apiKey {{$env.OPENAI_API_KEY}}
23
+ end model
24
+ ```
25
+
26
+ This replaces the earlier one-line `model <Alias>: <provider>/<model-id>` form, which implied both the provider/model split and an undeclared environment-variable name. `apiKey` and `baseUrl` are ordinary Norn templates resolved per run, so the file says exactly which value it needs and the secret itself stays in the environment.
27
+
5
28
  ## [2.9.1] - 2026-06-22
6
29
 
7
30
  ### Fixed
package/CLAUDE.md ADDED
@@ -0,0 +1,68 @@
1
+ # Norn Extension — Claude Code instructions
2
+
3
+ ## Read these first
4
+
5
+ 1. **[`NOW.md`](NOW.md)** — the tool-agnostic active-work board shared by every agent and by
6
+ Peter. Whatever sits under **Active now** is the current priority. This board, not any
7
+ assistant's memory, is the source of truth for what we are working on.
8
+ 2. **[`AGENTS.md`](AGENTS.md)** — the full repo instructions (CLI parity requirement, how to
9
+ run the local CLI, code style, testing). Everything in that file applies here.
10
+
11
+ ## Skills: consult `.github/skills/` — always
12
+
13
+ This repo keeps its accumulated implementation knowledge in **[`.github/skills/`](.github/skills/)**,
14
+ maintained by every agent that works here. Treat it as required reading, not an optional
15
+ reference.
16
+
17
+ **Before touching any area of the codebase, check `.github/skills/` for a skill covering it
18
+ and read that skill.** Do this even for changes that look small — the skills exist precisely
19
+ because these areas have non-obvious rules that are easy to break.
20
+
21
+ Current skills and what they cover:
22
+
23
+ | Skill | Read it when working on |
24
+ |-------|------------------------|
25
+ | `norn-syntax-highlighting` | TextMate grammars in `syntaxes/`, token scopes, highlighting bugs |
26
+ | `norn-agent-workbench` | `.nornagent` sidecars, agent `run` steps, providers, contracts, record/replay, agent graph |
27
+ | `norn-intellisense-preferences` | Completion providers, IntelliSense behaviour |
28
+ | `norn-value-resolution` | Variables, interpolation, `.nornenv`, value precedence |
29
+ | `add-norn-keyword` | Adding a keyword to the `.norn` language |
30
+ | `add-assertion-operator` | Adding an assertion operator |
31
+ | `add-sequence-step-type` | Adding a new sequence step type |
32
+ | `add-imported-sidecar-language` | Adding a new imported sidecar file type |
33
+ | `norn-cli-integration` | `src/cli.ts` and CLI parity |
34
+ | `norn-debugger-integration` | DAP / debugger work |
35
+ | `norn-response-panel` | The response webview panel |
36
+ | `norn-test-explorer` | VS Code Test Explorer integration |
37
+ | `norn-sequence-tags` | Sequence tags and filtering |
38
+ | `norn-api-coverage` | `.nornapi` coverage features |
39
+ | `norn-mcp-tools` | MCP client and tool wiring |
40
+ | `norn-k8s` | Kubernetes features |
41
+ | `norn-terminal` | Norn Terminal / PTY work |
42
+ | `norn-env-templates` | `.nornenv` templates |
43
+ | `norn-publishing` | Packaging and publishing the extension |
44
+ | `norn-starter-catalog` | Starter templates catalog |
45
+ | `norn-title-cards` | Title cards |
46
+ | `demo-task-helper` | Building demos |
47
+ | `readme-sync` | Keeping README/docs in sync with features |
48
+
49
+ The table can go stale — list the directory when the work doesn't obviously map to a row.
50
+
51
+ ### Maintaining skills
52
+
53
+ Per `AGENTS.md`, skills are living documents:
54
+
55
+ - If a skill is **wrong or outdated**, fix it as part of the change.
56
+ - If a skill is **missing** for an area you just learned, write one in Agent Skills format.
57
+ - After creating or editing any skill, run `npm run validate:skills`.
58
+
59
+ Capture the lesson that was hard to learn, not a restatement of the code.
60
+
61
+ ## Verification
62
+
63
+ Never use `npx norn` — that runs the published package, not local changes:
64
+
65
+ ```bash
66
+ npm run compile
67
+ node ./dist/cli.js tests/file.norn --env prelive
68
+ ```
package/NOW.md ADDED
@@ -0,0 +1,157 @@
1
+ # NOW — Active Work Board
2
+
3
+ > **The single source of truth for what we are working on _right now_** — for Peter and
4
+ > for every AI agent (Claude, Codex, Copilot, Cursor, any). Deliberately tool-agnostic:
5
+ > it lives here in the repo, **not** in any one assistant's memory or private notes.
6
+ >
7
+ > We have a lot of planning docs; this board exists so nobody (human or AI) has to guess
8
+ > what's current. If it's on this board, it's the work. If it's not, it isn't — yet.
9
+
10
+ ## How this board works
11
+
12
+ - **Empty by default.** Nothing is "current" unless it has been explicitly **Accepted**
13
+ onto this board by Peter. An empty "Active now" means: no directed work in flight —
14
+ don't start big work off a backlog/plan doc without Peter accepting it here first.
15
+ - **Accepting work:** Peter says "Accept X" (or moves it here). That's the signal it
16
+ becomes current. Until then, ideas live in the plan/backlog docs, not here.
17
+ - **This board is an _index_, not the detail.** Each item is one or two lines + a link to
18
+ its real plan doc. Keep the detail in the linked doc; keep this page short and scannable.
19
+ - **Finishing work:** when an item is done, move it to "Recently done" (keep the last
20
+ handful), then prune. Don't let this page grow.
21
+ - **Agents:** treat items under "Active now" as the current priority order. Do these
22
+ before picking up anything from the wider `Docs/` set.
23
+
24
+ ---
25
+
26
+ ## Active now
27
+
28
+ ### 🧪 Agent Workbench — Slice 5: AI evals
29
+ The last of the four things nothing else does in one place: let an **AI judge score what a
30
+ contract cannot**, with thresholds useful in CI rather than noisy. An umbrella, not a buildable
31
+ slice — it breaks into six items (authoring, rubric/scoring, judge execution, thresholds,
32
+ reporting, fixtures/docs), specced and built one at a time.
33
+ → Spec: [`Docs/to-be-built/agent-workbench-slice-5.md`](Docs/to-be-built/agent-workbench-slice-5.md)
34
+
35
+ **Next step:** **5f — fixtures, docs, positioning**, the last item of slice 5. The feature is
36
+ complete and reported across every surface; what is missing is a demo anyone else can run and an
37
+ explanation of what it is for. 5f has no spec yet.
38
+
39
+ **5b and 5d are closed.** 5b was made unnecessary by the checklist scoring model. 5d's content
40
+ was decided in conversation — a failed judge stops the sequence, the CLI already exits `1`, and
41
+ per-expectation strictness plus the replay-drift check are future work Peter deferred
42
+ 2026-08-10.
43
+ → 5a spec (built): [`Docs/to-be-built/agent-workbench-slice-5a.md`](Docs/to-be-built/agent-workbench-slice-5a.md)
44
+ → 5c spec (built): [`Docs/to-be-built/agent-workbench-slice-5c.md`](Docs/to-be-built/agent-workbench-slice-5c.md)
45
+ → 5e spec (built): [`Docs/to-be-built/agent-workbench-slice-5e.md`](Docs/to-be-built/agent-workbench-slice-5e.md)
46
+
47
+ ---
48
+
49
+ ## Backlog / accepted-but-not-started
50
+
51
+ _(empty)_
52
+
53
+ ## Parked
54
+
55
+ ### Contract & Drift + Voiden-playbook marketing
56
+ The previous direction, parked 2026-08-07 — untested, not failed. Stages 1+2 are built and
57
+ shipped in 2.9.8; the QA dogfood and the "why Norn, now" hook never happened. Full
58
+ narrative and how to resume:
59
+ [`Docs/random/parked_contract_drift_and_marketing.md`](Docs/random/parked_contract_drift_and_marketing.md)
60
+
61
+ ## Recently done
62
+
63
+ - **Agent Workbench Slice 5e — reporting** (2026-08-10) — **done.**
64
+ A verdict is now visible in the response panel, the canvas, the debugger, and JUnit, derived
65
+ once through `agentHopPresentation` so the surfaces cannot disagree. The load-bearing move was
66
+ putting the verdict on `AgentHopResult` rather than the step: a recording persists the trace
67
+ and never step results, so a step-only verdict vanished the moment a past run was reopened —
68
+ the round-trip is now a test. Two bugs the build surfaced: a judged hop has `success === true`
69
+ even when the verdict failed, so the canvas drew a **green node for a failed run** until the
70
+ verdict was counted explicitly; and the runner emits its `completed` event before the verdict
71
+ exists, so the deep-cloned event log needed patching or replay playback showed no verdict.
72
+ Also: judge statements now draw a sequence handoff edge from whatever produced the value they
73
+ judge, live progress is emitted as `stepType: 'judge'`, and JUnit gets one testcase per
74
+ statement with `ExpectationsUnmet` and `JudgeError` kept distinct. 13 new focused tests; 159
75
+ pass across every runnable suite, plus type-check, lint, bundles, regression, skill validation,
76
+ and a real CLI run producing JUnit.
77
+ → [`Docs/to-be-built/agent-workbench-slice-5e.md`](Docs/to-be-built/agent-workbench-slice-5e.md)
78
+ - **Agent Workbench Slice 5c — judge execution and the result contract** (2026-08-10) — **done,
79
+ one deliverable deferred.** A `judge` statement now runs its judge and produces a verdict.
80
+ The design that carries it: **the model is never asked for an overall result** — it rules on
81
+ each expectation and quotes evidence, and Norn computes `passed`, checks one ruling per
82
+ expectation with indexes in range, and verifies each quote occurs in the judged text. A judge
83
+ is an ordinary agent invocation; everything judge-specific lives in a derived definition, so
84
+ there is no judge runtime and no `if (isJudge)` in the runner. `builtInContracts` generalises
85
+ "a contract that lives in code" and rides the existing generation/validation path.
86
+ A provider or contract failure sets an error and no verdict, so "the judge said no" never
87
+ reads like "the judge never answered". **A failed judge stops the sequence**, the same as a
88
+ failed `assert` — it is the concluding check on an orchestration, so later steps must not run
89
+ on a disproved premise; the run is still recorded and replayable. A malformed reply *shape* is
90
+ corrected up to `contract_retries` via the opt-in `retryReturnsContract` (off for `run
91
+ <Agent>`); the verdict itself is never re-rolled. Evidence that cannot be found in the judged
92
+ text is flagged and still passes — strictness is intended to become per-expectation, not
93
+ global. **Deferred:** the automatic replay drift check —
94
+ `expectationsHash` is recorded and diffable, but pure replay never re-reads the `.norn`, so
95
+ the comparison needs `runLineNumber` plumbing and lands with 5d, which owns the policy.
96
+ Fixed along the way: judge hops reported false prompt drift on every replay. 18 new focused
97
+ tests; 141 pass across every runnable suite, plus type-check, lint, bundles, regression, skill
98
+ validation, and a real CLI run against a deterministic judge server.
99
+ → [`Docs/to-be-built/agent-workbench-slice-5c.md`](Docs/to-be-built/agent-workbench-slice-5c.md)
100
+ - **Agent Workbench Slice 5a — eval authoring** (2026-08-10) — **done.**
101
+ `judge <subject> with <Agent> expects "…"` in `.norn` sequences — **one line, like `assert`**,
102
+ with `expects file <path>` for a list. Built as a `judge … end judge` block first and reworked
103
+ on Peter's correction: the sequence is the orchestrator, so it stays a flat list of steps
104
+ rather than nesting a mini-language. Expectations are ordinary Norn strings, file lists resolve
105
+ at parse time with per-expectation origins, plus eleven `judge-*` diagnostics, grammar,
106
+ completions, hover, and CLI/JSON step output. Driven by Peter's answer that the first eval is a
107
+ checklist — all expectations must be met, exceeding them is not a failure. Reuse held:
108
+ expectation strings go through `agentPromptBlock.ts`'s existing escaping rather than a second
109
+ stack, one shared module serves runner/diagnostics/completions/hover, and statements validate
110
+ in the existing agent preflight so a malformed eval cannot spend a model call. A judge step
111
+ reports as explicitly *not judged* until 5c rather than silently passing. 15 new focused tests;
112
+ 123 pass across every runnable suite, plus type-check, lint, bundles, regression, and skill
113
+ validation.
114
+ → [`Docs/to-be-built/agent-workbench-slice-5a.md`](Docs/to-be-built/agent-workbench-slice-5a.md)
115
+ - **Agent Workbench Slice 4** (2026-08-09) — **done and reviewed.**
116
+ Default-on, secret-safe recordings under `.norn-cache/runs`; fixed 20-run retention;
117
+ byte-identical offline replay with zero provider/MCP side effects; strict CLI replay with JSON,
118
+ environment restoration, graph-drift failure, and live `--from`; agent frames/scopes,
119
+ breakpoints, stepping, and contract-failure navigation in the existing DAP; paused-hop canvas
120
+ highlighting and validated node-to-invocation navigation. Review: replay is a clean
121
+ `AgentProviderAdapter` with no runtime special-casing; the debugger is the existing DAP session
122
+ extended, not a second one; secrets are stable placeholders re-substituted from `.nornenv` at
123
+ replay time behind a preflight that fails naming the missing variables; and the debugger became
124
+ the third surface consuming the shared `agentHopPresentation` helper. 55 focused tests pass
125
+ across all agent suites, plus type-check, lint, and bundle build.
126
+ → [`Docs/to-be-built/agent-workbench-slice-4.md`](Docs/to-be-built/agent-workbench-slice-4.md)
127
+ - **Agent Workbench Slice 3** (2026-08-09) — **done and reviewed.**
128
+ Read-only VS Code agent canvas with parser-first static graphs, redacted live/completed trace
129
+ overlays, deterministic cycle-safe layout, first-class contract states, hover/copy payloads,
130
+ source navigation, command and CodeLens entry points, and shared response-panel presentation.
131
+ Recovery resolves successful correction invocations rather than trusting a flag. Review: the
132
+ shared `agentHopPresentation` helper is consumed by both the canvas and the response panel so
133
+ they cannot drift; the trace is redacted before anything reads it; CSP is `default-src 'none'`
134
+ with nonce'd scripts and no external hosts; no graph library was added; the panel never writes.
135
+ 43 focused tests pass across all agent suites, plus type-check, lint, and bundle build.
136
+ → [`Docs/to-be-built/agent-workbench-slice-3.md`](Docs/to-be-built/agent-workbench-slice-3.md)
137
+ - **Agent Workbench Slice 2** (2026-08-08) — **done and reviewed.**
138
+ Model-directed agent-as-tool delegation, two-way boundary correction, subtree conversations,
139
+ configurable spend guards, recursive CLI/JSON/response-panel traces, language tooling, docs,
140
+ and a deterministic violate → correct → pass demo. Review: sub-agents go through the same tool
141
+ path as MCP tools, config precedence is agent → provider → global → built-in, parallel tool use
142
+ is disabled on both adapters, and the runner was extracted to `src/agents/agentRunner.ts` rather
143
+ than growing `sequenceRunner`. 32 focused tests pass; the language suite needs the VS Code host.
144
+ `max_tokens` default stays 16000 — a real OpenAI model rejected 32000 and reported a 16384
145
+ cap, so the generous default was walked back; raise it per-provider where a model allows it.
146
+ → [`Docs/to-be-built/agent-workbench-slice-2.md`](Docs/to-be-built/agent-workbench-slice-2.md)
147
+ - **Agent Workbench Slice 1** (2026-08-08) — **done and verified running by Peter.**
148
+ `.nornagent` definitions, linear agent `run` handoffs, OpenAI/local/Anthropic/Google adapters,
149
+ MCP tools, automatic boundary contracts, live CLI + JSON traces, language tooling, focused
150
+ tests, and a runnable demo. Reviewed: reuse held (contracts go through `matchesSchema` and the
151
+ existing Contract View path), provider interface held (Google landed later as a pure new file).
152
+ → [`Docs/to-be-built/agent-workbench-slice-1.md`](Docs/to-be-built/agent-workbench-slice-1.md)
153
+ - **Website k8s de-promotion** (2026-08-07) — promo surfaces stripped and deployed; docs
154
+ and functionality untouched. Was Phase 0 of the marketing plan.
155
+
156
+ ---
157
+ _Last updated: 2026-08-10_
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Norn
2
2
 
3
- Norn keeps API tests, database checks, and Kubernetes runbooks in repeatable, version-controlled files your whole team can trust. Author, inspect, and debug them in VS Code, then run the same files from the CLI and CI.
3
+ Norn keeps API tests and database checks in repeatable, version-controlled files your whole team can trust. Author, inspect, and debug them in VS Code, then run the same files from the CLI and CI.
4
4
 
5
5
  ### Simple API Requests
6
6
 
@@ -27,7 +27,7 @@ That means you can:
27
27
  - `.nornenv` files for environments and secrets
28
28
  - `.nornapi` files for reusable endpoint definitions
29
29
  - `.nornsql` files for database queries and commands
30
- - `.nornk8s` files for Kubernetes commands and runbooks
30
+ - `.nornagent` files for contract-checked AI agents and model-directed delegation
31
31
  - syntax highlighting, IntelliSense, and diagnostics
32
32
  - response inspection, JSON diffing, and click-to-generate assertions
33
33
  - tagged and parameterized test execution in VS Code and the CLI
@@ -64,7 +64,6 @@ Use Norn to:
64
64
  - run a whole sequence from the editor
65
65
  - debug a sequence with breakpoints
66
66
  - run test sequences from the Testing view
67
- - inspect live Kubernetes pods and logs from `.nornk8s` files
68
67
 
69
68
  ## In The CLI
70
69
 
@@ -73,35 +72,8 @@ The CLI uses the same execution model as the extension, so local runs and CI run
73
72
  ```bash
74
73
  npm install -g norn-cli
75
74
  norn ./tests/smoke.norn -e dev
76
- norn ./runbooks/triage.nornk8s --context prelive
77
75
  ```
78
76
 
79
- ## Kubernetes Runbooks
80
-
81
- `.nornk8s` files turn common kubectl triage commands into reusable runbooks. Standalone
82
- `get pods`, `describe pod`, and `logs` commands open styled VS Code views with live pod
83
- and log updates. Helper sequences run explicitly, while `runbook sequence` blocks are
84
- automatic CLI entry points.
85
-
86
- ```nornk8s
87
- namespace {{ordersNs}}
88
-
89
- get pods
90
- logs orders-api-7d9f --tail 500
91
-
92
- sequence RestartOrders
93
- restart deployment/orders-api
94
- end sequence
95
-
96
- runbook sequence ClusterCheck
97
- get pods
98
- get pods -n kube-system
99
- end sequence
100
- ```
101
-
102
- Select a Kubernetes context from the VS Code context lens or pass `--context` to the
103
- CLI. Restart commands require a VS Code confirmation or CLI `--yes`.
104
-
105
77
  ## `.nornenv` Templates And Extends
106
78
 
107
79
  Use `[template:name]` sections for reusable environment building blocks, then compose selectable `[env:name]` sections with `extends`. Only templates can be extended. Templates are not selectable from the VS Code environment picker or CLI; only `[env:...]` names can be used with `-e`.
@@ -180,13 +152,163 @@ Behavior:
180
152
  - Tool `structuredContent` is validated against the MCP tool's advertised `outputSchema` when present.
181
153
  - Sessions are closed automatically when the outermost sequence finishes or fails.
182
154
 
155
+ ## Contract-Checked Agent Graphs
156
+
157
+ Define agents in an imported `.nornagent` sidecar. An agent can grant another
158
+ agent through `agents`; the callee then appears to the calling model as an
159
+ ordinary tool. Its `accepts` schema constrains and validates the generated tool
160
+ input, and its `returns` schema validates the result before it travels back up
161
+ the graph.
162
+
163
+ A `model` block declares the provider, the provider's model, and where the API key
164
+ comes from. Nothing is implied by convention: `apiKey` points at a `.nornenv`
165
+ variable, so the file says exactly which value it needs and the secret itself never
166
+ leaves the environment.
167
+
168
+ ```nornagent
169
+ model Workbench
170
+ provider openai # openai, anthropic, google, or local
171
+ name gpt-4o
172
+ apiKey {{$env.OPENAI_API_KEY}}
173
+ # baseUrl {{$env.OPENAI_BASE_URL}} # optional; required for local
174
+ end model
175
+
176
+ agent DomainExpert
177
+ model Workbench
178
+ describe "Call for domain questions and include the ticket context."
179
+ accepts contracts/domain-question.schema.json
180
+ returns contracts/domain-answer.schema.json
181
+ system "Answer only the supplied domain question."
182
+ end agent
183
+
184
+ agent TicketRouter
185
+ model Workbench
186
+ agents DomainExpert
187
+ system "Consult the domain expert when the ticket needs it."
188
+ end agent
189
+ ```
190
+
191
+ A prompt that outgrows its sidecar can live in its own file instead. `describe`
192
+ and `system` both accept `file <path>`, resolved relative to the `.nornagent`
193
+ file just like an import:
194
+
195
+ ```nornagent
196
+ agent TicketRouter
197
+ model Workbench
198
+ agents DomainExpert
199
+ system file prompts/ticket-router.md
200
+ end agent
201
+ ```
202
+
203
+ The file's text is the prompt verbatim — no escaping, so quotes and backslashes
204
+ stay as written — and `{{...}}` references in it resolve exactly as they do
205
+ inline. A missing or empty prompt file is a parse error on the directive line.
206
+
207
+ Run the graph from an ordinary sequence; the same nested contract and retry
208
+ trace appears in VS Code and the CLI:
209
+
210
+ ```norn
211
+ import "./agents.nornagent"
212
+
213
+ test sequence RouteTicket
214
+ var ticket = run readJson "./ticket.json"
215
+ var verdict = run TicketRouter ticket
216
+ assert verdict.text exists
217
+ end sequence
218
+ ```
219
+
220
+ Malformed handoffs are returned to the calling model as field-level tool errors
221
+ so it can correct them. The default guardrails are depth 5, 25 total agent
222
+ invocations, and a two-failed-attempt contract cap. Configure them globally,
223
+ per provider, or on an individual agent (most specific wins):
224
+
225
+ ```json
226
+ {
227
+ "version": 1,
228
+ "agents": {
229
+ "max_tokens": 16000,
230
+ "max_depth": 5,
231
+ "max_invocations": 25,
232
+ "contract_retries": 2,
233
+ "recording": { "enabled": true },
234
+ "providers": {
235
+ "local": { "max_tokens": 4096 }
236
+ }
237
+ }
238
+ }
239
+ ```
240
+
241
+ `max_tokens` is an output ceiling, not prepaid usage: raising it does not spend
242
+ tokens by itself. Depth, invocation, and retry limits can permit additional
243
+ model calls, so tighten those three when controlling run cost. See the runnable
244
+ [`demos/agent-workbench`](./demos/agent-workbench) example for delegation and a
245
+ visible contract correction.
246
+
247
+ Every sequence run that reaches an agent is recorded under
248
+ `.norn-cache/runs/` by default, with a rolling cap of 20 files. Recordings hold
249
+ the resolved provider request, response, tool transcript, contracts,
250
+ conversation state, and ordered hop events. Values declared as secrets in
251
+ `.nornenv` are written as stable named placeholders and restored from the
252
+ selected environment when replay starts; a missing value stops replay before a
253
+ provider or tool can run. Set `agents.recording.enabled` to `false` to disable
254
+ automatic recording.
255
+
256
+ Replay a recording without model calls, MCP calls, or other sequence side
257
+ effects using the local CLI:
258
+
259
+ ```bash
260
+ node ./dist/cli.js replay .norn-cache/runs/<recording>.json
261
+ node ./dist/cli.js replay .norn-cache/runs/<recording>.json --json
262
+ node ./dist/cli.js replay .norn-cache/runs/<recording>.json --from CriteriaComparer
263
+ ```
264
+
265
+ Pure replay reproduces the stored trace and exits non-zero for failed hops or
266
+ contracts, so a deliberately copied recording can be used as a no-key CI
267
+ fixture. `--from` accepts a unique agent name or canonical path such as
268
+ `TicketRouter[1]/CriteriaComparer[1]`; the prefix stays replayed and that hop
269
+ onward runs live against the current graph.
270
+
271
+ **Norn: Show Agent Graph**, or the CodeLens on any agent block, draws the graph
272
+ for a `.nornagent` file: who calls whom, the `accepts`/`returns` contract on each
273
+ boundary, and how each one turned out. Hover an agent or a boundary for the full
274
+ detail — models, timings, usage, prompts, payloads, contract issues.
275
+
276
+ The graph's toolbar also lists the project's recorded runs, each labelled with the
277
+ `.norn` file it came from. Pick one to see how it finished, or press **Play run**
278
+ to watch it unfold hop by hop on the canvas. Playback replays the recording's own
279
+ events, so nothing re-executes and no model is called; press **Stop** to jump to
280
+ the end. A run from a file the open one took no part in is drawn from that
281
+ recording alone — the toolbar says which file it came from, and **This file —
282
+ current view** returns you to the open file, ready for its next live run.
283
+ Recording is on by default and keeps the last 20 runs under `.norn-cache/runs/`.
284
+
285
+ To step a recording in VS Code, use the existing `norn` debugger and add the
286
+ artifact to a launch configuration:
287
+
288
+ ```json
289
+ {
290
+ "type": "norn",
291
+ "request": "launch",
292
+ "name": "Replay RouteTicket",
293
+ "file": "${workspaceFolder}/route-ticket.norn",
294
+ "sequence": "RouteTicket",
295
+ "recording": "${workspaceFolder}/fixtures/route-ticket-run.json",
296
+ "stopOnEntry": true
297
+ }
298
+ ```
299
+
300
+ Agent calls appear as nested frames with Request, Response, Contracts, and
301
+ Conversation scopes. Normal step controls navigate the recorded hop timeline;
302
+ **Norn Debug: Run to Next Contract Failure** stops on retry violations as well
303
+ as final contract failures. With Agent Graph open, the paused invocation is
304
+ highlighted and selecting a node moves the replay cursor to that invocation.
305
+
183
306
  ## Good Fit For
184
307
 
185
308
  - backend teams validating APIs during development
186
309
  - QA and automation work that needs readable test flows
187
310
  - regression and smoke suites that should run the same way locally and in CI
188
311
  - projects that want API requests and API tests to live next to the code
189
- - teams that want repeatable Kubernetes triage runbooks beside their services
190
312
 
191
313
  Diff preview test line.
192
314
 
@@ -0,0 +1,156 @@
1
+ # Agent Workbench Slice 2
2
+
3
+ This demo includes the Slice 1 linear graph and a Slice 2 model-driven graph.
4
+ `TicketRouter` decides which domain experts to invoke as tools, while Norn
5
+ contract-checks both sides of every handoff. It uses no MCP server. The checked-in
6
+ model declaration uses OpenAI, but you can swap it to Google, Anthropic, or an
7
+ OpenAI-compatible local runtime (subject to each model's tool/structured-output
8
+ support).
9
+
10
+ The checked-in [`norn.config.json`](./norn.config.json) spells out the runtime
11
+ defaults used by the demo: 16,000 output tokens for hosted providers, 4,096 for
12
+ `local`, depth 5, 25 total invocations, and 2 failed contract attempts.
13
+ Agent recording is enabled by default and keeps the latest 20 artifacts under
14
+ `.norn-cache/runs/`.
15
+
16
+ ## Run it
17
+
18
+ Build the local CLI, export a key, then run the local build:
19
+
20
+ ```bash
21
+ npm run compile
22
+ export OPENAI_API_KEY="your-key"
23
+ node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence HappyPath
24
+ ```
25
+
26
+ The variable name is not a convention Norn assumes — it is written in the model
27
+ block in [agents.nornagent](./agents.nornagent) as `apiKey {{$env.OPENAI_API_KEY}}`,
28
+ so any name works as long as the file and the environment agree.
29
+
30
+ Alternatively, store `OPENAI_API_KEY` as a secret under `[env:openai]` in a
31
+ local `.nornenv`. Norn gives the selected `.nornenv` values precedence over
32
+ process environment variables; add `--env openai` to the commands in that case.
33
+
34
+ `HappyPath` runs `TicketReader -> CriteriaComparer -> Responder`. The first two
35
+ agents produce structured output; `accepts` checks each incoming handoff and
36
+ `returns` checks each outgoing handoff. The final result remains available to
37
+ ordinary Norn assertions and interpolation through `verdict.body` and
38
+ `reply.text`.
39
+
40
+ ## Demonstrate model-driven delegation and correction
41
+
42
+ ```bash
43
+ node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence DelegatedRouting
44
+ ```
45
+
46
+ `TicketRouter` consults `FrontendExpert` and `BackendExpert`. Its prompt is the
47
+ longest in the graph, so it lives in
48
+ [`prompts/ticket-router.md`](./prompts/ticket-router.md) and is pulled in with
49
+ `system file prompts/ticket-router.md`. The file's text is the prompt verbatim,
50
+ which is why the quoted instructions there need no escaping. For the checked-in
51
+ `DEMO-201` fixture, its instructions deliberately make the first backend
52
+ question omit the contract-only `DOMAIN QUESTION:` prefix. That `pattern` is
53
+ removed from the provider's generation schema but remains in Norn's untouched
54
+ validation contract, so the calling model receives the exact field-level error,
55
+ retries with conforming input, and completes. The human trace shows the nested
56
+ agents and `retry 1/2 — model corrected the call`; `--json` preserves the same
57
+ graph under each hop's `children` array.
58
+
59
+ For a deterministic, no-key proof of the exact violate → correct → pass
60
+ trace, start the checked-in OpenAI-compatible demo server in one terminal:
61
+
62
+ ```bash
63
+ node demos/agent-workbench/fake-openai-server.js
64
+ ```
65
+
66
+ Then run the normal demo from another terminal:
67
+
68
+ ```bash
69
+ OPENAI_BASE_URL=http://127.0.0.1:18766/v1 \
70
+ OPENAI_API_KEY=demo \
71
+ node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence DelegatedRouting
72
+ ```
73
+
74
+ The fake server is only a deterministic demo fixture. Normal runs continue to
75
+ use the model declared in `agents.nornagent` and the provider credentials below.
76
+
77
+ To inspect the machine-readable run artifact:
78
+
79
+ ```bash
80
+ node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence DelegatedRouting --json
81
+ ```
82
+
83
+ After either run, replay its newest `.norn-cache/runs/*.json` artifact without
84
+ the fake server, an API key, or any model/tool side effects:
85
+
86
+ ```bash
87
+ node ./dist/cli.js replay .norn-cache/runs/<recording>.json
88
+ node ./dist/cli.js replay .norn-cache/runs/<recording>.json --from BackendExpert
89
+ ```
90
+
91
+ The first command is fully offline. The second replays everything before the
92
+ selected hop and then switches to the currently configured live provider from
93
+ that hop onward.
94
+
95
+ ## Demonstrate a broken handoff
96
+
97
+ ```bash
98
+ node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence ContractFailure
99
+ ```
100
+
101
+ The failure fixture asks `CriteriaComparer` for the summary `deliberately
102
+ invalid`, while the untouched contract permits only `aligned` or `drift found`.
103
+ The provider-generation schema omits the unsupported `pattern` constraint, but
104
+ Norn then validates the parsed output against the full schema. The run should
105
+ therefore fail at `summary`, retain the raw model text, and exit non-zero.
106
+
107
+ Model output is nondeterministic. If the model declines the deliberate bad-output
108
+ instruction, rerun the failure sequence or use the injected mock adapter in a
109
+ focused runtime test for deterministic automation.
110
+
111
+ ## Swap the model
112
+
113
+ Edit the one `model Workbench` block in
114
+ [agents.nornagent](./agents.nornagent). Every agent references the alias, so the
115
+ whole graph moves together:
116
+
117
+ ```nornagent
118
+ model Workbench
119
+ provider openai
120
+ name gpt-4o
121
+ apiKey {{$env.OPENAI_API_KEY}}
122
+ end model
123
+ ```
124
+
125
+ The other providers, with the variable names this demo assumes. `apiKey` and
126
+ `baseUrl` accept any `.nornenv` variable — these are only the conventional names:
127
+
128
+ ```nornagent
129
+ model Workbench
130
+ provider google # native Gemini Developer API, never Vertex AI
131
+ name gemini-3-flash-preview
132
+ apiKey {{$env.GOOGLE_API_KEY}}
133
+ end model
134
+
135
+ model Workbench
136
+ provider anthropic
137
+ name claude-sonnet-4-6
138
+ apiKey {{$env.ANTHROPIC_API_KEY}}
139
+ end model
140
+
141
+ model Workbench
142
+ provider local # any OpenAI-compatible runtime
143
+ name qwen3-coder
144
+ baseUrl {{$env.LOCAL_BASE_URL}} # required for local
145
+ apiKey {{$env.LOCAL_API_KEY}} # optional
146
+ end model
147
+ ```
148
+
149
+ Omitting `apiKey` falls back to each provider's conventional variable
150
+ (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`) and
151
+ then to the vendor SDK's own credential chain. Declaring it is preferred — it is
152
+ the difference between a file that documents what it needs and one that fails at
153
+ the provider call. For Google, `baseUrl` is only for a compatible proxy: leave it
154
+ unset for normal use and never paste a full `:generateContent` operation URL.
155
+
156
+ There is no CLI model override: the authored graph stays the source of truth.