norn-cli 2.9.7 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/CLAUDE.md +68 -0
- package/NOW.md +157 -0
- package/README.md +153 -31
- package/demos/agent-workbench/README.md +156 -0
- package/demos/agent-workbench/agents.nornagent +87 -0
- package/demos/agent-workbench/contracts/domain-answer.schema.json +28 -0
- package/demos/agent-workbench/contracts/domain-question.schema.json +34 -0
- package/demos/agent-workbench/contracts/router-verdict.schema.json +43 -0
- package/demos/agent-workbench/contracts/ticket.schema.json +29 -0
- package/demos/agent-workbench/contracts/verdict.schema.json +48 -0
- package/demos/agent-workbench/fake-openai-server.js +128 -0
- package/demos/agent-workbench/fixtures/aligned-ticket.json +10 -0
- package/demos/agent-workbench/fixtures/cross-domain-ticket.json +10 -0
- package/demos/agent-workbench/fixtures/invalid-output-ticket.json +8 -0
- package/demos/agent-workbench/norn.config.json +14 -0
- package/demos/agent-workbench/prompts/ticket-router.md +11 -0
- package/demos/agent-workbench/workbench.norn +32 -0
- package/dist/cli.js +184789 -111685
- package/package.json +86 -8
- package/playground/ai.norn +9 -0
- package/playground/ai_orchastration.nornagent +10 -0
- package/playground/knowedge_base/nexus_system_prompt.md +1 -0
- package/schemas/norn.config.schema.json +211 -0
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,29 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to the "Norn" extension will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [3.0.0] - 2026-08-11
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- **`.nornagent` files — the Agent Workbench** - a new file type that declares AI agents alongside your API tests. An imported `.nornagent` sidecar declares the models and agents; a `.norn` sequence owns execution order and calls them with `var reply = run <Agent> "…"`. Agents can be given MCP-backed tools, `accepts` / `returns` JSON Schema contracts that are validated on every boundary, and other agents they are allowed to call.
|
|
9
|
+
- **Agent graph canvas** - a read-only VS Code canvas for a `.nornagent` file showing who calls whom, the contract on each boundary, and how each hop turned out. Hover for models, timings, usage, prompts, payloads, and contract issues.
|
|
10
|
+
- **Run recording, replay, and stepping** - runs are recorded by default under `.norn-cache/runs/` (last 20 kept). Replay a recording to step it in the existing `norn` debugger, or play it back hop by hop on the canvas — nothing re-executes and no model is called. The graph toolbar labels each run with the `.norn` file it came from, and **This file — current view** returns you to the open file.
|
|
11
|
+
- **`judge` statements — AI evals in a sequence** - `judge <value> with <Agent> expects "…"` scores free text that an assertion cannot. The judge rules on each expectation independently and quotes its evidence; Norn computes the pass or fail, so the verdict never comes from the model. A long checklist lives in a file with `expects file <path>`, where an ordinary Markdown list works as-is. Every ruling carries a reason, and a quote that cannot be found in the judged text is flagged.
|
|
12
|
+
- **Judge results everywhere a run is reported** - verdicts appear in the response panel, on the agent graph, in the CLI summary, and in per-statement JUnit output, and travel with the run's trace and recording. A failed verdict fails the sequence and exits the CLI non-zero.
|
|
13
|
+
- **Request body variable highlighting** - a variable used as a request body is now coloured as a variable rather than as a header group reference.
|
|
14
|
+
|
|
15
|
+
### Changed
|
|
16
|
+
- **`model` blocks in `.nornagent`** - a model is declared as a block naming every input it needs:
|
|
17
|
+
|
|
18
|
+
```nornagent
|
|
19
|
+
model Workbench
|
|
20
|
+
provider openai
|
|
21
|
+
name gpt-4o
|
|
22
|
+
apiKey {{$env.OPENAI_API_KEY}}
|
|
23
|
+
end model
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
This replaces the earlier one-line `model <Alias>: <provider>/<model-id>` form, which implied both the provider/model split and an undeclared environment-variable name. `apiKey` and `baseUrl` are ordinary Norn templates resolved per run, so the file says exactly which value it needs and the secret itself stays in the environment.
|
|
27
|
+
|
|
5
28
|
## [2.9.1] - 2026-06-22
|
|
6
29
|
|
|
7
30
|
### Fixed
|
package/CLAUDE.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# Norn Extension — Claude Code instructions
|
|
2
|
+
|
|
3
|
+
## Read these first
|
|
4
|
+
|
|
5
|
+
1. **[`NOW.md`](NOW.md)** — the tool-agnostic active-work board shared by every agent and by
|
|
6
|
+
Peter. Whatever sits under **Active now** is the current priority. This board, not any
|
|
7
|
+
assistant's memory, is the source of truth for what we are working on.
|
|
8
|
+
2. **[`AGENTS.md`](AGENTS.md)** — the full repo instructions (CLI parity requirement, how to
|
|
9
|
+
run the local CLI, code style, testing). Everything in that file applies here.
|
|
10
|
+
|
|
11
|
+
## Skills: consult `.github/skills/` — always
|
|
12
|
+
|
|
13
|
+
This repo keeps its accumulated implementation knowledge in **[`.github/skills/`](.github/skills/)**,
|
|
14
|
+
maintained by every agent that works here. Treat it as required reading, not an optional
|
|
15
|
+
reference.
|
|
16
|
+
|
|
17
|
+
**Before touching any area of the codebase, check `.github/skills/` for a skill covering it
|
|
18
|
+
and read that skill.** Do this even for changes that look small — the skills exist precisely
|
|
19
|
+
because these areas have non-obvious rules that are easy to break.
|
|
20
|
+
|
|
21
|
+
Current skills and what they cover:
|
|
22
|
+
|
|
23
|
+
| Skill | Read it when working on |
|
|
24
|
+
|-------|------------------------|
|
|
25
|
+
| `norn-syntax-highlighting` | TextMate grammars in `syntaxes/`, token scopes, highlighting bugs |
|
|
26
|
+
| `norn-agent-workbench` | `.nornagent` sidecars, agent `run` steps, providers, contracts, record/replay, agent graph |
|
|
27
|
+
| `norn-intellisense-preferences` | Completion providers, IntelliSense behaviour |
|
|
28
|
+
| `norn-value-resolution` | Variables, interpolation, `.nornenv`, value precedence |
|
|
29
|
+
| `add-norn-keyword` | Adding a keyword to the `.norn` language |
|
|
30
|
+
| `add-assertion-operator` | Adding an assertion operator |
|
|
31
|
+
| `add-sequence-step-type` | Adding a new sequence step type |
|
|
32
|
+
| `add-imported-sidecar-language` | Adding a new imported sidecar file type |
|
|
33
|
+
| `norn-cli-integration` | `src/cli.ts` and CLI parity |
|
|
34
|
+
| `norn-debugger-integration` | DAP / debugger work |
|
|
35
|
+
| `norn-response-panel` | The response webview panel |
|
|
36
|
+
| `norn-test-explorer` | VS Code Test Explorer integration |
|
|
37
|
+
| `norn-sequence-tags` | Sequence tags and filtering |
|
|
38
|
+
| `norn-api-coverage` | `.nornapi` coverage features |
|
|
39
|
+
| `norn-mcp-tools` | MCP client and tool wiring |
|
|
40
|
+
| `norn-k8s` | Kubernetes features |
|
|
41
|
+
| `norn-terminal` | Norn Terminal / PTY work |
|
|
42
|
+
| `norn-env-templates` | `.nornenv` templates |
|
|
43
|
+
| `norn-publishing` | Packaging and publishing the extension |
|
|
44
|
+
| `norn-starter-catalog` | Starter templates catalog |
|
|
45
|
+
| `norn-title-cards` | Title cards |
|
|
46
|
+
| `demo-task-helper` | Building demos |
|
|
47
|
+
| `readme-sync` | Keeping README/docs in sync with features |
|
|
48
|
+
|
|
49
|
+
The table can go stale — list the directory when the work doesn't obviously map to a row.
|
|
50
|
+
|
|
51
|
+
### Maintaining skills
|
|
52
|
+
|
|
53
|
+
Per `AGENTS.md`, skills are living documents:
|
|
54
|
+
|
|
55
|
+
- If a skill is **wrong or outdated**, fix it as part of the change.
|
|
56
|
+
- If a skill is **missing** for an area you just learned, write one in Agent Skills format.
|
|
57
|
+
- After creating or editing any skill, run `npm run validate:skills`.
|
|
58
|
+
|
|
59
|
+
Capture the lesson that was hard to learn, not a restatement of the code.
|
|
60
|
+
|
|
61
|
+
## Verification
|
|
62
|
+
|
|
63
|
+
Never use `npx norn` — that runs the published package, not local changes:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
npm run compile
|
|
67
|
+
node ./dist/cli.js tests/file.norn --env prelive
|
|
68
|
+
```
|
package/NOW.md
ADDED
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
# NOW — Active Work Board
|
|
2
|
+
|
|
3
|
+
> **The single source of truth for what we are working on _right now_** — for Peter and
|
|
4
|
+
> for every AI agent (Claude, Codex, Copilot, Cursor, any). Deliberately tool-agnostic:
|
|
5
|
+
> it lives here in the repo, **not** in any one assistant's memory or private notes.
|
|
6
|
+
>
|
|
7
|
+
> We have a lot of planning docs; this board exists so nobody (human or AI) has to guess
|
|
8
|
+
> what's current. If it's on this board, it's the work. If it's not, it isn't — yet.
|
|
9
|
+
|
|
10
|
+
## How this board works
|
|
11
|
+
|
|
12
|
+
- **Empty by default.** Nothing is "current" unless it has been explicitly **Accepted**
|
|
13
|
+
onto this board by Peter. An empty "Active now" means: no directed work in flight —
|
|
14
|
+
don't start big work off a backlog/plan doc without Peter accepting it here first.
|
|
15
|
+
- **Accepting work:** Peter says "Accept X" (or moves it here). That's the signal it
|
|
16
|
+
becomes current. Until then, ideas live in the plan/backlog docs, not here.
|
|
17
|
+
- **This board is an _index_, not the detail.** Each item is one or two lines + a link to
|
|
18
|
+
its real plan doc. Keep the detail in the linked doc; keep this page short and scannable.
|
|
19
|
+
- **Finishing work:** when an item is done, move it to "Recently done" (keep the last
|
|
20
|
+
handful), then prune. Don't let this page grow.
|
|
21
|
+
- **Agents:** treat items under "Active now" as the current priority order. Do these
|
|
22
|
+
before picking up anything from the wider `Docs/` set.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Active now
|
|
27
|
+
|
|
28
|
+
### 🧪 Agent Workbench — Slice 5: AI evals
|
|
29
|
+
The last of the four things nothing else does in one place: let an **AI judge score what a
|
|
30
|
+
contract cannot**, with thresholds useful in CI rather than noisy. An umbrella, not a buildable
|
|
31
|
+
slice — it breaks into six items (authoring, rubric/scoring, judge execution, thresholds,
|
|
32
|
+
reporting, fixtures/docs), specced and built one at a time.
|
|
33
|
+
→ Spec: [`Docs/to-be-built/agent-workbench-slice-5.md`](Docs/to-be-built/agent-workbench-slice-5.md)
|
|
34
|
+
|
|
35
|
+
**Next step:** **5f — fixtures, docs, positioning**, the last item of slice 5. The feature is
|
|
36
|
+
complete and reported across every surface; what is missing is a demo anyone else can run and an
|
|
37
|
+
explanation of what it is for. 5f has no spec yet.
|
|
38
|
+
|
|
39
|
+
**5b and 5d are closed.** 5b was made unnecessary by the checklist scoring model. 5d's content
|
|
40
|
+
was decided in conversation — a failed judge stops the sequence, the CLI already exits `1`, and
|
|
41
|
+
per-expectation strictness plus the replay-drift check are future work Peter deferred
|
|
42
|
+
2026-08-10.
|
|
43
|
+
→ 5a spec (built): [`Docs/to-be-built/agent-workbench-slice-5a.md`](Docs/to-be-built/agent-workbench-slice-5a.md)
|
|
44
|
+
→ 5c spec (built): [`Docs/to-be-built/agent-workbench-slice-5c.md`](Docs/to-be-built/agent-workbench-slice-5c.md)
|
|
45
|
+
→ 5e spec (built): [`Docs/to-be-built/agent-workbench-slice-5e.md`](Docs/to-be-built/agent-workbench-slice-5e.md)
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Backlog / accepted-but-not-started
|
|
50
|
+
|
|
51
|
+
_(empty)_
|
|
52
|
+
|
|
53
|
+
## Parked
|
|
54
|
+
|
|
55
|
+
### Contract & Drift + Voiden-playbook marketing
|
|
56
|
+
The previous direction, parked 2026-08-07 — untested, not failed. Stages 1+2 are built and
|
|
57
|
+
shipped in 2.9.8; the QA dogfood and the "why Norn, now" hook never happened. Full
|
|
58
|
+
narrative and how to resume:
|
|
59
|
+
[`Docs/random/parked_contract_drift_and_marketing.md`](Docs/random/parked_contract_drift_and_marketing.md)
|
|
60
|
+
|
|
61
|
+
## Recently done
|
|
62
|
+
|
|
63
|
+
- **Agent Workbench Slice 5e — reporting** (2026-08-10) — **done.**
|
|
64
|
+
A verdict is now visible in the response panel, the canvas, the debugger, and JUnit, derived
|
|
65
|
+
once through `agentHopPresentation` so the surfaces cannot disagree. The load-bearing move was
|
|
66
|
+
putting the verdict on `AgentHopResult` rather than the step: a recording persists the trace
|
|
67
|
+
and never step results, so a step-only verdict vanished the moment a past run was reopened —
|
|
68
|
+
the round-trip is now a test. Two bugs the build surfaced: a judged hop has `success === true`
|
|
69
|
+
even when the verdict failed, so the canvas drew a **green node for a failed run** until the
|
|
70
|
+
verdict was counted explicitly; and the runner emits its `completed` event before the verdict
|
|
71
|
+
exists, so the deep-cloned event log needed patching or replay playback showed no verdict.
|
|
72
|
+
Also: judge statements now draw a sequence handoff edge from whatever produced the value they
|
|
73
|
+
judge, live progress is emitted as `stepType: 'judge'`, and JUnit gets one testcase per
|
|
74
|
+
statement with `ExpectationsUnmet` and `JudgeError` kept distinct. 13 new focused tests; 159
|
|
75
|
+
pass across every runnable suite, plus type-check, lint, bundles, regression, skill validation,
|
|
76
|
+
and a real CLI run producing JUnit.
|
|
77
|
+
→ [`Docs/to-be-built/agent-workbench-slice-5e.md`](Docs/to-be-built/agent-workbench-slice-5e.md)
|
|
78
|
+
- **Agent Workbench Slice 5c — judge execution and the result contract** (2026-08-10) — **done,
|
|
79
|
+
one deliverable deferred.** A `judge` statement now runs its judge and produces a verdict.
|
|
80
|
+
The design that carries it: **the model is never asked for an overall result** — it rules on
|
|
81
|
+
each expectation and quotes evidence, and Norn computes `passed`, checks one ruling per
|
|
82
|
+
expectation with indexes in range, and verifies each quote occurs in the judged text. A judge
|
|
83
|
+
is an ordinary agent invocation; everything judge-specific lives in a derived definition, so
|
|
84
|
+
there is no judge runtime and no `if (isJudge)` in the runner. `builtInContracts` generalises
|
|
85
|
+
"a contract that lives in code" and rides the existing generation/validation path.
|
|
86
|
+
A provider or contract failure sets an error and no verdict, so "the judge said no" never
|
|
87
|
+
reads like "the judge never answered". **A failed judge stops the sequence**, the same as a
|
|
88
|
+
failed `assert` — it is the concluding check on an orchestration, so later steps must not run
|
|
89
|
+
on a disproved premise; the run is still recorded and replayable. A malformed reply *shape* is
|
|
90
|
+
corrected up to `contract_retries` via the opt-in `retryReturnsContract` (off for `run
|
|
91
|
+
<Agent>`); the verdict itself is never re-rolled. Evidence that cannot be found in the judged
|
|
92
|
+
text is flagged and still passes — strictness is intended to become per-expectation, not
|
|
93
|
+
global. **Deferred:** the automatic replay drift check —
|
|
94
|
+
`expectationsHash` is recorded and diffable, but pure replay never re-reads the `.norn`, so
|
|
95
|
+
the comparison needs `runLineNumber` plumbing and lands with 5d, which owns the policy.
|
|
96
|
+
Fixed along the way: judge hops reported false prompt drift on every replay. 18 new focused
|
|
97
|
+
tests; 141 pass across every runnable suite, plus type-check, lint, bundles, regression, skill
|
|
98
|
+
validation, and a real CLI run against a deterministic judge server.
|
|
99
|
+
→ [`Docs/to-be-built/agent-workbench-slice-5c.md`](Docs/to-be-built/agent-workbench-slice-5c.md)
|
|
100
|
+
- **Agent Workbench Slice 5a — eval authoring** (2026-08-10) — **done.**
|
|
101
|
+
`judge <subject> with <Agent> expects "…"` in `.norn` sequences — **one line, like `assert`**,
|
|
102
|
+
with `expects file <path>` for a list. Built as a `judge … end judge` block first and reworked
|
|
103
|
+
on Peter's correction: the sequence is the orchestrator, so it stays a flat list of steps
|
|
104
|
+
rather than nesting a mini-language. Expectations are ordinary Norn strings, file lists resolve
|
|
105
|
+
at parse time with per-expectation origins, plus eleven `judge-*` diagnostics, grammar,
|
|
106
|
+
completions, hover, and CLI/JSON step output. Driven by Peter's answer that the first eval is a
|
|
107
|
+
checklist — all expectations must be met, exceeding them is not a failure. Reuse held:
|
|
108
|
+
expectation strings go through `agentPromptBlock.ts`'s existing escaping rather than a second
|
|
109
|
+
stack, one shared module serves runner/diagnostics/completions/hover, and statements validate
|
|
110
|
+
in the existing agent preflight so a malformed eval cannot spend a model call. A judge step
|
|
111
|
+
reports as explicitly *not judged* until 5c rather than silently passing. 15 new focused tests;
|
|
112
|
+
123 pass across every runnable suite, plus type-check, lint, bundles, regression, and skill
|
|
113
|
+
validation.
|
|
114
|
+
→ [`Docs/to-be-built/agent-workbench-slice-5a.md`](Docs/to-be-built/agent-workbench-slice-5a.md)
|
|
115
|
+
- **Agent Workbench Slice 4** (2026-08-09) — **done and reviewed.**
|
|
116
|
+
Default-on, secret-safe recordings under `.norn-cache/runs`; fixed 20-run retention;
|
|
117
|
+
byte-identical offline replay with zero provider/MCP side effects; strict CLI replay with JSON,
|
|
118
|
+
environment restoration, graph-drift failure, and live `--from`; agent frames/scopes,
|
|
119
|
+
breakpoints, stepping, and contract-failure navigation in the existing DAP; paused-hop canvas
|
|
120
|
+
highlighting and validated node-to-invocation navigation. Review: replay is a clean
|
|
121
|
+
`AgentProviderAdapter` with no runtime special-casing; the debugger is the existing DAP session
|
|
122
|
+
extended, not a second one; secrets are stable placeholders re-substituted from `.nornenv` at
|
|
123
|
+
replay time behind a preflight that fails naming the missing variables; and the debugger became
|
|
124
|
+
the third surface consuming the shared `agentHopPresentation` helper. 55 focused tests pass
|
|
125
|
+
across all agent suites, plus type-check, lint, and bundle build.
|
|
126
|
+
→ [`Docs/to-be-built/agent-workbench-slice-4.md`](Docs/to-be-built/agent-workbench-slice-4.md)
|
|
127
|
+
- **Agent Workbench Slice 3** (2026-08-09) — **done and reviewed.**
|
|
128
|
+
Read-only VS Code agent canvas with parser-first static graphs, redacted live/completed trace
|
|
129
|
+
overlays, deterministic cycle-safe layout, first-class contract states, hover/copy payloads,
|
|
130
|
+
source navigation, command and CodeLens entry points, and shared response-panel presentation.
|
|
131
|
+
Recovery resolves successful correction invocations rather than trusting a flag. Review: the
|
|
132
|
+
shared `agentHopPresentation` helper is consumed by both the canvas and the response panel so
|
|
133
|
+
they cannot drift; the trace is redacted before anything reads it; CSP is `default-src 'none'`
|
|
134
|
+
with nonce'd scripts and no external hosts; no graph library was added; the panel never writes.
|
|
135
|
+
43 focused tests pass across all agent suites, plus type-check, lint, and bundle build.
|
|
136
|
+
→ [`Docs/to-be-built/agent-workbench-slice-3.md`](Docs/to-be-built/agent-workbench-slice-3.md)
|
|
137
|
+
- **Agent Workbench Slice 2** (2026-08-08) — **done and reviewed.**
|
|
138
|
+
Model-directed agent-as-tool delegation, two-way boundary correction, subtree conversations,
|
|
139
|
+
configurable spend guards, recursive CLI/JSON/response-panel traces, language tooling, docs,
|
|
140
|
+
and a deterministic violate → correct → pass demo. Review: sub-agents go through the same tool
|
|
141
|
+
path as MCP tools, config precedence is agent → provider → global → built-in, parallel tool use
|
|
142
|
+
is disabled on both adapters, and the runner was extracted to `src/agents/agentRunner.ts` rather
|
|
143
|
+
than growing `sequenceRunner`. 32 focused tests pass; the language suite needs the VS Code host.
|
|
144
|
+
`max_tokens` default stays 16000 — a real OpenAI model rejected 32000 and reported a 16384
|
|
145
|
+
cap, so the generous default was walked back; raise it per-provider where a model allows it.
|
|
146
|
+
→ [`Docs/to-be-built/agent-workbench-slice-2.md`](Docs/to-be-built/agent-workbench-slice-2.md)
|
|
147
|
+
- **Agent Workbench Slice 1** (2026-08-08) — **done and verified running by Peter.**
|
|
148
|
+
`.nornagent` definitions, linear agent `run` handoffs, OpenAI/local/Anthropic/Google adapters,
|
|
149
|
+
MCP tools, automatic boundary contracts, live CLI + JSON traces, language tooling, focused
|
|
150
|
+
tests, and a runnable demo. Reviewed: reuse held (contracts go through `matchesSchema` and the
|
|
151
|
+
existing Contract View path), provider interface held (Google landed later as a pure new file).
|
|
152
|
+
→ [`Docs/to-be-built/agent-workbench-slice-1.md`](Docs/to-be-built/agent-workbench-slice-1.md)
|
|
153
|
+
- **Website k8s de-promotion** (2026-08-07) — promo surfaces stripped and deployed; docs
|
|
154
|
+
and functionality untouched. Was Phase 0 of the marketing plan.
|
|
155
|
+
|
|
156
|
+
---
|
|
157
|
+
_Last updated: 2026-08-10_
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Norn
|
|
2
2
|
|
|
3
|
-
Norn keeps API tests
|
|
3
|
+
Norn keeps API tests and database checks in repeatable, version-controlled files your whole team can trust. Author, inspect, and debug them in VS Code, then run the same files from the CLI and CI.
|
|
4
4
|
|
|
5
5
|
### Simple API Requests
|
|
6
6
|
|
|
@@ -27,7 +27,7 @@ That means you can:
|
|
|
27
27
|
- `.nornenv` files for environments and secrets
|
|
28
28
|
- `.nornapi` files for reusable endpoint definitions
|
|
29
29
|
- `.nornsql` files for database queries and commands
|
|
30
|
-
- `.
|
|
30
|
+
- `.nornagent` files for contract-checked AI agents and model-directed delegation
|
|
31
31
|
- syntax highlighting, IntelliSense, and diagnostics
|
|
32
32
|
- response inspection, JSON diffing, and click-to-generate assertions
|
|
33
33
|
- tagged and parameterized test execution in VS Code and the CLI
|
|
@@ -64,7 +64,6 @@ Use Norn to:
|
|
|
64
64
|
- run a whole sequence from the editor
|
|
65
65
|
- debug a sequence with breakpoints
|
|
66
66
|
- run test sequences from the Testing view
|
|
67
|
-
- inspect live Kubernetes pods and logs from `.nornk8s` files
|
|
68
67
|
|
|
69
68
|
## In The CLI
|
|
70
69
|
|
|
@@ -73,35 +72,8 @@ The CLI uses the same execution model as the extension, so local runs and CI run
|
|
|
73
72
|
```bash
|
|
74
73
|
npm install -g norn-cli
|
|
75
74
|
norn ./tests/smoke.norn -e dev
|
|
76
|
-
norn ./runbooks/triage.nornk8s --context prelive
|
|
77
75
|
```
|
|
78
76
|
|
|
79
|
-
## Kubernetes Runbooks
|
|
80
|
-
|
|
81
|
-
`.nornk8s` files turn common kubectl triage commands into reusable runbooks. Standalone
|
|
82
|
-
`get pods`, `describe pod`, and `logs` commands open styled VS Code views with live pod
|
|
83
|
-
and log updates. Helper sequences run explicitly, while `runbook sequence` blocks are
|
|
84
|
-
automatic CLI entry points.
|
|
85
|
-
|
|
86
|
-
```nornk8s
|
|
87
|
-
namespace {{ordersNs}}
|
|
88
|
-
|
|
89
|
-
get pods
|
|
90
|
-
logs orders-api-7d9f --tail 500
|
|
91
|
-
|
|
92
|
-
sequence RestartOrders
|
|
93
|
-
restart deployment/orders-api
|
|
94
|
-
end sequence
|
|
95
|
-
|
|
96
|
-
runbook sequence ClusterCheck
|
|
97
|
-
get pods
|
|
98
|
-
get pods -n kube-system
|
|
99
|
-
end sequence
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
Select a Kubernetes context from the VS Code context lens or pass `--context` to the
|
|
103
|
-
CLI. Restart commands require a VS Code confirmation or CLI `--yes`.
|
|
104
|
-
|
|
105
77
|
## `.nornenv` Templates And Extends
|
|
106
78
|
|
|
107
79
|
Use `[template:name]` sections for reusable environment building blocks, then compose selectable `[env:name]` sections with `extends`. Only templates can be extended. Templates are not selectable from the VS Code environment picker or CLI; only `[env:...]` names can be used with `-e`.
|
|
@@ -180,13 +152,163 @@ Behavior:
|
|
|
180
152
|
- Tool `structuredContent` is validated against the MCP tool's advertised `outputSchema` when present.
|
|
181
153
|
- Sessions are closed automatically when the outermost sequence finishes or fails.
|
|
182
154
|
|
|
155
|
+
## Contract-Checked Agent Graphs
|
|
156
|
+
|
|
157
|
+
Define agents in an imported `.nornagent` sidecar. An agent can grant another
|
|
158
|
+
agent through `agents`; the callee then appears to the calling model as an
|
|
159
|
+
ordinary tool. Its `accepts` schema constrains and validates the generated tool
|
|
160
|
+
input, and its `returns` schema validates the result before it travels back up
|
|
161
|
+
the graph.
|
|
162
|
+
|
|
163
|
+
A `model` block declares the provider, the provider's model, and where the API key
|
|
164
|
+
comes from. Nothing is implied by convention: `apiKey` points at a `.nornenv`
|
|
165
|
+
variable, so the file says exactly which value it needs and the secret itself never
|
|
166
|
+
leaves the environment.
|
|
167
|
+
|
|
168
|
+
```nornagent
|
|
169
|
+
model Workbench
|
|
170
|
+
provider openai # openai, anthropic, google, or local
|
|
171
|
+
name gpt-4o
|
|
172
|
+
apiKey {{$env.OPENAI_API_KEY}}
|
|
173
|
+
# baseUrl {{$env.OPENAI_BASE_URL}} # optional; required for local
|
|
174
|
+
end model
|
|
175
|
+
|
|
176
|
+
agent DomainExpert
|
|
177
|
+
model Workbench
|
|
178
|
+
describe "Call for domain questions and include the ticket context."
|
|
179
|
+
accepts contracts/domain-question.schema.json
|
|
180
|
+
returns contracts/domain-answer.schema.json
|
|
181
|
+
system "Answer only the supplied domain question."
|
|
182
|
+
end agent
|
|
183
|
+
|
|
184
|
+
agent TicketRouter
|
|
185
|
+
model Workbench
|
|
186
|
+
agents DomainExpert
|
|
187
|
+
system "Consult the domain expert when the ticket needs it."
|
|
188
|
+
end agent
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
A prompt that outgrows its sidecar can live in its own file instead. `describe`
|
|
192
|
+
and `system` both accept `file <path>`, resolved relative to the `.nornagent`
|
|
193
|
+
file just like an import:
|
|
194
|
+
|
|
195
|
+
```nornagent
|
|
196
|
+
agent TicketRouter
|
|
197
|
+
model Workbench
|
|
198
|
+
agents DomainExpert
|
|
199
|
+
system file prompts/ticket-router.md
|
|
200
|
+
end agent
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
The file's text is the prompt verbatim — no escaping, so quotes and backslashes
|
|
204
|
+
stay as written — and `{{...}}` references in it resolve exactly as they do
|
|
205
|
+
inline. A missing or empty prompt file is a parse error on the directive line.
|
|
206
|
+
|
|
207
|
+
Run the graph from an ordinary sequence; the same nested contract and retry
|
|
208
|
+
trace appears in VS Code and the CLI:
|
|
209
|
+
|
|
210
|
+
```norn
|
|
211
|
+
import "./agents.nornagent"
|
|
212
|
+
|
|
213
|
+
test sequence RouteTicket
|
|
214
|
+
var ticket = run readJson "./ticket.json"
|
|
215
|
+
var verdict = run TicketRouter ticket
|
|
216
|
+
assert verdict.text exists
|
|
217
|
+
end sequence
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Malformed handoffs are returned to the calling model as field-level tool errors
|
|
221
|
+
so it can correct them. The default guardrails are depth 5, 25 total agent
|
|
222
|
+
invocations, and a two-failed-attempt contract cap. Configure them globally,
|
|
223
|
+
per provider, or on an individual agent (most specific wins):
|
|
224
|
+
|
|
225
|
+
```json
|
|
226
|
+
{
|
|
227
|
+
"version": 1,
|
|
228
|
+
"agents": {
|
|
229
|
+
"max_tokens": 16000,
|
|
230
|
+
"max_depth": 5,
|
|
231
|
+
"max_invocations": 25,
|
|
232
|
+
"contract_retries": 2,
|
|
233
|
+
"recording": { "enabled": true },
|
|
234
|
+
"providers": {
|
|
235
|
+
"local": { "max_tokens": 4096 }
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
`max_tokens` is an output ceiling, not prepaid usage: raising it does not spend
|
|
242
|
+
tokens by itself. Depth, invocation, and retry limits can permit additional
|
|
243
|
+
model calls, so tighten those three when controlling run cost. See the runnable
|
|
244
|
+
[`demos/agent-workbench`](./demos/agent-workbench) example for delegation and a
|
|
245
|
+
visible contract correction.
|
|
246
|
+
|
|
247
|
+
Every sequence run that reaches an agent is recorded under
|
|
248
|
+
`.norn-cache/runs/` by default, with a rolling cap of 20 files. Recordings hold
|
|
249
|
+
the resolved provider request, response, tool transcript, contracts,
|
|
250
|
+
conversation state, and ordered hop events. Values declared as secrets in
|
|
251
|
+
`.nornenv` are written as stable named placeholders and restored from the
|
|
252
|
+
selected environment when replay starts; a missing value stops replay before a
|
|
253
|
+
provider or tool can run. Set `agents.recording.enabled` to `false` to disable
|
|
254
|
+
automatic recording.
|
|
255
|
+
|
|
256
|
+
Replay a recording without model calls, MCP calls, or other sequence side
|
|
257
|
+
effects using the local CLI:
|
|
258
|
+
|
|
259
|
+
```bash
|
|
260
|
+
node ./dist/cli.js replay .norn-cache/runs/<recording>.json
|
|
261
|
+
node ./dist/cli.js replay .norn-cache/runs/<recording>.json --json
|
|
262
|
+
node ./dist/cli.js replay .norn-cache/runs/<recording>.json --from CriteriaComparer
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
Pure replay reproduces the stored trace and exits non-zero for failed hops or
|
|
266
|
+
contracts, so a deliberately copied recording can be used as a no-key CI
|
|
267
|
+
fixture. `--from` accepts a unique agent name or canonical path such as
|
|
268
|
+
`TicketRouter[1]/CriteriaComparer[1]`; the prefix stays replayed and that hop
|
|
269
|
+
onward runs live against the current graph.
|
|
270
|
+
|
|
271
|
+
**Norn: Show Agent Graph**, or the CodeLens on any agent block, draws the graph
|
|
272
|
+
for a `.nornagent` file: who calls whom, the `accepts`/`returns` contract on each
|
|
273
|
+
boundary, and how each one turned out. Hover an agent or a boundary for the full
|
|
274
|
+
detail — models, timings, usage, prompts, payloads, contract issues.
|
|
275
|
+
|
|
276
|
+
The graph's toolbar also lists the project's recorded runs, each labelled with the
|
|
277
|
+
`.norn` file it came from. Pick one to see how it finished, or press **Play run**
|
|
278
|
+
to watch it unfold hop by hop on the canvas. Playback replays the recording's own
|
|
279
|
+
events, so nothing re-executes and no model is called; press **Stop** to jump to
|
|
280
|
+
the end. A run from a file the open one took no part in is drawn from that
|
|
281
|
+
recording alone — the toolbar says which file it came from, and **This file —
|
|
282
|
+
current view** returns you to the open file, ready for its next live run.
|
|
283
|
+
Recording is on by default and keeps the last 20 runs under `.norn-cache/runs/`.
|
|
284
|
+
|
|
285
|
+
To step a recording in VS Code, use the existing `norn` debugger and add the
|
|
286
|
+
artifact to a launch configuration:
|
|
287
|
+
|
|
288
|
+
```json
|
|
289
|
+
{
|
|
290
|
+
"type": "norn",
|
|
291
|
+
"request": "launch",
|
|
292
|
+
"name": "Replay RouteTicket",
|
|
293
|
+
"file": "${workspaceFolder}/route-ticket.norn",
|
|
294
|
+
"sequence": "RouteTicket",
|
|
295
|
+
"recording": "${workspaceFolder}/fixtures/route-ticket-run.json",
|
|
296
|
+
"stopOnEntry": true
|
|
297
|
+
}
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
Agent calls appear as nested frames with Request, Response, Contracts, and
|
|
301
|
+
Conversation scopes. Normal step controls navigate the recorded hop timeline;
|
|
302
|
+
**Norn Debug: Run to Next Contract Failure** stops on retry violations as well
|
|
303
|
+
as final contract failures. With Agent Graph open, the paused invocation is
|
|
304
|
+
highlighted and selecting a node moves the replay cursor to that invocation.
|
|
305
|
+
|
|
183
306
|
## Good Fit For
|
|
184
307
|
|
|
185
308
|
- backend teams validating APIs during development
|
|
186
309
|
- QA and automation work that needs readable test flows
|
|
187
310
|
- regression and smoke suites that should run the same way locally and in CI
|
|
188
311
|
- projects that want API requests and API tests to live next to the code
|
|
189
|
-
- teams that want repeatable Kubernetes triage runbooks beside their services
|
|
190
312
|
|
|
191
313
|
Diff preview test line.
|
|
192
314
|
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Agent Workbench Slice 2
|
|
2
|
+
|
|
3
|
+
This demo includes the Slice 1 linear graph and a Slice 2 model-driven graph.
|
|
4
|
+
`TicketRouter` decides which domain experts to invoke as tools, while Norn
|
|
5
|
+
contract-checks both sides of every handoff. It uses no MCP server. The checked-in
|
|
6
|
+
model declaration uses OpenAI, but you can swap it to Google, Anthropic, or an
|
|
7
|
+
OpenAI-compatible local runtime (subject to each model's tool/structured-output
|
|
8
|
+
support).
|
|
9
|
+
|
|
10
|
+
The checked-in [`norn.config.json`](./norn.config.json) spells out the runtime
|
|
11
|
+
defaults used by the demo: 16,000 output tokens for hosted providers, 4,096 for
|
|
12
|
+
`local`, depth 5, 25 total invocations, and 2 failed contract attempts.
|
|
13
|
+
Agent recording is enabled by default and keeps the latest 20 artifacts under
|
|
14
|
+
`.norn-cache/runs/`.
|
|
15
|
+
|
|
16
|
+
## Run it
|
|
17
|
+
|
|
18
|
+
Build the local CLI, export a key, then run the local build:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npm run compile
|
|
22
|
+
export OPENAI_API_KEY="your-key"
|
|
23
|
+
node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence HappyPath
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The variable name is not a convention Norn assumes — it is written in the model
|
|
27
|
+
block in [agents.nornagent](./agents.nornagent) as `apiKey {{$env.OPENAI_API_KEY}}`,
|
|
28
|
+
so any name works as long as the file and the environment agree.
|
|
29
|
+
|
|
30
|
+
Alternatively, store `OPENAI_API_KEY` as a secret under `[env:openai]` in a
|
|
31
|
+
local `.nornenv`. Norn gives the selected `.nornenv` values precedence over
|
|
32
|
+
process environment variables; add `--env openai` to the commands in that case.
|
|
33
|
+
|
|
34
|
+
`HappyPath` runs `TicketReader -> CriteriaComparer -> Responder`. The first two
|
|
35
|
+
agents produce structured output; `accepts` checks each incoming handoff and
|
|
36
|
+
`returns` checks each outgoing handoff. The final result remains available to
|
|
37
|
+
ordinary Norn assertions and interpolation through `verdict.body` and
|
|
38
|
+
`reply.text`.
|
|
39
|
+
|
|
40
|
+
## Demonstrate model-driven delegation and correction
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence DelegatedRouting
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
`TicketRouter` consults `FrontendExpert` and `BackendExpert`. Its prompt is the
|
|
47
|
+
longest in the graph, so it lives in
|
|
48
|
+
[`prompts/ticket-router.md`](./prompts/ticket-router.md) and is pulled in with
|
|
49
|
+
`system file prompts/ticket-router.md`. The file's text is the prompt verbatim,
|
|
50
|
+
which is why the quoted instructions there need no escaping. For the checked-in
|
|
51
|
+
`DEMO-201` fixture, its instructions deliberately make the first backend
|
|
52
|
+
question omit the contract-only `DOMAIN QUESTION:` prefix. That `pattern` is
|
|
53
|
+
removed from the provider's generation schema but remains in Norn's untouched
|
|
54
|
+
validation contract, so the calling model receives the exact field-level error,
|
|
55
|
+
retries with conforming input, and completes. The human trace shows the nested
|
|
56
|
+
agents and `retry 1/2 — model corrected the call`; `--json` preserves the same
|
|
57
|
+
graph under each hop's `children` array.
|
|
58
|
+
|
|
59
|
+
For a deterministic, no-key proof of the exact violate → correct → pass
|
|
60
|
+
trace, start the checked-in OpenAI-compatible demo server in one terminal:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
node demos/agent-workbench/fake-openai-server.js
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Then run the normal demo from another terminal:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
OPENAI_BASE_URL=http://127.0.0.1:18766/v1 \
|
|
70
|
+
OPENAI_API_KEY=demo \
|
|
71
|
+
node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence DelegatedRouting
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
The fake server is only a deterministic demo fixture. Normal runs continue to
|
|
75
|
+
use the model declared in `agents.nornagent` and the provider credentials below.
|
|
76
|
+
|
|
77
|
+
To inspect the machine-readable run artifact:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence DelegatedRouting --json
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
After either run, replay its newest `.norn-cache/runs/*.json` artifact without
|
|
84
|
+
the fake server, an API key, or any model/tool side effects:
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
node ./dist/cli.js replay .norn-cache/runs/<recording>.json
|
|
88
|
+
node ./dist/cli.js replay .norn-cache/runs/<recording>.json --from BackendExpert
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
The first command is fully offline. The second replays everything before the
|
|
92
|
+
selected hop and then switches to the currently configured live provider from
|
|
93
|
+
that hop onward.
|
|
94
|
+
|
|
95
|
+
## Demonstrate a broken handoff
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
node ./dist/cli.js demos/agent-workbench/workbench.norn --sequence ContractFailure
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
The failure fixture asks `CriteriaComparer` for the summary `deliberately
|
|
102
|
+
invalid`, while the untouched contract permits only `aligned` or `drift found`.
|
|
103
|
+
The provider-generation schema omits the unsupported `pattern` constraint, but
|
|
104
|
+
Norn then validates the parsed output against the full schema. The run should
|
|
105
|
+
therefore fail at `summary`, retain the raw model text, and exit non-zero.
|
|
106
|
+
|
|
107
|
+
Model output is nondeterministic. If the model declines the deliberate bad-output
|
|
108
|
+
instruction, rerun the failure sequence or use the injected mock adapter in a
|
|
109
|
+
focused runtime test for deterministic automation.
|
|
110
|
+
|
|
111
|
+
## Swap the model
|
|
112
|
+
|
|
113
|
+
Edit the one `model Workbench` block in
|
|
114
|
+
[agents.nornagent](./agents.nornagent). Every agent references the alias, so the
|
|
115
|
+
whole graph moves together:
|
|
116
|
+
|
|
117
|
+
```nornagent
|
|
118
|
+
model Workbench
|
|
119
|
+
provider openai
|
|
120
|
+
name gpt-4o
|
|
121
|
+
apiKey {{$env.OPENAI_API_KEY}}
|
|
122
|
+
end model
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
The other providers, with the variable names this demo assumes. `apiKey` and
|
|
126
|
+
`baseUrl` accept any `.nornenv` variable — these are only the conventional names:
|
|
127
|
+
|
|
128
|
+
```nornagent
|
|
129
|
+
model Workbench
|
|
130
|
+
provider google # native Gemini Developer API, never Vertex AI
|
|
131
|
+
name gemini-3-flash-preview
|
|
132
|
+
apiKey {{$env.GOOGLE_API_KEY}}
|
|
133
|
+
end model
|
|
134
|
+
|
|
135
|
+
model Workbench
|
|
136
|
+
provider anthropic
|
|
137
|
+
name claude-sonnet-4-6
|
|
138
|
+
apiKey {{$env.ANTHROPIC_API_KEY}}
|
|
139
|
+
end model
|
|
140
|
+
|
|
141
|
+
model Workbench
|
|
142
|
+
provider local # any OpenAI-compatible runtime
|
|
143
|
+
name qwen3-coder
|
|
144
|
+
baseUrl {{$env.LOCAL_BASE_URL}} # required for local
|
|
145
|
+
apiKey {{$env.LOCAL_API_KEY}} # optional
|
|
146
|
+
end model
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
Omitting `apiKey` falls back to each provider's conventional variable
|
|
150
|
+
(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`) and
|
|
151
|
+
then to the vendor SDK's own credential chain. Declaring it is preferred — it is
|
|
152
|
+
the difference between a file that documents what it needs and one that fails at
|
|
153
|
+
the provider call. For Google, `baseUrl` is only for a compatible proxy: leave it
|
|
154
|
+
unset for normal use and never paste a full `:generateContent` operation URL.
|
|
155
|
+
|
|
156
|
+
There is no CLI model override: the authored graph stays the source of truth.
|