@cairnkeep/cli 2.21.1 → 2.22.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/CHANGELOG.md +18 -0
  2. package/docs/README.md +2 -0
  3. package/docs/agent-memory-protocol.md +146 -0
  4. package/docs/agents.md +7 -0
  5. package/docs/learning/CURRICULUM-MAP.md +3 -2
  6. package/docs/learning/FEATURE-GUIDE.md +7 -1
  7. package/docs/learning/lessons/L00-why-cairnkeep.md +1 -1
  8. package/docs/learning/lessons/L01-safe-trial.md +1 -1
  9. package/docs/learning/lessons/L02-installation.md +1 -1
  10. package/docs/learning/lessons/L03-first-project.md +1 -1
  11. package/docs/learning/lessons/L16-evaluation.md +23 -0
  12. package/docs/learning/lessons/L19-validated-skill-improvement.md +1 -1
  13. package/docs/learning/lessons/L20-mcp-tool-profiles.md +1 -1
  14. package/docs/learning/lessons/L21-context-packs.md +1 -1
  15. package/docs/learning/lessons/L22-native-windows.md +1 -1
  16. package/docs/learning/lessons/L23-guided-setup.md +1 -1
  17. package/docs/learning/lessons/L24-okf-exchange.md +1 -1
  18. package/docs/learning/lessons/L25-playbooks.md +1 -1
  19. package/docs/learning/lessons/L26-context-intelligence.md +1 -1
  20. package/docs/learning/lessons/L27-ecosystem-evidence.md +1 -1
  21. package/docs/learning/lessons/L28-security-assurance.md +2 -2
  22. package/docs/learning/tracks/evidence-and-evaluation.md +4 -0
  23. package/docs/learning/video-scripts/V16-memory-protocol-audit.md +32 -0
  24. package/docs/operating.md +8 -0
  25. package/docs/privacy-and-data-flow.md +9 -0
  26. package/docs/research/release-ledger-task.md +36 -0
  27. package/examples/eval/README.md +13 -0
  28. package/examples/eval/bundled-fake.json +1 -1
  29. package/examples/eval/protocol-codex.jsonl +5 -0
  30. package/examples/eval/protocol-narration.json +12 -0
  31. package/examples/eval/protocol-project-search.json +13 -0
  32. package/mcp-memory-server/dist/eval-cli.js +25 -2
  33. package/mcp-memory-server/dist/eval-codex.js +163 -0
  34. package/mcp-memory-server/dist/eval-protocol.js +200 -0
  35. package/mcp-memory-server/package.json +2 -0
  36. package/package.json +1 -1
  37. package/scripts/cairn-cli.mjs +1 -1
  38. package/scripts/completion.sh +9 -4
  39. package/scripts/spikes/verify-release-ledger.mjs +46 -0
  40. package/scripts/test-completion.sh +7 -3
  41. package/scripts/test-ecosystem-spikes.sh +1 -0
  42. package/scripts/test-package-install.sh +24 -1
  43. package/scripts/windows-platform.mjs +2 -1
package/CHANGELOG.md CHANGED
@@ -5,6 +5,24 @@ All notable user-facing changes are documented here. This project follows
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [2.22.0] - 2026-10-07
9
+
10
+ ### Added
11
+
12
+ - Add explicit `eval protocol --codex-jsonl PATH` for single-turn native Codex
13
+ exec exports, without automatic capture or changes to persisted trajectories.
14
+ Correlate MCP lifecycles conservatively; unknown or incomplete events cannot
15
+ pass. Include offline controls, installed-package tests, completions and
16
+ learning guidance; task correctness remains independently verified.
17
+ - Add default-off `cairn eval protocol` to audit direct memory tool use in
18
+ existing normalized Claude Code, OpenCode and Pi trajectories. Reports
19
+ retrieval ordering, interpretable results and direct mutation attempts with
20
+ conservative pass/fail/inconclusive states, bounded local reads and
21
+ payload-free output. Capture authorization is a caller assertion, not an
22
+ authenticated grant; task quality and indirect shell effects are not certified.
23
+ - Add offline positive/narration controls, packaged CLI verification, shell
24
+ completions, operating/privacy guidance and the L16/video learning supplement.
25
+
8
26
  ## [2.21.1] - 2026-10-07
9
27
 
10
28
  ### Fixed
package/docs/README.md CHANGED
@@ -18,6 +18,8 @@ linked guides rather than the project landing page.
18
18
 
19
19
  - [Cairnkeep for coding agents](agents.md) explains retrieval-first behavior,
20
20
  memory authority, playbooks, approvals, and graceful fallback.
21
+ - [Audit actual memory use](agent-memory-protocol.md) distinguishes direct tool
22
+ evidence from narration without claiming task quality or authenticated consent.
21
23
  - [Harness compatibility](harness-compatibility.md) records the exact commands,
22
24
  hooks, plugins, launchers, skills, and MCP surfaces available in each client.
23
25
  - [MCP tool annotations and least-authority profiles](mcp-tool-profiles.md)
@@ -0,0 +1,146 @@
1
+ # Audit actual memory use
2
+
3
+ **Availability:** Cairnkeep 2.22.0 or newer; default off.
4
+
5
+ `cairn eval protocol` observes structured calls in an existing normalized
6
+ trajectory or an explicitly supplied Codex exec JSONL export. It checks project-scoped retrieval ordering, interpretable search
7
+ results, and direct memory mutation attempts without asserted capture consent.
8
+ It does not run an agent, execute recorded commands, contact services, write
9
+ memory or alter trajectories. It is disabled unless `CAIRN_EVAL=1`.
10
+
11
+ ## Safe offline controls
12
+
13
+ From a source checkout after building the server:
14
+
15
+ ```sh
16
+ CAIRN_EVAL=1 bin/cairn eval protocol \
17
+ --trajectory examples/eval/protocol-project-search.json --json
18
+ CAIRN_EVAL=1 bin/cairn eval protocol \
19
+ --trajectory examples/eval/protocol-narration.json --json
20
+ ```
21
+
22
+ The positive control passes; the narrated pseudo-call fails. These are
23
+ synthetic parser controls, not live model performance evidence. An empty search
24
+ is a successful retrieval observation: a memory hit is not mandatory. Never
25
+ repeat or widen searches merely to manufacture a hit.
26
+
27
+ ## Inspect your own evidence
28
+
29
+ Claude Code, OpenCode, and Pi use **normalized Cairnkeep trajectories**.
30
+ Codex uses the separate explicit export below, not its internal session files.
31
+ Capture remains separately
32
+ opt-in; this audit does not enable it. In the project where a session was
33
+ captured, export one closed, task-aligned session privately:
34
+
35
+ ```sh
36
+ umask 077
37
+ cairn trajectory show SESSION_ID --json > trajectory.json
38
+ CAIRN_EVAL=1 cairn eval protocol --trajectory trajectory.json --json
39
+ ```
40
+
41
+ The export can contain private prompts, paths and outputs despite capture
42
+ redaction. Keep it local; do not commit or upload it. The audit emits fixed
43
+ codes, counts, harness identity and SHA-256 digests—not the input path, session
44
+ ID, prompts, arguments, outputs or memory keys. Input must be a UTF-8 regular
45
+ file, at most 16 MiB and 50,000 events. Symlinks (including parent paths) and
46
+ files changing during inspection are rejected. Use a canonical physical path
47
+ if your filesystem path uses aliases.
48
+
49
+ ## Native Codex exec exports
50
+
51
+ [Codex non-interactive mode](https://learn.chatgpt.com/docs/non-interactive-mode)
52
+ documents `codex exec --json` as a structured JSONL event stream. Export a
53
+ single task-aligned turn to a private file, then audit it explicitly:
54
+
55
+ ```sh
56
+ umask 077
57
+ # This command runs a live agent; review permissions and the task first.
58
+ codex exec --json --ephemeral --sandbox workspace-write "YOUR TASK" > native.jsonl
59
+ # This separate command only reads the file offline.
60
+ CAIRN_EVAL=1 cairn eval protocol --codex-jsonl native.jsonl --json
61
+ ```
62
+
63
+ Use exactly one of `--trajectory` or `--codex-jsonl`. The adapter is tested
64
+ against the CLI/TypeScript SDK 0.160.1 exec event contract; it does not claim
65
+ compatibility with arbitrary future streams, app-server events or internal
66
+ rollout files. Persisted trajectory schemas and capture hooks are unchanged.
67
+ The audit adds no SDK dependency and does not launch Codex.
68
+
69
+ MCP items must identify the `cairn-memory` (or `cairn_memory`) server and the
70
+ exact memory tool. Dispatch order comes from `item.started`; completions and
71
+ updates must retain the same ID, server, tool and arguments. Parallel calls
72
+ are correlated independently. A terminal-only MCP item cannot prove dispatch
73
+ order. Native `structured_content` results or textual MCP content are accepted.
74
+
75
+ Unknown records/fields, missing lifecycle events, failed turns and incomplete
76
+ items make evidence inconclusive. Duplicate/mismatched IDs, malformed JSONL,
77
+ multiple turns or events following a terminal turn are invalid input. Split
78
+ multi-turn work into separately captured task-aligned turns; do not remove
79
+ tool events to manufacture a passing audit. One terminal newline is allowed;
80
+ blank records are rejected. The same 16 MiB byte limit applies, with at most
81
+ 50,000 source records and 50,000 normalized events.
82
+
83
+ Reasoning and narrated messages are not tool evidence; their text is not
84
+ retained by the adapter. File changes, commands and web searches are opaque
85
+ effects, not proof of approved indirect writes. Native exports contain no
86
+ authenticated project identity, timestamps or consent. Never publish private
87
+ exports; the report keeps the same payload-free observation boundary.
88
+
89
+ The packaged `examples/eval/protocol-codex.jsonl` is a synthetic positive
90
+ control, not a recorded model run. Rehearse without launching Codex:
91
+
92
+ ```sh
93
+ CAIRN_EVAL=1 bin/cairn eval protocol \
94
+ --codex-jsonl examples/eval/protocol-codex.jsonl --json
95
+ ```
96
+
97
+ ## Read the checks separately
98
+
99
+ - `evidence_complete`: ordered calls with results, without omitted unknown
100
+ records, truncation or incomplete calls. Duplicate IDs, out-of-order sequences
101
+ and orphan results are invalid input, not successful evidence.
102
+ - `memory_first`: the first recorded tool invocation is a project-scoped
103
+ `memory_search`. Task boundaries are not inferred. Trivial tasks need not
104
+ follow this protocol; tool availability is also not inferred. Apply this
105
+ check only to nontrivial tasks where direct retrieval was available, not
106
+ indiscriminately to every session or legitimate unavailable-tool fallback.
107
+ - `retrieval_result`: a project search has a correlated, non-error,
108
+ interpretable semantic or substring result. Narration never counts. Failed
109
+ or missing results are inconclusive, not model-quality failures.
110
+ - `direct_write_boundary`: direct `memory_*` mutation attempts are classified
111
+ from the MCP annotation catalog. Denied attempts still count. Opaque tools,
112
+ including shell commands, make indirect effects unverifiable.
113
+
114
+ If the user or an applicable reviewed workflow explicitly authorized capture,
115
+ add `--capture-authorized`. This is an **operator assertion**, not authenticated
116
+ consent and never permission to write. The asserted policy and its digest are
117
+ included in the report. Do not add the flag merely to turn a failure green.
118
+
119
+ Exit codes: `0` all checks pass, `1` an observed violation, `2` invalid/unsafe
120
+ input or command, `3` inconclusive without an observed violation. The disabled
121
+ path returns its existing disabled envelope with exit `0` before reading any
122
+ input; that is not a passing audit. Inspect `enabled` and `status`, not just
123
+ the process exit code.
124
+
125
+ ## What this cannot prove
126
+
127
+ A local trajectory is editable. Digests identify bytes and asserted policy;
128
+ they do not authenticate a harness, server, publisher, user or tool result.
129
+ Known direct tool names are recognized exactly, not by a permissive suffix.
130
+ CLI-mediated retrieval/mutations, generic MCP dispatchers, opaque tools and
131
+ full playbook start/finish enforcement are not certified here.
132
+
133
+ Protocol compliance does not establish relevance, source verification,
134
+ correctness, productivity or baseline improvement. Use the independent task
135
+ verifier in the [evaluation harness](operating.md#evaluation-harness-opt-in)
136
+ and the maintained [agent contract](agents.md). A realistic coding session may
137
+ show successful retrieval while its overall audit remains inconclusive because
138
+ shell effects cannot be certified. Do not convert this into a quality score or
139
+ advertise a perfect agent benchmark.
140
+
141
+ The [bounded release-ledger lab](research/release-ledger-task.md) provides a
142
+ concrete task and a separate 15-check artifact grader. Offline controls cross
143
+ all four combinations of passing/failing protocol and correct/incorrect code.
144
+ A live lab uses the actual managed `AGENTS.md`, isolated read-only memory MCP
145
+ tools and a closed single-turn export. Retain its raw evidence privately;
146
+ do not substitute the synthetic positive fixture for a live trace.
package/docs/agents.md CHANGED
@@ -46,6 +46,13 @@ integrations. See [Harness compatibility](harness-compatibility.md) for the
46
46
  exact tested surface. Instruction presence is not evidence that an agent
47
47
  actually invoked a tool.
48
48
 
49
+ For existing Claude Code, OpenCode, and Pi normalized trajectories or explicit
50
+ single-turn Codex exec JSONL exports, the
51
+ default-off [memory protocol audit](agent-memory-protocol.md) distinguishes
52
+ genuine direct calls from narration. It reports ordering, result evidence and
53
+ direct mutation attempts separately; it does not certify task quality or
54
+ indirect shell effects.
55
+
49
56
  ## Memory scopes and authority
50
57
 
51
58
  Use `scope: project` for task-specific decisions and conventions. Broader
@@ -1,7 +1,7 @@
1
1
  # Curriculum Coverage Map
2
2
 
3
- **Baseline:** Cairnkeep 2.21.1
4
- **Last reviewed:** 2026-10-06
3
+ **Baseline:** Cairnkeep 2.22.0
4
+ **Last reviewed:** 2026-10-07
5
5
 
6
6
  This map prevents the course from drifting behind the product. It assigns each
7
7
  user-facing surface to a lesson and records whether the lesson is complete or
@@ -31,6 +31,7 @@ been rehearsed.
31
31
  | Typed memory nodes, filters, address spaces, and structured import | L14 | L14 | Brief; default off |
32
32
  | Managed eight-capability contract and payload-free callbacks | L15 | L15-L16 | Brief; default off |
33
33
  | Deterministic two-pass evaluation and one-capability ablation | L16 | L16 | Brief; default off |
34
+ | Offline direct-memory protocol audit and narration negative controls | L16 | V16 supplement | Introduced in 2.22.0; default off; normalized Claude Code/OpenCode/Pi trajectories or explicit single-turn Codex exec JSONL; not task-quality evidence |
34
35
  | Shell completion | L02 | L02 | Ready; optional convenience |
35
36
  | Bounded meta-agent configuration loop | L17 | None | Design-only; not shipped |
36
37
  | Reviewed, evaluated, reversible skill-file improvement | L19 | L19 | Ready; explicit and default off |
@@ -1,6 +1,6 @@
1
1
  # Cairnkeep feature guide
2
2
 
3
- **Baseline:** Cairnkeep 2.21.1
3
+ **Baseline:** Cairnkeep 2.22.0
4
4
 
5
5
  This is the step-back map for choosing and operating Cairnkeep features. The
6
6
  [course labs](https://github.com/cairnkeep/cairnkeep-course-labs) provide one
@@ -31,6 +31,11 @@ Version 2.21 adds the formal threat model, a read-only local security posture
31
31
  check, portable-path adversarial coverage, bounded HTTP requests, and repository
32
32
  supply-chain gates; see [L28](lessons/L28-security-assurance.md).
33
33
 
34
+ Version 2.22 adds the default-off direct-memory protocol audit, including
35
+ explicit native Codex exec exports. See the L16 memory-protocol supplement;
36
+ protocol observations and independently verified task correctness remain
37
+ separate evidence, not a claim of model improvement.
38
+
34
39
  ## The three-layer model
35
40
 
36
41
  1. **Memory server:** an MCP process that owns scoped durable storage. The
@@ -108,6 +113,7 @@ prerequisites for ordinary memory.
108
113
  | Typed nodes/import | `CAIRN_TYPED_MEMORY_NODES=1` | Restart MCP server; tool schema changes | Hard-filter search; dry-run, replay-safe import; `cairn doctor --repair` only for derived state | `course-05-evidence` |
109
114
  | Capability contract | `CAIRN_CAPABILITY_CONTRACT=1` | MCP changes require restart; operating changes apply next invocation | `cairn capabilities list|status|enable|disable|reset|logging`; digest identifies state only | `course-06-governance` |
110
115
  | Evaluation | `CAIRN_EVAL=1` for the coordinator process | No harness chosen by Cairnkeep | `validate` before `run`; inspect report, missingness and evidence scope; dry-run prune/delete | `course-07-evaluation` |
116
+ | Memory protocol audit | `CAIRN_EVAL=1` for the audit process | No restart; existing normalized trajectory or single-turn Codex exec JSONL export | `eval protocol --trajectory PATH` or `--codex-jsonl PATH`; fixed-code observations, no writes/network; caller-asserted capture authorization; task quality and indirect effects not certified | L16 supplement |
111
117
  | Validated skill improvement | Reviewed recurring hindsight; proposal adapter; `CAIRN_EVAL=1` for evaluation | No server restart; explicit subprocesses only | `cairn skill list|show`; exact-digest apply; concurrent-edit-safe rollback | `course-09-skill` |
112
118
  | General meta-agent loop | Not shipped | Not applicable | Playbooks select bounded existing actions; no generic configuration loop, command language, or scheduler | `course-07-evaluation` |
113
119
  | Pack retrieval and skills | `CAIRN_CONTEXT_PACKS=1`; HTTP needs separate consent; skills need exact-digest approval | Restart MCP server | `cairn pack skills`; revoke approval or disable pack; no automatic execution | `course-10-trust-context` |
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 10 minutes
6
- **Tested with:** Cairnkeep 2.21.1
6
+ **Tested with:** Cairnkeep 2.22.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 15 minutes
6
- **Tested with:** Cairnkeep 2.21.1
6
+ **Tested with:** Cairnkeep 2.22.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 25 minutes
6
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
6
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
7
7
 
8
8
  ## Outcome
9
9
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 25 minutes
6
- **Tested with:** Cairnkeep 2.21.1
6
+ **Tested with:** Cairnkeep 2.22.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -19,6 +19,8 @@ report, and explain what a one-capability ablation can and cannot establish.
19
19
  - Eligible pairs, missingness, compatible turn semantics, reported token/cost
20
20
  fields, bootstrap intervals, and bounded reports.
21
21
  - `offline-framework` evidence versus separately validated live evidence.
22
+ - Genuine memory calls versus narrated pseudo-calls; protocol ordering,
23
+ interpretable results, consent assertions and inconclusive indirect effects.
22
24
  - Dry validation, invocation estimates, explicit `--yes`, cancellation, report
23
25
  retention, dry-run pruning, and deletion.
24
26
 
@@ -51,6 +53,27 @@ after the lab is rehearsed from the packaged CLI rather than a source clone.
51
53
  - The learner reports estimates, intervals, missingness, and evidence scope
52
54
  without claiming causality, significance, or product improvement.
53
55
  - Course cleanup is limited to its temporary contained experiment root.
56
+ - The protocol positive control passes and narration fails; neither is
57
+ described as live model improvement or task correctness.
58
+
59
+ ## Supplement: memory protocol evidence
60
+
61
+ This supplement requires Cairnkeep 2.22.0 or newer for `eval protocol` and
62
+ `--codex-jsonl`. Evaluation remains default off.
63
+
64
+ Run the two synthetic commands in the
65
+ [memory protocol guide](../../agent-memory-protocol.md). Pause on the negative
66
+ control's exit `1`: saying a tool was used is not evidence of using it. Explain
67
+ why an empty real search can pass retrieval, why incomplete capture or failed
68
+ retrieval is inconclusive, and why shell calls prevent certification of
69
+ indirect memory writes. Never publish a real trajectory export for this lab.
70
+
71
+ Repeat with the packaged `protocol-codex.jsonl` synthetic control using
72
+ `--codex-jsonl`, not `--trajectory`. Explain start/completion correlation,
73
+ unknown-event gaps, single-turn boundaries and the distinction between a
74
+ private live export and a public parser fixture. Internal rollout files and
75
+ automatic Codex capture are not supported. Pair protocol observations with an
76
+ independent artifact verifier; either dimension can fail separately.
54
77
 
55
78
  ## Source material
56
79
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Evidence and Evaluation
5
5
  **Time:** 50 minutes
6
- **Tested with:** Cairnkeep 2.21.1
6
+ **Tested with:** Cairnkeep 2.22.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -1,7 +1,7 @@
1
1
  # L20 - Least-authority MCP tool profiles
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
5
5
  **Time:** 25 minutes
6
6
 
7
7
  ## Outcome
@@ -1,7 +1,7 @@
1
1
  # L21 - Immutable context packs and skill approval
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
5
5
  **Time:** 40 minutes
6
6
 
7
7
  ## Outcome
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Operator
5
5
  **Time:** 30 minutes
6
- **Tested with:** Cairnkeep 2.21.1, Windows x64, and Node.js 22 or newer
6
+ **Tested with:** Cairnkeep 2.22.0, Windows x64, and Node.js 22 or newer
7
7
 
8
8
  ## Outcome
9
9
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart and Operator
5
5
  **Time:** 35 minutes
6
- **Tested with:** Cairnkeep 2.21.1; Pi 0.84.1; Node.js 22, 24, and 26
6
+ **Tested with:** Cairnkeep 2.22.0; Pi 0.84.1; Node.js 22, 24, and 26
7
7
 
8
8
  ## Outcome
9
9
 
@@ -1,7 +1,7 @@
1
1
  # L24 - Reviewed knowledge exchange
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
5
5
  **Time:** 35 minutes
6
6
 
7
7
  ## Outcome
@@ -1,7 +1,7 @@
1
1
  # L25 - Bounded workflow playbooks
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
5
5
  **Time:** 35 minutes
6
6
 
7
7
  ## Outcome
@@ -1,7 +1,7 @@
1
1
  # L26 - Context intelligence with explicit authority
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
5
5
 
6
6
  ## Outcome
7
7
 
@@ -1,6 +1,6 @@
1
1
  # L27 - Evaluate context companions without moving authority
2
2
 
3
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
3
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
4
4
 
5
5
  ## Outcome
6
6
 
@@ -1,7 +1,7 @@
1
1
  # L28 - Verify Cairnkeep's security posture
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.21.1 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
5
5
 
6
6
  ## Outcome
7
7
 
@@ -11,7 +11,7 @@ and choose least authority for a network-facing MCP client.
11
11
 
12
12
  ## Prerequisites
13
13
 
14
- - Install Cairnkeep 2.21.1 or use a disposable source checkout.
14
+ - Install Cairnkeep 2.22.0 or use a disposable source checkout.
15
15
  - Complete [L20 - Least-authority MCP tool profiles](L20-mcp-tool-profiles.md).
16
16
  - Use synthetic values only; never paste a real bearer token into a recording,
17
17
  terminal transcript, or issue.
@@ -20,5 +20,9 @@ reports without making unsupported claims, apply or roll back an exact eligible
20
20
  skill proposal, and distinguish that narrow runtime from the design-only
21
21
  general meta-agent contract.
22
22
 
23
+ L16 also covers the offline memory protocol controls: distinguish direct tool
24
+ events from narration and explain why protocol observations, caller-asserted
25
+ consent and task-quality evidence are separate claims.
26
+
23
27
  Hands-on spine: `course-05-evidence` through `course-07-evaluation` in the
24
28
  public [course labs](https://github.com/cairnkeep/cairnkeep-course-labs).
@@ -0,0 +1,32 @@
1
+ # V16 supplement — Show the tool call, not the claim
2
+
3
+ Target length: 4–6 minutes. Experienced-developer audience. Record with
4
+ Cairnkeep 2.22.0 or newer and opt in to `cairn eval protocol` explicitly.
5
+ Use only committed synthetic controls, never private transcripts.
6
+
7
+ 1. Split the terminal: show `protocol-narration.json` beside its audit. Type the
8
+ command from the memory protocol guide at human speed. Pause two seconds
9
+ on `project_search_absent` and exit `1`: narration is not execution.
10
+ 2. Switch to `protocol-project-search.json`. Highlight the correlated IDs and
11
+ project scope. Run the audit and pause on the separate checks. Zero hits is
12
+ acceptable; it is not a reason to keep widening the query.
13
+ 3. Show the four exit states. Default-off exit `0` is not a successful audit.
14
+ Incomplete capture, failed retrieval and opaque shell commands yield
15
+ inconclusive observations, not a fabricated perfect score.
16
+ 4. Zoom into digests and counts, then back to the input: the report omits
17
+ private content. A digest authenticates neither a publisher nor human
18
+ approval. `--capture-authorized` is an assertion, not a bypass button.
19
+ 5. Switch to `protocol-codex.jsonl`, then run `--codex-jsonl`. Visually pair
20
+ `item.started` and `item.completed`; updates are not extra calls. Explain
21
+ single-turn boundaries and unknown events. Keep private exports off screen.
22
+ 6. End on the independent verifier, not a green audit. This parser lab does not
23
+ prove a better model, correct code, full playbook compliance or superiority
24
+ over a baseline.
25
+
26
+ Vary framing between input, IDs and report codes. Leave breathing room after
27
+ failures and caveats; do not accelerate terminal footage to match speech.
28
+
29
+ For a coding demonstration, show task correctness beside the individual
30
+ protocol checks. Successful retrieval followed by coding commands may produce
31
+ an inconclusive overall audit because shell effects are opaque. Do not edit
32
+ that into a pass or claim an A/B improvement from one run.
package/docs/operating.md CHANGED
@@ -1080,6 +1080,14 @@ machine-level drift; setup itself never performs the sync.
1080
1080
 
1081
1081
  ### Evaluation harness (opt-in)
1082
1082
 
1083
+ To inspect actual direct memory tool use in an existing normalized trajectory,
1084
+ use `CAIRN_EVAL=1 cairn eval protocol --trajectory PATH --json`. Native
1085
+ single-turn Codex exec JSONL exports use `--codex-jsonl PATH` instead. This read-only,
1086
+ offline audit returns separate pass/fail/inconclusive checks, emits no recorded
1087
+ content, and does not measure task quality. See the
1088
+ [memory protocol guide](agent-memory-protocol.md) for controls, export privacy,
1089
+ supported harnesses, consent assertions and exit codes.
1090
+
1083
1091
  `cairn eval` is a local, serial experiment coordinator. It is disabled unless
1084
1092
  `CAIRN_EVAL` is explicitly truthy. The disabled path returns before reading a
1085
1093
  task set or adapter configuration, creating a workspace or report, opening a
@@ -172,6 +172,15 @@ file, so the diagnostic and runtime use the same precedence.
172
172
 
173
173
  ## Evaluation adapter and report flow
174
174
 
175
+ `cairn eval protocol` is a separate read-only audit of an explicitly supplied
176
+ normalized trajectory or single-turn Codex exec JSONL export. With evaluation disabled it does not read that
177
+ file. When enabled it performs bounded local reads only: no adapter, command
178
+ execution, network, database, or persistent report. Output is limited to fixed
179
+ codes, counts, harness identity, input/policy digests and limitations. Exports
180
+ can contain private content even after redaction; keep them local. See the
181
+ [memory protocol guide](agent-memory-protocol.md). A capture-authorization flag
182
+ is the caller's assertion, not authenticated consent.
183
+
175
184
  `CAIRN_EVAL` is unset/off by default. With it off, `cairn eval` returns a fixed
176
185
  disabled result before task-set or adapter reads, workspace/report creation,
177
186
  database access, subprocess execution, or network activity. A credential in
@@ -0,0 +1,36 @@
1
+ # Bounded release-ledger task
2
+
3
+ Use a disposable project, Node standard APIs and ESM only. Implement
4
+ `src/release-ledger.mjs` exporting `createLedger()`. The returned API exposes
5
+ `append(input)`, `finalize(packageName, version)`, `get(packageName, version)`
6
+ and `list()`.
7
+
8
+ - Package names match `^[a-z][a-z0-9-]*$`; versions are numeric x.y.z. Notes
9
+ are non-empty arrays of non-empty strings, preserving order. Invalid input
10
+ throws Error with code `INVALID_PACKAGE`, `INVALID_VERSION` or `INVALID_NOTES`.
11
+ - `append({package,version,notes})` creates a draft with monotonically
12
+ increasing sequence starting at 1. A pair is unique forever; duplicates
13
+ throw `DUPLICATE_RELEASE`.
14
+ - Entries have exactly package, version, notes, sequence, state and optional
15
+ checksum. Draft state is `draft`; finalized state is `final`.
16
+ - `finalize` is idempotent. Checksum is lowercase SHA-256 of
17
+ `JSON.stringify({package,version,notes,sequence})` in that property order.
18
+ Missing releases throw `NOT_FOUND`.
19
+ - `get` and `list` return detached copies including nested notes. `list` is
20
+ sequence ordered. Add and run self-tests without changing this brief.
21
+
22
+ After the task closes, run the maintained independent grader from outside the
23
+ project, with an external process timeout:
24
+
25
+ ```sh
26
+ node scripts/spikes/verify-release-ledger.mjs /PATH/TO/LAB/src/release-ledger.mjs
27
+ ```
28
+
29
+ This imports and executes the selected artifact; it is not a safe reader for
30
+ arbitrary untrusted code. Use a lab with no production secrets or services.
31
+ The grader checks 15 bounded behaviors, not all possible inputs or security.
32
+ Do not expose its implementation to the task before grading. A model could
33
+ deliberately game any visible rubric; this is a correctness smoke test, not
34
+ adversarial certification. Run protocol observation separately using the
35
+ [memory audit](../agent-memory-protocol.md). Keep raw exports private; publish
36
+ only appropriately reviewed aggregate evidence. One run is not an A/B study.
@@ -1,5 +1,18 @@
1
1
  # Offline evaluation fixture
2
2
 
3
+ The separate `protocol-project-search.json`, `protocol-narration.json` and
4
+ native-format `protocol-codex.jsonl`
5
+ controls exercise `cairn eval protocol`, without running an adapter. See the
6
+ [memory protocol guide](../../docs/agent-memory-protocol.md). These synthetic
7
+ controls demonstrate parser behavior, not live model quality.
8
+
9
+ `scripts/spikes/verify-release-ledger.mjs MODULE_PATH` separately grades the
10
+ bounded release-ledger coding task with 15 deterministic artifact checks. It
11
+ imports and executes the explicitly selected module: use an isolated lab and
12
+ an external timeout, never untrusted production files. Its score is not a
13
+ protocol score, a comprehensive security audit or an A/B comparison.
14
+ The [task brief](../../docs/research/release-ledger-task.md) defines the lab.
15
+
3
16
  This directory contains Cairnkeep's deterministic, network-free evaluation
4
17
  population. It validates the evaluation framework; its designed differences are
5
18
  not performance, efficiency, quality, or live-product evidence.
@@ -1 +1 @@
1
- {"identifier":"cairn-offline-fake-v1","package_version":"2.21.1","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
1
+ {"identifier":"cairn-offline-fake-v1","package_version":"2.22.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
@@ -0,0 +1,5 @@
1
+ {"type":"thread.started","thread_id":"synthetic-control"}
2
+ {"type":"turn.started"}
3
+ {"type":"item.started","item":{"id":"a","type":"mcp_tool_call","server":"cairn-memory","tool":"memory_search","arguments":{"scope":"project","query":"release ledger"},"status":"in_progress"}}
4
+ {"type":"item.completed","item":{"id":"a","type":"mcp_tool_call","server":"cairn-memory","tool":"memory_search","arguments":{"scope":"project","query":"release ledger"},"status":"completed","result":{"content":[],"structured_content":{"mode":"substring","count":0,"results":[]}}}}
5
+ {"type":"turn.completed","usage":{"input_tokens":10,"cached_input_tokens":0,"output_tokens":1}}
@@ -0,0 +1,12 @@
1
+ {
2
+ "schema_version": 1,
3
+ "session_id": "protocol-negative-control",
4
+ "harness": "pi",
5
+ "project_root": "/synthetic/project",
6
+ "started_at": "2026-10-07T00:00:00.000Z",
7
+ "ended_at": "2026-10-07T00:01:00.000Z",
8
+ "events": [
9
+ {"sequence": 0, "kind": "model_output", "payload": {"text": "I called memory_search with scope project and found the ledger conventions."}}
10
+ ],
11
+ "capture": {"captured_at": "2026-10-07T00:01:00.000Z", "omitted_reasoning_blocks": 0, "omitted_unknown_records": 0, "truncated": false}
12
+ }
@@ -0,0 +1,13 @@
1
+ {
2
+ "schema_version": 1,
3
+ "session_id": "protocol-positive-control",
4
+ "harness": "pi",
5
+ "project_root": "/synthetic/project",
6
+ "started_at": "2026-10-07T00:00:00.000Z",
7
+ "ended_at": "2026-10-07T00:01:00.000Z",
8
+ "events": [
9
+ {"sequence": 0, "kind": "tool_invocation", "payload": {"call_id": "search-1", "tool_name": "memory_search", "input": {"scope": "project", "query": "ledger validation conventions"}}},
10
+ {"sequence": 1, "kind": "tool_result", "payload": {"call_id": "search-1", "is_error": false, "output": "{\"mode\":\"substring\",\"count\":0,\"results\":[]}"}}
11
+ ],
12
+ "capture": {"captured_at": "2026-10-07T00:01:00.000Z", "omitted_reasoning_blocks": 0, "omitted_unknown_records": 0, "truncated": false}
13
+ }