@cairnkeep/cli 2.21.0 → 2.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +34 -0
- package/docs/README.md +2 -0
- package/docs/agent-memory-protocol.md +146 -0
- package/docs/agents.md +7 -0
- package/docs/learning/CURRICULUM-MAP.md +3 -2
- package/docs/learning/FEATURE-GUIDE.md +7 -1
- package/docs/learning/lessons/L00-why-cairnkeep.md +1 -1
- package/docs/learning/lessons/L01-safe-trial.md +1 -1
- package/docs/learning/lessons/L02-installation.md +1 -1
- package/docs/learning/lessons/L03-first-project.md +1 -1
- package/docs/learning/lessons/L16-evaluation.md +23 -0
- package/docs/learning/lessons/L19-validated-skill-improvement.md +1 -1
- package/docs/learning/lessons/L20-mcp-tool-profiles.md +1 -1
- package/docs/learning/lessons/L21-context-packs.md +1 -1
- package/docs/learning/lessons/L22-native-windows.md +1 -1
- package/docs/learning/lessons/L23-guided-setup.md +1 -1
- package/docs/learning/lessons/L24-okf-exchange.md +1 -1
- package/docs/learning/lessons/L25-playbooks.md +1 -1
- package/docs/learning/lessons/L26-context-intelligence.md +1 -1
- package/docs/learning/lessons/L27-ecosystem-evidence.md +1 -1
- package/docs/learning/lessons/L28-security-assurance.md +2 -2
- package/docs/learning/tracks/evidence-and-evaluation.md +4 -0
- package/docs/learning/video-scripts/V16-memory-protocol-audit.md +32 -0
- package/docs/operating.md +12 -0
- package/docs/privacy-and-data-flow.md +9 -0
- package/docs/releasing.md +7 -0
- package/docs/research/release-ledger-task.md +36 -0
- package/docs/security-assurance.md +8 -0
- package/examples/eval/README.md +13 -0
- package/examples/eval/bundled-fake.json +1 -1
- package/examples/eval/protocol-codex.jsonl +5 -0
- package/examples/eval/protocol-narration.json +12 -0
- package/examples/eval/protocol-project-search.json +13 -0
- package/mcp-memory-server/dist/eval-cli.js +25 -2
- package/mcp-memory-server/dist/eval-codex.js +163 -0
- package/mcp-memory-server/dist/eval-protocol.js +200 -0
- package/mcp-memory-server/package.json +3 -1
- package/package.json +1 -1
- package/scripts/cairn-cli.mjs +1 -1
- package/scripts/completion.sh +9 -4
- package/scripts/doctor.sh +20 -4
- package/scripts/spikes/verify-release-ledger.mjs +46 -0
- package/scripts/test-ci-sharding.sh +33 -0
- package/scripts/test-completion.sh +7 -3
- package/scripts/test-doctor.sh +29 -11
- package/scripts/test-ecosystem-spikes.sh +1 -0
- package/scripts/test-package-install.sh +24 -1
- package/scripts/windows-platform.mjs +2 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,40 @@ All notable user-facing changes are documented here. This project follows
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [2.22.0] - 2026-10-07
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- Add explicit `eval protocol --codex-jsonl PATH` for single-turn native Codex
|
|
13
|
+
exec exports, without automatic capture or changes to persisted trajectories.
|
|
14
|
+
Correlate MCP lifecycles conservatively; unknown or incomplete events cannot
|
|
15
|
+
pass. Include offline controls, installed-package tests, completions and
|
|
16
|
+
learning guidance; task correctness remains independently verified.
|
|
17
|
+
- Add default-off `cairn eval protocol` to audit direct memory tool use in
|
|
18
|
+
existing normalized Claude Code, OpenCode and Pi trajectories. Reports
|
|
19
|
+
retrieval ordering, interpretable results and direct mutation attempts with
|
|
20
|
+
conservative pass/fail/inconclusive states, bounded local reads and
|
|
21
|
+
payload-free output. Capture authorization is a caller assertion, not an
|
|
22
|
+
authenticated grant; task quality and indirect shell effects are not certified.
|
|
23
|
+
- Add offline positive/narration controls, packaged CLI verification, shell
|
|
24
|
+
completions, operating/privacy guidance and the L16/video learning supplement.
|
|
25
|
+
|
|
26
|
+
## [2.21.1] - 2026-10-07
|
|
27
|
+
|
|
28
|
+
### Fixed
|
|
29
|
+
|
|
30
|
+
- Classify embedding health failures by sanitized HTTP status so operators can
|
|
31
|
+
distinguish authorization, missing endpoint/model, throttling, transport,
|
|
32
|
+
and upstream failures without exposing provider response bodies, request
|
|
33
|
+
identifiers, or credentials.
|
|
34
|
+
|
|
35
|
+
### CI
|
|
36
|
+
|
|
37
|
+
- Split all 48 memory-server smoke contracts exactly once across three
|
|
38
|
+
deterministically balanced native-Windows jobs while retaining Windows CLI,
|
|
39
|
+
package, global-install, Node 22/24/26, and full Linux matrix coverage. This
|
|
40
|
+
removes the serial 15-minute suite duplication from every Windows lane.
|
|
41
|
+
|
|
8
42
|
## [2.21.0] - 2026-10-07
|
|
9
43
|
|
|
10
44
|
### Added
|
package/docs/README.md
CHANGED
|
@@ -18,6 +18,8 @@ linked guides rather than the project landing page.
|
|
|
18
18
|
|
|
19
19
|
- [Cairnkeep for coding agents](agents.md) explains retrieval-first behavior,
|
|
20
20
|
memory authority, playbooks, approvals, and graceful fallback.
|
|
21
|
+
- [Audit actual memory use](agent-memory-protocol.md) distinguishes direct tool
|
|
22
|
+
evidence from narration without claiming task quality or authenticated consent.
|
|
21
23
|
- [Harness compatibility](harness-compatibility.md) records the exact commands,
|
|
22
24
|
hooks, plugins, launchers, skills, and MCP surfaces available in each client.
|
|
23
25
|
- [MCP tool annotations and least-authority profiles](mcp-tool-profiles.md)
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# Audit actual memory use
|
|
2
|
+
|
|
3
|
+
**Availability:** Cairnkeep 2.22.0 or newer; default off.
|
|
4
|
+
|
|
5
|
+
`cairn eval protocol` observes structured calls in an existing normalized
|
|
6
|
+
trajectory or an explicitly supplied Codex exec JSONL export. It checks project-scoped retrieval ordering, interpretable search
|
|
7
|
+
results, and direct memory mutation attempts without asserted capture consent.
|
|
8
|
+
It does not run an agent, execute recorded commands, contact services, write
|
|
9
|
+
memory or alter trajectories. It is disabled unless `CAIRN_EVAL=1`.
|
|
10
|
+
|
|
11
|
+
## Safe offline controls
|
|
12
|
+
|
|
13
|
+
From a source checkout after building the server:
|
|
14
|
+
|
|
15
|
+
```sh
|
|
16
|
+
CAIRN_EVAL=1 bin/cairn eval protocol \
|
|
17
|
+
--trajectory examples/eval/protocol-project-search.json --json
|
|
18
|
+
CAIRN_EVAL=1 bin/cairn eval protocol \
|
|
19
|
+
--trajectory examples/eval/protocol-narration.json --json
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
The positive control passes; the narrated pseudo-call fails. These are
|
|
23
|
+
synthetic parser controls, not live model performance evidence. An empty search
|
|
24
|
+
is a successful retrieval observation: a memory hit is not mandatory. Never
|
|
25
|
+
repeat or widen searches merely to manufacture a hit.
|
|
26
|
+
|
|
27
|
+
## Inspect your own evidence
|
|
28
|
+
|
|
29
|
+
Claude Code, OpenCode, and Pi use **normalized Cairnkeep trajectories**.
|
|
30
|
+
Codex uses the separate explicit export below, not its internal session files.
|
|
31
|
+
Capture remains separately
|
|
32
|
+
opt-in; this audit does not enable it. In the project where a session was
|
|
33
|
+
captured, export one closed, task-aligned session privately:
|
|
34
|
+
|
|
35
|
+
```sh
|
|
36
|
+
umask 077
|
|
37
|
+
cairn trajectory show SESSION_ID --json > trajectory.json
|
|
38
|
+
CAIRN_EVAL=1 cairn eval protocol --trajectory trajectory.json --json
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
The export can contain private prompts, paths and outputs despite capture
|
|
42
|
+
redaction. Keep it local; do not commit or upload it. The audit emits fixed
|
|
43
|
+
codes, counts, harness identity and SHA-256 digests—not the input path, session
|
|
44
|
+
ID, prompts, arguments, outputs or memory keys. Input must be a UTF-8 regular
|
|
45
|
+
file, at most 16 MiB and 50,000 events. Symlinks (including parent paths) and
|
|
46
|
+
files changing during inspection are rejected. Use a canonical physical path
|
|
47
|
+
if your filesystem path uses aliases.
|
|
48
|
+
|
|
49
|
+
## Native Codex exec exports
|
|
50
|
+
|
|
51
|
+
[Codex non-interactive mode](https://learn.chatgpt.com/docs/non-interactive-mode)
|
|
52
|
+
documents `codex exec --json` as a structured JSONL event stream. Export a
|
|
53
|
+
single task-aligned turn to a private file, then audit it explicitly:
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
umask 077
|
|
57
|
+
# This command runs a live agent; review permissions and the task first.
|
|
58
|
+
codex exec --json --ephemeral --sandbox workspace-write "YOUR TASK" > native.jsonl
|
|
59
|
+
# This separate command only reads the file offline.
|
|
60
|
+
CAIRN_EVAL=1 cairn eval protocol --codex-jsonl native.jsonl --json
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Use exactly one of `--trajectory` or `--codex-jsonl`. The adapter is tested
|
|
64
|
+
against the CLI/TypeScript SDK 0.160.1 exec event contract; it does not claim
|
|
65
|
+
compatibility with arbitrary future streams, app-server events or internal
|
|
66
|
+
rollout files. Persisted trajectory schemas and capture hooks are unchanged.
|
|
67
|
+
The audit adds no SDK dependency and does not launch Codex.
|
|
68
|
+
|
|
69
|
+
MCP items must identify the `cairn-memory` (or `cairn_memory`) server and the
|
|
70
|
+
exact memory tool. Dispatch order comes from `item.started`; completions and
|
|
71
|
+
updates must retain the same ID, server, tool and arguments. Parallel calls
|
|
72
|
+
are correlated independently. A terminal-only MCP item cannot prove dispatch
|
|
73
|
+
order. Native `structured_content` results or textual MCP content are accepted.
|
|
74
|
+
|
|
75
|
+
Unknown records/fields, missing lifecycle events, failed turns and incomplete
|
|
76
|
+
items make evidence inconclusive. Duplicate/mismatched IDs, malformed JSONL,
|
|
77
|
+
multiple turns or events following a terminal turn are invalid input. Split
|
|
78
|
+
multi-turn work into separately captured task-aligned turns; do not remove
|
|
79
|
+
tool events to manufacture a passing audit. One terminal newline is allowed;
|
|
80
|
+
blank records are rejected. The same 16 MiB byte limit applies, with at most
|
|
81
|
+
50,000 source records and 50,000 normalized events.
|
|
82
|
+
|
|
83
|
+
Reasoning and narrated messages are not tool evidence; their text is not
|
|
84
|
+
retained by the adapter. File changes, commands and web searches are opaque
|
|
85
|
+
effects, not proof of approved indirect writes. Native exports contain no
|
|
86
|
+
authenticated project identity, timestamps or consent. Never publish private
|
|
87
|
+
exports; the report keeps the same payload-free observation boundary.
|
|
88
|
+
|
|
89
|
+
The packaged `examples/eval/protocol-codex.jsonl` is a synthetic positive
|
|
90
|
+
control, not a recorded model run. Rehearse without launching Codex:
|
|
91
|
+
|
|
92
|
+
```sh
|
|
93
|
+
CAIRN_EVAL=1 bin/cairn eval protocol \
|
|
94
|
+
--codex-jsonl examples/eval/protocol-codex.jsonl --json
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Read the checks separately
|
|
98
|
+
|
|
99
|
+
- `evidence_complete`: ordered calls with results, without omitted unknown
|
|
100
|
+
records, truncation or incomplete calls. Duplicate IDs, out-of-order sequences
|
|
101
|
+
and orphan results are invalid input, not successful evidence.
|
|
102
|
+
- `memory_first`: the first recorded tool invocation is a project-scoped
|
|
103
|
+
`memory_search`. Task boundaries are not inferred. Trivial tasks need not
|
|
104
|
+
follow this protocol; tool availability is also not inferred. Apply this
|
|
105
|
+
check only to nontrivial tasks where direct retrieval was available, not
|
|
106
|
+
indiscriminately to every session or legitimate unavailable-tool fallback.
|
|
107
|
+
- `retrieval_result`: a project search has a correlated, non-error,
|
|
108
|
+
interpretable semantic or substring result. Narration never counts. Failed
|
|
109
|
+
or missing results are inconclusive, not model-quality failures.
|
|
110
|
+
- `direct_write_boundary`: direct `memory_*` mutation attempts are classified
|
|
111
|
+
from the MCP annotation catalog. Denied attempts still count. Opaque tools,
|
|
112
|
+
including shell commands, make indirect effects unverifiable.
|
|
113
|
+
|
|
114
|
+
If the user or an applicable reviewed workflow explicitly authorized capture,
|
|
115
|
+
add `--capture-authorized`. This is an **operator assertion**, not authenticated
|
|
116
|
+
consent and never permission to write. The asserted policy and its digest are
|
|
117
|
+
included in the report. Do not add the flag merely to turn a failure green.
|
|
118
|
+
|
|
119
|
+
Exit codes: `0` all checks pass, `1` an observed violation, `2` invalid/unsafe
|
|
120
|
+
input or command, `3` inconclusive without an observed violation. The disabled
|
|
121
|
+
path returns its existing disabled envelope with exit `0` before reading any
|
|
122
|
+
input; that is not a passing audit. Inspect `enabled` and `status`, not just
|
|
123
|
+
the process exit code.
|
|
124
|
+
|
|
125
|
+
## What this cannot prove
|
|
126
|
+
|
|
127
|
+
A local trajectory is editable. Digests identify bytes and asserted policy;
|
|
128
|
+
they do not authenticate a harness, server, publisher, user or tool result.
|
|
129
|
+
Known direct tool names are recognized exactly, not by a permissive suffix.
|
|
130
|
+
CLI-mediated retrieval/mutations, generic MCP dispatchers, opaque tools and
|
|
131
|
+
full playbook start/finish enforcement are not certified here.
|
|
132
|
+
|
|
133
|
+
Protocol compliance does not establish relevance, source verification,
|
|
134
|
+
correctness, productivity or baseline improvement. Use the independent task
|
|
135
|
+
verifier in the [evaluation harness](operating.md#evaluation-harness-opt-in)
|
|
136
|
+
and the maintained [agent contract](agents.md). A realistic coding session may
|
|
137
|
+
show successful retrieval while its overall audit remains inconclusive because
|
|
138
|
+
shell effects cannot be certified. Do not convert this into a quality score or
|
|
139
|
+
advertise a perfect agent benchmark.
|
|
140
|
+
|
|
141
|
+
The [bounded release-ledger lab](research/release-ledger-task.md) provides a
|
|
142
|
+
concrete task and a separate 15-check artifact grader. Offline controls cross
|
|
143
|
+
all four combinations of passing/failing protocol and correct/incorrect code.
|
|
144
|
+
A live lab uses the actual managed `AGENTS.md`, isolated read-only memory MCP
|
|
145
|
+
tools and a closed single-turn export. Retain its raw evidence privately;
|
|
146
|
+
do not substitute the synthetic positive fixture for a live trace.
|
package/docs/agents.md
CHANGED
|
@@ -46,6 +46,13 @@ integrations. See [Harness compatibility](harness-compatibility.md) for the
|
|
|
46
46
|
exact tested surface. Instruction presence is not evidence that an agent
|
|
47
47
|
actually invoked a tool.
|
|
48
48
|
|
|
49
|
+
For existing Claude Code, OpenCode, and Pi normalized trajectories or explicit
|
|
50
|
+
single-turn Codex exec JSONL exports, the
|
|
51
|
+
default-off [memory protocol audit](agent-memory-protocol.md) distinguishes
|
|
52
|
+
genuine direct calls from narration. It reports ordering, result evidence and
|
|
53
|
+
direct mutation attempts separately; it does not certify task quality or
|
|
54
|
+
indirect shell effects.
|
|
55
|
+
|
|
49
56
|
## Memory scopes and authority
|
|
50
57
|
|
|
51
58
|
Use `scope: project` for task-specific decisions and conventions. Broader
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Curriculum Coverage Map
|
|
2
2
|
|
|
3
|
-
**Baseline:** Cairnkeep 2.
|
|
4
|
-
**Last reviewed:** 2026-10-
|
|
3
|
+
**Baseline:** Cairnkeep 2.22.0
|
|
4
|
+
**Last reviewed:** 2026-10-07
|
|
5
5
|
|
|
6
6
|
This map prevents the course from drifting behind the product. It assigns each
|
|
7
7
|
user-facing surface to a lesson and records whether the lesson is complete or
|
|
@@ -31,6 +31,7 @@ been rehearsed.
|
|
|
31
31
|
| Typed memory nodes, filters, address spaces, and structured import | L14 | L14 | Brief; default off |
|
|
32
32
|
| Managed eight-capability contract and payload-free callbacks | L15 | L15-L16 | Brief; default off |
|
|
33
33
|
| Deterministic two-pass evaluation and one-capability ablation | L16 | L16 | Brief; default off |
|
|
34
|
+
| Offline direct-memory protocol audit and narration negative controls | L16 | V16 supplement | Introduced in 2.22.0; default off; normalized Claude Code/OpenCode/Pi trajectories or explicit single-turn Codex exec JSONL; not task-quality evidence |
|
|
34
35
|
| Shell completion | L02 | L02 | Ready; optional convenience |
|
|
35
36
|
| Bounded meta-agent configuration loop | L17 | None | Design-only; not shipped |
|
|
36
37
|
| Reviewed, evaluated, reversible skill-file improvement | L19 | L19 | Ready; explicit and default off |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Cairnkeep feature guide
|
|
2
2
|
|
|
3
|
-
**Baseline:** Cairnkeep 2.
|
|
3
|
+
**Baseline:** Cairnkeep 2.22.0
|
|
4
4
|
|
|
5
5
|
This is the step-back map for choosing and operating Cairnkeep features. The
|
|
6
6
|
[course labs](https://github.com/cairnkeep/cairnkeep-course-labs) provide one
|
|
@@ -31,6 +31,11 @@ Version 2.21 adds the formal threat model, a read-only local security posture
|
|
|
31
31
|
check, portable-path adversarial coverage, bounded HTTP requests, and repository
|
|
32
32
|
supply-chain gates; see [L28](lessons/L28-security-assurance.md).
|
|
33
33
|
|
|
34
|
+
Version 2.22 adds the default-off direct-memory protocol audit, including
|
|
35
|
+
explicit native Codex exec exports. See the L16 memory-protocol supplement;
|
|
36
|
+
protocol observations and independently verified task correctness remain
|
|
37
|
+
separate evidence, not a claim of model improvement.
|
|
38
|
+
|
|
34
39
|
## The three-layer model
|
|
35
40
|
|
|
36
41
|
1. **Memory server:** an MCP process that owns scoped durable storage. The
|
|
@@ -108,6 +113,7 @@ prerequisites for ordinary memory.
|
|
|
108
113
|
| Typed nodes/import | `CAIRN_TYPED_MEMORY_NODES=1` | Restart MCP server; tool schema changes | Hard-filter search; dry-run, replay-safe import; `cairn doctor --repair` only for derived state | `course-05-evidence` |
|
|
109
114
|
| Capability contract | `CAIRN_CAPABILITY_CONTRACT=1` | MCP changes require restart; operating changes apply next invocation | `cairn capabilities list|status|enable|disable|reset|logging`; digest identifies state only | `course-06-governance` |
|
|
110
115
|
| Evaluation | `CAIRN_EVAL=1` for the coordinator process | No harness chosen by Cairnkeep | `validate` before `run`; inspect report, missingness and evidence scope; dry-run prune/delete | `course-07-evaluation` |
|
|
116
|
+
| Memory protocol audit | `CAIRN_EVAL=1` for the audit process | No restart; existing normalized trajectory or single-turn Codex exec JSONL export | `eval protocol --trajectory PATH` or `--codex-jsonl PATH`; fixed-code observations, no writes/network; caller-asserted capture authorization; task quality and indirect effects not certified | L16 supplement |
|
|
111
117
|
| Validated skill improvement | Reviewed recurring hindsight; proposal adapter; `CAIRN_EVAL=1` for evaluation | No server restart; explicit subprocesses only | `cairn skill list|show`; exact-digest apply; concurrent-edit-safe rollback | `course-09-skill` |
|
|
112
118
|
| General meta-agent loop | Not shipped | Not applicable | Playbooks select bounded existing actions; no generic configuration loop, command language, or scheduler | `course-07-evaluation` |
|
|
113
119
|
| Pack retrieval and skills | `CAIRN_CONTEXT_PACKS=1`; HTTP needs separate consent; skills need exact-digest approval | Restart MCP server | `cairn pack skills`; revoke approval or disable pack; no automatic execution | `course-10-trust-context` |
|
|
@@ -19,6 +19,8 @@ report, and explain what a one-capability ablation can and cannot establish.
|
|
|
19
19
|
- Eligible pairs, missingness, compatible turn semantics, reported token/cost
|
|
20
20
|
fields, bootstrap intervals, and bounded reports.
|
|
21
21
|
- `offline-framework` evidence versus separately validated live evidence.
|
|
22
|
+
- Genuine memory calls versus narrated pseudo-calls; protocol ordering,
|
|
23
|
+
interpretable results, consent assertions and inconclusive indirect effects.
|
|
22
24
|
- Dry validation, invocation estimates, explicit `--yes`, cancellation, report
|
|
23
25
|
retention, dry-run pruning, and deletion.
|
|
24
26
|
|
|
@@ -51,6 +53,27 @@ after the lab is rehearsed from the packaged CLI rather than a source clone.
|
|
|
51
53
|
- The learner reports estimates, intervals, missingness, and evidence scope
|
|
52
54
|
without claiming causality, significance, or product improvement.
|
|
53
55
|
- Course cleanup is limited to its temporary contained experiment root.
|
|
56
|
+
- The protocol positive control passes and narration fails; neither is
|
|
57
|
+
described as live model improvement or task correctness.
|
|
58
|
+
|
|
59
|
+
## Supplement: memory protocol evidence
|
|
60
|
+
|
|
61
|
+
This supplement requires Cairnkeep 2.22.0 or newer for `eval protocol` and
|
|
62
|
+
`--codex-jsonl`. Evaluation remains default off.
|
|
63
|
+
|
|
64
|
+
Run the two synthetic commands in the
|
|
65
|
+
[memory protocol guide](../../agent-memory-protocol.md). Pause on the negative
|
|
66
|
+
control's exit `1`: saying a tool was used is not evidence of using it. Explain
|
|
67
|
+
why an empty real search can pass retrieval, why incomplete capture or failed
|
|
68
|
+
retrieval is inconclusive, and why shell calls prevent certification of
|
|
69
|
+
indirect memory writes. Never publish a real trajectory export for this lab.
|
|
70
|
+
|
|
71
|
+
Repeat with the packaged `protocol-codex.jsonl` synthetic control using
|
|
72
|
+
`--codex-jsonl`, not `--trajectory`. Explain start/completion correlation,
|
|
73
|
+
unknown-event gaps, single-turn boundaries and the distinction between a
|
|
74
|
+
private live export and a public parser fixture. Internal rollout files and
|
|
75
|
+
automatic Codex capture are not supported. Pair protocol observations with an
|
|
76
|
+
independent artifact verifier; either dimension can fail separately.
|
|
54
77
|
|
|
55
78
|
## Source material
|
|
56
79
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# L28 - Verify Cairnkeep's security posture
|
|
2
2
|
|
|
3
3
|
**Status:** Ready
|
|
4
|
-
**Tested with:** Cairnkeep 2.
|
|
4
|
+
**Tested with:** Cairnkeep 2.22.0 and Node.js 22 or newer
|
|
5
5
|
|
|
6
6
|
## Outcome
|
|
7
7
|
|
|
@@ -11,7 +11,7 @@ and choose least authority for a network-facing MCP client.
|
|
|
11
11
|
|
|
12
12
|
## Prerequisites
|
|
13
13
|
|
|
14
|
-
- Install Cairnkeep 2.
|
|
14
|
+
- Install Cairnkeep 2.22.0 or use a disposable source checkout.
|
|
15
15
|
- Complete [L20 - Least-authority MCP tool profiles](L20-mcp-tool-profiles.md).
|
|
16
16
|
- Use synthetic values only; never paste a real bearer token into a recording,
|
|
17
17
|
terminal transcript, or issue.
|
|
@@ -20,5 +20,9 @@ reports without making unsupported claims, apply or roll back an exact eligible
|
|
|
20
20
|
skill proposal, and distinguish that narrow runtime from the design-only
|
|
21
21
|
general meta-agent contract.
|
|
22
22
|
|
|
23
|
+
L16 also covers the offline memory protocol controls: distinguish direct tool
|
|
24
|
+
events from narration and explain why protocol observations, caller-asserted
|
|
25
|
+
consent and task-quality evidence are separate claims.
|
|
26
|
+
|
|
23
27
|
Hands-on spine: `course-05-evidence` through `course-07-evaluation` in the
|
|
24
28
|
public [course labs](https://github.com/cairnkeep/cairnkeep-course-labs).
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# V16 supplement — Show the tool call, not the claim
|
|
2
|
+
|
|
3
|
+
Target length: 4–6 minutes. Experienced-developer audience. Record with
|
|
4
|
+
Cairnkeep 2.22.0 or newer and opt in to `cairn eval protocol` explicitly.
|
|
5
|
+
Use only committed synthetic controls, never private transcripts.
|
|
6
|
+
|
|
7
|
+
1. Split the terminal: show `protocol-narration.json` beside its audit. Type the
|
|
8
|
+
command from the memory protocol guide at human speed. Pause two seconds
|
|
9
|
+
on `project_search_absent` and exit `1`: narration is not execution.
|
|
10
|
+
2. Switch to `protocol-project-search.json`. Highlight the correlated IDs and
|
|
11
|
+
project scope. Run the audit and pause on the separate checks. Zero hits is
|
|
12
|
+
acceptable; it is not a reason to keep widening the query.
|
|
13
|
+
3. Show the four exit states. Default-off exit `0` is not a successful audit.
|
|
14
|
+
Incomplete capture, failed retrieval and opaque shell commands yield
|
|
15
|
+
inconclusive observations, not a fabricated perfect score.
|
|
16
|
+
4. Zoom into digests and counts, then back to the input: the report omits
|
|
17
|
+
private content. A digest authenticates neither a publisher nor human
|
|
18
|
+
approval. `--capture-authorized` is an assertion, not a bypass button.
|
|
19
|
+
5. Switch to `protocol-codex.jsonl`, then run `--codex-jsonl`. Visually pair
|
|
20
|
+
`item.started` and `item.completed`; updates are not extra calls. Explain
|
|
21
|
+
single-turn boundaries and unknown events. Keep private exports off screen.
|
|
22
|
+
6. End on the independent verifier, not a green audit. This parser lab does not
|
|
23
|
+
prove a better model, correct code, full playbook compliance or superiority
|
|
24
|
+
over a baseline.
|
|
25
|
+
|
|
26
|
+
Vary framing between input, IDs and report codes. Leave breathing room after
|
|
27
|
+
failures and caveats; do not accelerate terminal footage to match speech.
|
|
28
|
+
|
|
29
|
+
For a coding demonstration, show task correctness beside the individual
|
|
30
|
+
protocol checks. Successful retrieval followed by coding commands may produce
|
|
31
|
+
an inconclusive overall audit because shell effects are opaque. Do not edit
|
|
32
|
+
that into a pass or claim an A/B improvement from one run.
|
package/docs/operating.md
CHANGED
|
@@ -1049,6 +1049,10 @@ then checks `./.ai/.env` (or the current environment). It does not inspect a
|
|
|
1049
1049
|
harness's remote HTTP registration. Unconfigured optional dependencies are
|
|
1050
1050
|
skipped; it exits non-zero when the local server probe fails or a configured
|
|
1051
1051
|
dependency (LLM/embedding endpoint, writable store) is unreachable.
|
|
1052
|
+
Embedding probes report only a bounded classification and HTTP status, such as
|
|
1053
|
+
`authorization rejected (HTTP 403)` or `endpoint or model not found (HTTP
|
|
1054
|
+
404)`. Provider response bodies and request identifiers are discarded so a
|
|
1055
|
+
health check cannot echo sensitive upstream diagnostics.
|
|
1052
1056
|
|
|
1053
1057
|
It also checks an existing project-local trajectory database for SQLite
|
|
1054
1058
|
integrity, schema compatibility and index consistency. Because capture is
|
|
@@ -1076,6 +1080,14 @@ machine-level drift; setup itself never performs the sync.
|
|
|
1076
1080
|
|
|
1077
1081
|
### Evaluation harness (opt-in)
|
|
1078
1082
|
|
|
1083
|
+
To inspect actual direct memory tool use in an existing normalized trajectory,
|
|
1084
|
+
use `CAIRN_EVAL=1 cairn eval protocol --trajectory PATH --json`. Native
|
|
1085
|
+
single-turn Codex exec JSONL exports use `--codex-jsonl PATH` instead. This read-only,
|
|
1086
|
+
offline audit returns separate pass/fail/inconclusive checks, emits no recorded
|
|
1087
|
+
content, and does not measure task quality. See the
|
|
1088
|
+
[memory protocol guide](agent-memory-protocol.md) for controls, export privacy,
|
|
1089
|
+
supported harnesses, consent assertions and exit codes.
|
|
1090
|
+
|
|
1079
1091
|
`cairn eval` is a local, serial experiment coordinator. It is disabled unless
|
|
1080
1092
|
`CAIRN_EVAL` is explicitly truthy. The disabled path returns before reading a
|
|
1081
1093
|
task set or adapter configuration, creating a workspace or report, opening a
|
|
@@ -172,6 +172,15 @@ file, so the diagnostic and runtime use the same precedence.
|
|
|
172
172
|
|
|
173
173
|
## Evaluation adapter and report flow
|
|
174
174
|
|
|
175
|
+
`cairn eval protocol` is a separate read-only audit of an explicitly supplied
|
|
176
|
+
normalized trajectory or single-turn Codex exec JSONL export. With evaluation disabled it does not read that
|
|
177
|
+
file. When enabled it performs bounded local reads only: no adapter, command
|
|
178
|
+
execution, network, database, or persistent report. Output is limited to fixed
|
|
179
|
+
codes, counts, harness identity, input/policy digests and limitations. Exports
|
|
180
|
+
can contain private content even after redaction; keep them local. See the
|
|
181
|
+
[memory protocol guide](agent-memory-protocol.md). A capture-authorization flag
|
|
182
|
+
is the caller's assertion, not authenticated consent.
|
|
183
|
+
|
|
175
184
|
`CAIRN_EVAL` is unset/off by default. With it off, `cairn eval` returns a fixed
|
|
176
185
|
disabled result before task-set or adapter reads, workspace/report creation,
|
|
177
186
|
database access, subprocess execution, or network activity. A credential in
|
package/docs/releasing.md
CHANGED
|
@@ -29,6 +29,13 @@ from a workstation.
|
|
|
29
29
|
9. Verify both image digests, SBOM attestations, provenance attestations, and an
|
|
30
30
|
anonymous pull before announcing the release.
|
|
31
31
|
|
|
32
|
+
The native-Windows lanes retain stable check names for branch protection. Each
|
|
33
|
+
lane exercises the Windows CLI and packed global install on its Node version,
|
|
34
|
+
while the complete memory-server smoke catalog is deterministically balanced
|
|
35
|
+
across the three lanes. The Linux Node 22/24/26 matrix independently runs the
|
|
36
|
+
complete catalog on every supported runtime. This preserves coverage while
|
|
37
|
+
removing three serial copies of the slow Windows suite from the critical path.
|
|
38
|
+
|
|
32
39
|
Stable versions must use a stable GitHub Release and publish to npm's `latest`
|
|
33
40
|
tag. SemVer prerelease versions must use a GitHub prerelease and publish to
|
|
34
41
|
`next`. A mismatch fails before publication.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Bounded release-ledger task
|
|
2
|
+
|
|
3
|
+
Use a disposable project, Node standard APIs and ESM only. Implement
|
|
4
|
+
`src/release-ledger.mjs` exporting `createLedger()`. The returned API exposes
|
|
5
|
+
`append(input)`, `finalize(packageName, version)`, `get(packageName, version)`
|
|
6
|
+
and `list()`.
|
|
7
|
+
|
|
8
|
+
- Package names match `^[a-z][a-z0-9-]*$`; versions are numeric x.y.z. Notes
|
|
9
|
+
are non-empty arrays of non-empty strings, preserving order. Invalid input
|
|
10
|
+
throws Error with code `INVALID_PACKAGE`, `INVALID_VERSION` or `INVALID_NOTES`.
|
|
11
|
+
- `append({package,version,notes})` creates a draft with monotonically
|
|
12
|
+
increasing sequence starting at 1. A pair is unique forever; duplicates
|
|
13
|
+
throw `DUPLICATE_RELEASE`.
|
|
14
|
+
- Entries have exactly package, version, notes, sequence, state and optional
|
|
15
|
+
checksum. Draft state is `draft`; finalized state is `final`.
|
|
16
|
+
- `finalize` is idempotent. Checksum is lowercase SHA-256 of
|
|
17
|
+
`JSON.stringify({package,version,notes,sequence})` in that property order.
|
|
18
|
+
Missing releases throw `NOT_FOUND`.
|
|
19
|
+
- `get` and `list` return detached copies including nested notes. `list` is
|
|
20
|
+
sequence ordered. Add and run self-tests without changing this brief.
|
|
21
|
+
|
|
22
|
+
After the task closes, run the maintained independent grader from outside the
|
|
23
|
+
project, with an external process timeout:
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
node scripts/spikes/verify-release-ledger.mjs /PATH/TO/LAB/src/release-ledger.mjs
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This imports and executes the selected artifact; it is not a safe reader for
|
|
30
|
+
arbitrary untrusted code. Use a lab with no production secrets or services.
|
|
31
|
+
The grader checks 15 bounded behaviors, not all possible inputs or security.
|
|
32
|
+
Do not expose its implementation to the task before grading. A model could
|
|
33
|
+
deliberately game any visible rubric; this is a correctness smoke test, not
|
|
34
|
+
adversarial certification. Run protocol observation separately using the
|
|
35
|
+
[memory audit](../agent-memory-protocol.md). Keep raw exports private; publish
|
|
36
|
+
only appropriately reviewed aggregate evidence. One run is not an A/B study.
|
|
@@ -86,6 +86,10 @@ punctuation.
|
|
|
86
86
|
`cairn doctor` remains the operational dependency/store check. Run both before
|
|
87
87
|
exposing HTTP, after changing tool authority, and before a security-sensitive
|
|
88
88
|
release.
|
|
89
|
+
Its embedding probe retains only a sanitized failure class and HTTP status;
|
|
90
|
+
provider response bodies and request identifiers are never emitted. This makes
|
|
91
|
+
expired credentials distinguishable from a missing model without turning the
|
|
92
|
+
doctor output into a secondary secret or telemetry channel.
|
|
89
93
|
|
|
90
94
|
## Repository and release controls
|
|
91
95
|
|
|
@@ -120,3 +124,7 @@ The maintained gates are `npm run security:baseline`, both production
|
|
|
120
124
|
security-assurance and HTTP-guard smoke tests, native-Windows tests, and the
|
|
121
125
|
complete public test matrix. The baseline gate also rejects
|
|
122
126
|
`pull_request_target` and every unpinned non-local GitHub Action.
|
|
127
|
+
The native-Windows matrix runs the CLI, package, and global-install contracts on
|
|
128
|
+
Node 22, 24, and 26. It deterministically partitions the complete memory-server
|
|
129
|
+
smoke catalog across those lanes exactly once; the Linux matrix still runs the
|
|
130
|
+
complete catalog independently on every supported Node version.
|
package/examples/eval/README.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
|
1
1
|
# Offline evaluation fixture
|
|
2
2
|
|
|
3
|
+
The separate `protocol-project-search.json`, `protocol-narration.json` and
|
|
4
|
+
native-format `protocol-codex.jsonl`
|
|
5
|
+
controls exercise `cairn eval protocol`, without running an adapter. See the
|
|
6
|
+
[memory protocol guide](../../docs/agent-memory-protocol.md). These synthetic
|
|
7
|
+
controls demonstrate parser behavior, not live model quality.
|
|
8
|
+
|
|
9
|
+
`scripts/spikes/verify-release-ledger.mjs MODULE_PATH` separately grades the
|
|
10
|
+
bounded release-ledger coding task with 15 deterministic artifact checks. It
|
|
11
|
+
imports and executes the explicitly selected module: use an isolated lab and
|
|
12
|
+
an external timeout, never untrusted production files. Its score is not a
|
|
13
|
+
protocol score, a comprehensive security audit or an A/B comparison.
|
|
14
|
+
The [task brief](../../docs/research/release-ledger-task.md) defines the lab.
|
|
15
|
+
|
|
3
16
|
This directory contains Cairnkeep's deterministic, network-free evaluation
|
|
4
17
|
population. It validates the evaluation framework; its designed differences are
|
|
5
18
|
not performance, efficiency, quality, or live-product evidence.
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"identifier":"cairn-offline-fake-v1","package_version":"2.
|
|
1
|
+
{"identifier":"cairn-offline-fake-v1","package_version":"2.22.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
{"type":"thread.started","thread_id":"synthetic-control"}
|
|
2
|
+
{"type":"turn.started"}
|
|
3
|
+
{"type":"item.started","item":{"id":"a","type":"mcp_tool_call","server":"cairn-memory","tool":"memory_search","arguments":{"scope":"project","query":"release ledger"},"status":"in_progress"}}
|
|
4
|
+
{"type":"item.completed","item":{"id":"a","type":"mcp_tool_call","server":"cairn-memory","tool":"memory_search","arguments":{"scope":"project","query":"release ledger"},"status":"completed","result":{"content":[],"structured_content":{"mode":"substring","count":0,"results":[]}}}}
|
|
5
|
+
{"type":"turn.completed","usage":{"input_tokens":10,"cached_input_tokens":0,"output_tokens":1}}
|