@cairnkeep/cli 2.19.0 → 2.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -1
- package/docs/README.md +3 -0
- package/docs/ecosystem.md +33 -0
- package/docs/harness-compatibility.md +1 -1
- package/docs/learning/CURRICULUM-MAP.md +4 -2
- package/docs/learning/FEATURE-GUIDE.md +15 -1
- package/docs/learning/README.md +1 -0
- package/docs/learning/lessons/L00-why-cairnkeep.md +1 -1
- package/docs/learning/lessons/L01-safe-trial.md +1 -1
- package/docs/learning/lessons/L02-installation.md +1 -1
- package/docs/learning/lessons/L03-first-project.md +1 -1
- package/docs/learning/lessons/L19-validated-skill-improvement.md +1 -1
- package/docs/learning/lessons/L20-mcp-tool-profiles.md +1 -1
- package/docs/learning/lessons/L21-context-packs.md +1 -1
- package/docs/learning/lessons/L22-native-windows.md +1 -1
- package/docs/learning/lessons/L23-guided-setup.md +1 -1
- package/docs/learning/lessons/L24-okf-exchange.md +1 -1
- package/docs/learning/lessons/L25-playbooks.md +1 -1
- package/docs/learning/lessons/L27-ecosystem-evidence.md +93 -0
- package/docs/learning/video-scripts/V27-ecosystem-evidence.md +80 -0
- package/docs/operating.md +10 -4
- package/docs/privacy-and-data-flow.md +13 -2
- package/docs/research/ecosystem-evidence-2026-10.md +113 -0
- package/examples/eval/README.md +7 -0
- package/examples/eval/bundled-fake.json +1 -1
- package/examples/eval/ecosystem-decisions.json +42 -0
- package/mcp-memory-server/dist/compaction-normalize.js +37 -6
- package/mcp-memory-server/package.json +1 -1
- package/mcp-memory-server/scripts/fixtures/compaction-opencode-2.0.24.json +40 -0
- package/opencode/plugins/memory-capture.ts +354 -120
- package/opencode/plugins/memory-recall.ts +176 -57
- package/opencode/plugins/memory-wakeup.ts +147 -73
- package/package.json +2 -2
- package/scripts/lib/opencode-v2-plugin-harness.mjs +156 -0
- package/scripts/lib/trajectory-opencode-plugin-harness.mjs +6 -2
- package/scripts/spikes/run-clm-pi-evaluation.mjs +212 -0
- package/scripts/spikes/run-memfork-evaluation.mjs +153 -0
- package/scripts/spikes/run-ollaya-decision-evaluation.mjs +108 -0
- package/scripts/test-compaction-hooks.sh +6 -2
- package/scripts/test-ecosystem-spikes.sh +36 -0
- package/scripts/test-opencode-v2-plugins.sh +76 -0
- package/scripts/test-package-install.sh +9 -2
- package/scripts/verify-dependency-security.mjs +20 -2
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,36 @@ All notable user-facing changes are documented here. This project follows
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [2.20.0] - 2026-10-06
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- Support OpenCode 2.0.24's plugin definition, context hook, tool hook, event
|
|
13
|
+
stream, transcript union, and native compaction payload while retaining the
|
|
14
|
+
OpenCode 1.x plugin contract.
|
|
15
|
+
- Add a live OpenCode v2 loader gate plus deterministic trajectory,
|
|
16
|
+
pre-edit recall, session context, compaction capture, and recovery coverage.
|
|
17
|
+
- Ship operator-invoked, disposable comparison runners for MemFork, Pi CLM,
|
|
18
|
+
and Ollaya decision models, with a labelled Cairnkeep trust-decision set.
|
|
19
|
+
|
|
20
|
+
### Documentation
|
|
21
|
+
|
|
22
|
+
- Record the measured ecosystem comparisons and their authority boundaries:
|
|
23
|
+
branchable agent state can complement reviewed memory; CLM can reduce
|
|
24
|
+
long-horizon context but remains experimental and provider-cost-sensitive;
|
|
25
|
+
learned decision models cannot replace deterministic permission policy.
|
|
26
|
+
|
|
27
|
+
## [2.19.1] - 2026-10-06
|
|
28
|
+
|
|
29
|
+
### Security
|
|
30
|
+
|
|
31
|
+
- Upgrade the MCP SDK floor to 1.32.1 and refresh both production lock graphs
|
|
32
|
+
so `fast-uri` 3.1.8, `hono` 4.13.13, `ip-address` 10.7.3, and
|
|
33
|
+
`proxy-addr` 2.0.8 exclude the newly disclosed URI-normalization, request
|
|
34
|
+
parsing, SSRF, denial-of-service, XSS, and proxy-trust advisories.
|
|
35
|
+
- Extend the dependency-security gate to reject future lockfile regressions
|
|
36
|
+
below those patched transitive versions.
|
|
37
|
+
|
|
8
38
|
## [2.19.0] - 2026-09-28
|
|
9
39
|
|
|
10
40
|
### Added
|
|
@@ -572,7 +602,9 @@ stored memories.
|
|
|
572
602
|
- Preserve executable permissions and Bash 3.2 portability on macOS.
|
|
573
603
|
- Add backup-first uninstall and SQLite-safe memory export/import guidance.
|
|
574
604
|
|
|
575
|
-
[Unreleased]: https://github.com/cairnkeep/cairnkeep/compare/v2.
|
|
605
|
+
[Unreleased]: https://github.com/cairnkeep/cairnkeep/compare/v2.20.0...HEAD
|
|
606
|
+
[2.20.0]: https://github.com/cairnkeep/cairnkeep/compare/v2.19.1...v2.20.0
|
|
607
|
+
[2.19.1]: https://github.com/cairnkeep/cairnkeep/compare/v2.19.0...v2.19.1
|
|
576
608
|
[2.19.0]: https://github.com/cairnkeep/cairnkeep/compare/v2.18.0...v2.19.0
|
|
577
609
|
[2.18.0]: https://github.com/cairnkeep/cairnkeep/compare/v2.17.3...v2.18.0
|
|
578
610
|
[2.15.1]: https://github.com/cairnkeep/cairnkeep/compare/v2.15.0...v2.15.1
|
package/docs/README.md
CHANGED
|
@@ -63,6 +63,9 @@ linked guides rather than the project landing page.
|
|
|
63
63
|
and exporting portable knowledge bundles.
|
|
64
64
|
- [Companion tools and related projects](ecosystem.md) lists optional tools that
|
|
65
65
|
complement Cairnkeep without becoming runtime dependencies.
|
|
66
|
+
- [Ecosystem evaluation evidence](research/ecosystem-evidence-2026-10.md)
|
|
67
|
+
records the reproducible MemFork, CLM/Pi, and Ollaya comparison behind those
|
|
68
|
+
boundaries.
|
|
66
69
|
|
|
67
70
|
## Overlays and teams
|
|
68
71
|
|
package/docs/ecosystem.md
CHANGED
|
@@ -20,3 +20,36 @@ graph workflows have their own explicit companion requirements.
|
|
|
20
20
|
These integrations are accelerators, not part of Cairnkeep's trust boundary or
|
|
21
21
|
runtime dependency set. Review each project's installation and data-flow
|
|
22
22
|
documentation independently before enabling it.
|
|
23
|
+
|
|
24
|
+
## Context and agent-state projects
|
|
25
|
+
|
|
26
|
+
These projects overlap with one layer of Cairnkeep but do not supersede its
|
|
27
|
+
reviewed-memory and authority model.
|
|
28
|
+
|
|
29
|
+
| Project | Strongest capability | Relationship to Cairnkeep | Current decision |
|
|
30
|
+
|---|---|---|---|
|
|
31
|
+
| [OpenViking](https://github.com/volcengine/OpenViking) | Hierarchical external context retrieval | Complements project memory; Cairnkeep already offers an explicitly configured, read-only provider boundary | Keep optional and read-only; see [context intelligence](context-intelligence.md) |
|
|
32
|
+
| [MemFork](https://github.com/memforkdb/memfork) | Branch, merge, discard, time travel, and cross-agent handoff for mutable agent state | Adds cheap experimental state branches that Cairnkeep's review-gated durable memory intentionally does not model | Do not add as a dependency; consider an optional capability only for disposable agent state |
|
|
33
|
+
| [Context Language Models](https://github.com/facebookresearch/context-language-models) and [pi-clm](https://github.com/lolipopshock/pi-clm) | Model-managed projection of the current conversation | Operates inside a session; Cairnkeep preserves reviewed context across sessions and harnesses | Experimental Pi companion only; never a durable-memory replacement |
|
|
34
|
+
| [Ollaya](https://github.com/ollaya-dev/ollaya) | Fast local typed decision models | May rank ambiguous suggestions, but Cairnkeep permissions and promotion rules are deterministic policy | Advisory experiments only; never authorization or security enforcement |
|
|
35
|
+
|
|
36
|
+
The architectural split is deliberate:
|
|
37
|
+
|
|
38
|
+
- repository sources and reviewed documentation remain authoritative;
|
|
39
|
+
- Cairnkeep memory locates and carries reviewed project knowledge;
|
|
40
|
+
- context projection may reduce the current model request;
|
|
41
|
+
- branchable agent state may isolate speculative work;
|
|
42
|
+
- deterministic code, not a learned classifier, enforces permissions.
|
|
43
|
+
|
|
44
|
+
This also addresses the useful criticism that agents often need better access
|
|
45
|
+
to source context rather than an ever-growing opaque memory store. Cairnkeep
|
|
46
|
+
does not ask users to trust remembered prose over maintained sources: search
|
|
47
|
+
results are locators, provenance is explicit, and contradictions must be
|
|
48
|
+
resolved against the repository.
|
|
49
|
+
|
|
50
|
+
## Measured comparison
|
|
51
|
+
|
|
52
|
+
The pinned October 2026 comparison and exact reproduction commands are in
|
|
53
|
+
[ecosystem evaluation evidence](research/ecosystem-evidence-2026-10.md).
|
|
54
|
+
The runners are operator-invoked only. They are absent from CI, install no
|
|
55
|
+
runtime dependency, and use disposable stores and workspaces.
|
|
@@ -15,7 +15,7 @@ Cairnkeep commands, hooks or native trajectory capture.
|
|
|
15
15
|
| Harness | Supported surface | Runtime status |
|
|
16
16
|
|---|---|---|
|
|
17
17
|
| Claude Code | Memory MCP, commands (including `/cairn-work`), agents, hooks, launcher | Exercised by Cairnkeep tests |
|
|
18
|
-
| OpenCode | Memory MCP, commands (including `/cairn-work`), workflows, plugins, launcher |
|
|
18
|
+
| OpenCode | Memory MCP, commands (including `/cairn-work`), workflows, plugins, launcher | OpenCode 1.x contract-tested; OpenCode 2.0.24 contract-tested and live-loader tested |
|
|
19
19
|
| Kimi Code | Memory MCP, `AGENTS.md`, launcher, opt-in graph and cairn-work Skills | Launcher tested; remote MCP tested with Kimi Code 0.30.0; Skills contract-tested |
|
|
20
20
|
| Qwen Code | Memory MCP, launcher, project `AGENTS.md` durable-context and playbook guidance | Launcher tested; stdio and remote MCP tested with Qwen Code 0.21.1 |
|
|
21
21
|
| Pi | Memory MCP through maintained local stdio extension, native opt-in trajectory extension, launcher, graph and cairn-work prompts | Pi 0.84.1 validated minimum; deterministic and real bridge/lifecycle tests |
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Curriculum Coverage Map
|
|
2
2
|
|
|
3
|
-
**Baseline:** Cairnkeep 2.
|
|
4
|
-
**Last reviewed:** 2026-
|
|
3
|
+
**Baseline:** Cairnkeep 2.20.0
|
|
4
|
+
**Last reviewed:** 2026-10-06
|
|
5
5
|
|
|
6
6
|
This map prevents the course from drifting behind the product. It assigns each
|
|
7
7
|
user-facing surface to a lesson and records whether the lesson is complete or
|
|
@@ -42,6 +42,7 @@ been rehearsed.
|
|
|
42
42
|
| OKF 0.1/0.2 validation/import, provenance, links, regressions, and reviewed 0.2 export | L24 | L24 | Ready; local/offline by default |
|
|
43
43
|
| Bounded project playbooks, generic memory retrieval guidance, enforcement, and private receipts | L25 | L25 | Ready; balanced/local by default; invocation requires runtime evidence; actor identity unverified |
|
|
44
44
|
| Progressive pack retrieval, frozen benchmark, context-usage receipts, review-gated memory proposals, optional provider adapter | L26 | V26 | Ready; default flat search and AnythingLLM remain compatible; OpenViking and receipt mutation are explicitly gated |
|
|
45
|
+
| Branchable agent state, model-managed context, and learned decision-model comparisons | L27 | V27 | Ready; evidence runners are operator-invoked and no companion is a runtime dependency or authority |
|
|
45
46
|
|
|
46
47
|
## Release Review
|
|
47
48
|
|
|
@@ -76,6 +77,7 @@ repository remains the canonical source for explanations and scripts.
|
|
|
76
77
|
| `course-11-windows` | L22 | Native PowerShell lifecycle, ACL inspection, and reversible uninstall |
|
|
77
78
|
| `course-12-guided-setup` | L23 | Deterministic setup, Pi local stdio, recovery, and backup-first uninstall |
|
|
78
79
|
| `course-13-playbooks` | L25 | Policy profiles, lifecycle decisions, enforcement, receipts, and ownership-safe instructions |
|
|
80
|
+
| Disposable source checkout | L26-L27 | Context retrieval and ecosystem evidence without modifying course projects |
|
|
79
81
|
|
|
80
82
|
## Command Ownership
|
|
81
83
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Cairnkeep feature guide
|
|
2
2
|
|
|
3
|
-
**Baseline:** Cairnkeep 2.
|
|
3
|
+
**Baseline:** Cairnkeep 2.20.0
|
|
4
4
|
|
|
5
5
|
This is the step-back map for choosing and operating Cairnkeep features. The
|
|
6
6
|
[course labs](https://github.com/cairnkeep/cairnkeep-course-labs) provide one
|
|
@@ -24,6 +24,9 @@ Version 2.15.4 makes generic task-derived, project-scoped retrieval the first
|
|
|
24
24
|
tool or command, puts bounded playbook signal values inline, and adds
|
|
25
25
|
event-scoped check help to executable receipt guidance. Retrieval remains a
|
|
26
26
|
locator and actual tool invocation must be verified from runtime evidence.
|
|
27
|
+
Version 2.20 adds OpenCode 2.0.24 plugin and compaction compatibility plus
|
|
28
|
+
operator-invoked ecosystem evidence runners; see
|
|
29
|
+
[L27](lessons/L27-ecosystem-evidence.md).
|
|
27
30
|
|
|
28
31
|
## The three-layer model
|
|
29
32
|
|
|
@@ -104,6 +107,7 @@ prerequisites for ordinary memory.
|
|
|
104
107
|
| Validated skill improvement | Reviewed recurring hindsight; proposal adapter; `CAIRN_EVAL=1` for evaluation | No server restart; explicit subprocesses only | `cairn skill list|show`; exact-digest apply; concurrent-edit-safe rollback | `course-09-skill` |
|
|
105
108
|
| General meta-agent loop | Not shipped | Not applicable | Playbooks select bounded existing actions; no generic configuration loop, command language, or scheduler | `course-07-evaluation` |
|
|
106
109
|
| Pack retrieval and skills | `CAIRN_CONTEXT_PACKS=1`; HTTP needs separate consent; skills need exact-digest approval | Restart MCP server | `cairn pack skills`; revoke approval or disable pack; no automatic execution | `course-10-trust-context` |
|
|
110
|
+
| Ecosystem evidence runners | Explicit script invocation plus caller-supplied binaries/model configuration | None; disposable processes only | Temporary state is removed by default; inspect the JSON report and pinned evidence | Disposable source checkout, L27 |
|
|
107
111
|
|
|
108
112
|
## Harness and maintenance checks
|
|
109
113
|
|
|
@@ -139,3 +143,13 @@ L26 joins five related surfaces without collapsing their authority boundaries:
|
|
|
139
143
|
See [L26](lessons/L26-context-intelligence.md), its
|
|
140
144
|
[video script](video-scripts/V26-context-intelligence.md), and the
|
|
141
145
|
[context-intelligence reference](../context-intelligence.md).
|
|
146
|
+
|
|
147
|
+
## Companion evaluation
|
|
148
|
+
|
|
149
|
+
L27 keeps adjacent tools in separate layers. MemFork is evidence for
|
|
150
|
+
branchable speculative state, CLM projects the current request, and Ollaya can
|
|
151
|
+
produce advisory typed decisions. None receives Cairnkeep's durable-memory,
|
|
152
|
+
approval, project-isolation, or release authority. See
|
|
153
|
+
[L27](lessons/L27-ecosystem-evidence.md), its
|
|
154
|
+
[video script](video-scripts/V27-ecosystem-evidence.md), and the pinned
|
|
155
|
+
[evidence report](../research/ecosystem-evidence-2026-10.md).
|
package/docs/learning/README.md
CHANGED
|
@@ -61,6 +61,7 @@ expanded only after their labs are executable.
|
|
|
61
61
|
| L24 | [Reviewed knowledge exchange](lessons/L24-okf-exchange.md) | Ready | 35 min |
|
|
62
62
|
| L25 | [Bounded workflow playbooks](lessons/L25-playbooks.md) | Ready | 35 min |
|
|
63
63
|
| L26 | [Context intelligence with explicit authority](lessons/L26-context-intelligence.md) | Ready | 45 min |
|
|
64
|
+
| L27 | [Evaluate context companions without moving authority](lessons/L27-ecosystem-evidence.md) | Ready | 30 min |
|
|
64
65
|
|
|
65
66
|
The [curriculum coverage map](CURRICULUM-MAP.md) records where every public
|
|
66
67
|
feature is introduced, practised, and operated. It also marks design-only work
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# L27 - Evaluate context companions without moving authority
|
|
2
|
+
|
|
3
|
+
**Tested with:** Cairnkeep 2.20.0 and Node.js 22 or newer
|
|
4
|
+
|
|
5
|
+
## Outcome
|
|
6
|
+
|
|
7
|
+
You will distinguish durable reviewed memory, current-request context,
|
|
8
|
+
speculative agent state, external retrieval, and learned decisions. You will
|
|
9
|
+
also inspect reproducible evidence without turning an interesting companion
|
|
10
|
+
project into a Cairnkeep dependency or security authority.
|
|
11
|
+
|
|
12
|
+
## Prerequisites
|
|
13
|
+
|
|
14
|
+
- Complete [L16 - Evaluation and ablation](L16-evaluation.md) and
|
|
15
|
+
[L26 - Context intelligence](L26-context-intelligence.md).
|
|
16
|
+
- Use a disposable Cairnkeep source checkout.
|
|
17
|
+
- External binaries, models, and paid providers are optional. The offline
|
|
18
|
+
contract check requires none of them.
|
|
19
|
+
|
|
20
|
+
## 1. Start with the five layers
|
|
21
|
+
|
|
22
|
+
Use a different question for each layer:
|
|
23
|
+
|
|
24
|
+
| Layer | Question | Cairnkeep boundary |
|
|
25
|
+
|---|---|---|
|
|
26
|
+
| Maintained source | What is true now? | Repository and reviewed docs are authoritative |
|
|
27
|
+
| Durable memory | What prior decision may matter? | Search locates; verify against source |
|
|
28
|
+
| Current context | What must this model request retain? | A harness or CLM may project it temporarily |
|
|
29
|
+
| Speculative state | What should this experimental agent branch see? | A branchable store may isolate and rewind it |
|
|
30
|
+
| Policy decision | May this action execute? | Deterministic capability and approval code decides |
|
|
31
|
+
|
|
32
|
+
This separation prevents a context optimizer, memory database, or classifier
|
|
33
|
+
from silently becoming an authorization system.
|
|
34
|
+
|
|
35
|
+
## 2. Verify the frozen local contract
|
|
36
|
+
|
|
37
|
+
Run the network-free repository check:
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
scripts/test-ecosystem-spikes.sh
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
It syntax-checks every runner, validates 32 unique balanced decision cases,
|
|
44
|
+
recomputes their deterministic labels, and checks that the published evidence
|
|
45
|
+
matches the frozen results. It does not launch MemFork, Pi, Ollaya, a model, or
|
|
46
|
+
the network.
|
|
47
|
+
|
|
48
|
+
## 3. Read the evidence before the conclusion
|
|
49
|
+
|
|
50
|
+
Open [the October 2026 evidence report](../../research/ecosystem-evidence-2026-10.md)
|
|
51
|
+
and answer:
|
|
52
|
+
|
|
53
|
+
1. Which MemFork behavior is genuinely absent from Cairnkeep?
|
|
54
|
+
2. Did CLM improve the hidden verifier score, token use, cost, or some subset?
|
|
55
|
+
3. Which destructive or forbidden cases did the learned decision model miss?
|
|
56
|
+
4. Which result is a single-task observation rather than a product claim?
|
|
57
|
+
|
|
58
|
+
The correct interpretation is deliberately mixed. MemFork demonstrated useful
|
|
59
|
+
branching. CLM preserved quality with fewer total tokens but higher provider
|
|
60
|
+
cost in one run. The learned classifier was fast but unsafe as a guardrail.
|
|
61
|
+
|
|
62
|
+
## 4. Reproduce only the experiment you need
|
|
63
|
+
|
|
64
|
+
Each runner requires an explicit external binary or disposable Pi home:
|
|
65
|
+
|
|
66
|
+
```sh
|
|
67
|
+
MEMFORK_BIN=/path/to/memfork \
|
|
68
|
+
node scripts/spikes/run-memfork-evaluation.mjs --version 0.3.0
|
|
69
|
+
|
|
70
|
+
OLLAYA_BIN=/path/to/ollaya \
|
|
71
|
+
node scripts/spikes/run-ollaya-decision-evaluation.mjs \
|
|
72
|
+
--runtime-version 0.12.0 --model decima:agent
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
The longer Pi command is documented in the evidence report. It makes model
|
|
76
|
+
calls and may incur provider cost. Use an isolated home and never install the
|
|
77
|
+
CLM extension into a daily harness merely to run the comparison.
|
|
78
|
+
|
|
79
|
+
## Acceptance
|
|
80
|
+
|
|
81
|
+
- `scripts/test-ecosystem-spikes.sh` passes without network access.
|
|
82
|
+
- You can place each project in the correct layer without calling it a
|
|
83
|
+
Cairnkeep replacement.
|
|
84
|
+
- You do not infer a default integration from one benchmark.
|
|
85
|
+
- Learned output remains advisory and independently verified; deterministic
|
|
86
|
+
code retains permission, project-isolation, promotion, and release gates.
|
|
87
|
+
|
|
88
|
+
## Recovery
|
|
89
|
+
|
|
90
|
+
The runners delete temporary state by default. If you used `--keep-data` or
|
|
91
|
+
`--keep-workspaces`, inspect the printed exact temporary path, retain only
|
|
92
|
+
sanitized evidence you need, then remove that specific directory. Stop any
|
|
93
|
+
separately launched local model service through its own supported command.
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# V27 - Context companions: measured, not absorbed
|
|
2
|
+
|
|
3
|
+
Target length: 10–12 minutes. Audience: experienced developers evaluating
|
|
4
|
+
agent-memory and context-management tooling.
|
|
5
|
+
|
|
6
|
+
## Recording outline
|
|
7
|
+
|
|
8
|
+
### 0:00–1:15 — Draw the layers
|
|
9
|
+
|
|
10
|
+
Animate five labelled cards: maintained source, reviewed durable memory,
|
|
11
|
+
current model context, speculative agent state, deterministic policy. Keep the
|
|
12
|
+
cards visible as a small sidebar for the rest of the video.
|
|
13
|
+
|
|
14
|
+
Narration:
|
|
15
|
+
|
|
16
|
+
> Similar-looking tools often solve different time scales. The useful question
|
|
17
|
+
> is not which project wins, but which layer it improves without inheriting
|
|
18
|
+
> authority it should not have.
|
|
19
|
+
|
|
20
|
+
### 1:15–3:30 — MemFork in a disposable store
|
|
21
|
+
|
|
22
|
+
Type the real command at human speed and pause on the six booleans:
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
MEMFORK_BIN=/path/to/memfork \
|
|
26
|
+
node scripts/spikes/run-memfork-evaluation.mjs --version 0.3.0
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Show branch isolation, conflict rejection, discard lesson, second-client
|
|
30
|
+
handoff, time travel, and reviewed merge. Move the MemFork card beside
|
|
31
|
+
"speculative agent state", not "durable authority".
|
|
32
|
+
|
|
33
|
+
### 3:30–6:30 — CLM changes the request, not the truth
|
|
34
|
+
|
|
35
|
+
Use a split terminal with the same six-turn release-ledger task. Accelerate
|
|
36
|
+
dead wait time with cuts, but show real commands, the disappearing briefing
|
|
37
|
+
files, and the private context revision count. End on the table:
|
|
38
|
+
|
|
39
|
+
- both arms: 15/15;
|
|
40
|
+
- total tokens: 142,212 versus 51,530;
|
|
41
|
+
- reported cost: $0.021117 versus $0.0543623.
|
|
42
|
+
|
|
43
|
+
Pause after "fewer tokens, higher cost." Explain the artificial 12k budget,
|
|
44
|
+
single task, provider cache pricing, and why this is evidence for another
|
|
45
|
+
experiment rather than default installation.
|
|
46
|
+
|
|
47
|
+
### 6:30–8:45 — A fast classifier is not a guardrail
|
|
48
|
+
|
|
49
|
+
Run:
|
|
50
|
+
|
|
51
|
+
```sh
|
|
52
|
+
OLLAYA_BIN=/path/to/ollaya \
|
|
53
|
+
node scripts/spikes/run-ollaya-decision-evaluation.mjs \
|
|
54
|
+
--runtime-version 0.12.0 --model decima:agent
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Highlight 32/32 for explicit policy and 19/32 for the learned model. Zoom into
|
|
58
|
+
one missed deletion and one missed denial. Move Ollaya beside "advisory
|
|
59
|
+
ranking" and keep deterministic policy attached to the approval gate.
|
|
60
|
+
|
|
61
|
+
### 8:45–10:15 — The product decision
|
|
62
|
+
|
|
63
|
+
Return to all five cards:
|
|
64
|
+
|
|
65
|
+
1. Keep Cairnkeep's reviewed-memory and source-verification contract.
|
|
66
|
+
2. Keep OpenViking read-only and optional.
|
|
67
|
+
3. Consider MemFork only for disposable speculative state.
|
|
68
|
+
4. Re-test CLM across providers before integration.
|
|
69
|
+
5. Never replace permission code with a learned decision.
|
|
70
|
+
|
|
71
|
+
Close on `scripts/test-ecosystem-spikes.sh`, which validates the public fixture
|
|
72
|
+
and evidence offline without launching any companion.
|
|
73
|
+
|
|
74
|
+
## Recording cautions
|
|
75
|
+
|
|
76
|
+
- Show synthetic projects and no real provider key, endpoint, memory, or path.
|
|
77
|
+
- State every version and pin on screen.
|
|
78
|
+
- Do not call a one-task result a benchmark win.
|
|
79
|
+
- Do not imply that the scripts install or enable companions automatically.
|
|
80
|
+
- Let the tables breathe; use pauses rather than faster narration.
|
package/docs/operating.md
CHANGED
|
@@ -233,9 +233,12 @@ Each installs into `~/.config/opencode` (override with `OPENCODE_CONFIG_DIR` or
|
|
|
233
233
|
`--live-root`) and is idempotent — re-run with `--check` to see drift without
|
|
234
234
|
writing.
|
|
235
235
|
|
|
236
|
-
**No Claude install required.** The OpenCode memory
|
|
237
|
-
of Claude assets
|
|
238
|
-
`experimental.chat.system.transform
|
|
236
|
+
**No Claude install required.** The OpenCode memory plugins are self-sufficient
|
|
237
|
+
of Claude assets and never read `~/.claude`. They retain the OpenCode 1.x
|
|
238
|
+
`experimental.chat.system.transform`, tool, and event contracts while also
|
|
239
|
+
implementing OpenCode 2.0.24's `{ id, setup }` definition, `session.hook`,
|
|
240
|
+
`tool.hook`, event subscription, and message-union contracts. Unknown future
|
|
241
|
+
compaction shapes fail open instead of being treated as a verified adapter.
|
|
239
242
|
|
|
240
243
|
### Headless round-trip harness — model precondition
|
|
241
244
|
|
|
@@ -1231,7 +1234,10 @@ cairn sync --apply # Claude Code PostCompact + SessionStart
|
|
|
1231
1234
|
```
|
|
1232
1235
|
|
|
1233
1236
|
Compaction continuity accepts only the pinned Claude Code `PostCompact`
|
|
1234
|
-
2.1.219 and 2.1.220 payloads and OpenCode `session.compacted` 1.17.20
|
|
1237
|
+
2.1.219 and 2.1.220 payloads and OpenCode `session.compacted` 1.17.20 or
|
|
1238
|
+
2.0.24 payloads. OpenCode 2.0.24 carries its summary directly in the verified
|
|
1239
|
+
message union; the adapter selects the newest valid compaction message and
|
|
1240
|
+
does not reinterpret it as the legacy assistant-summary shape.
|
|
1235
1241
|
The Claude hook resolves the local CLI version and passes that exact version to
|
|
1236
1242
|
the normalizer. Unknown or malformed versions fail open, retain none of the
|
|
1237
1243
|
unknown payload, and leave only a
|
|
@@ -294,6 +294,17 @@ subprocess, file/database, network, stdout/stderr, or context-injection work.
|
|
|
294
294
|
Credentials never imply either flag. No model call, telemetry, automatic
|
|
295
295
|
remote discovery, or default egress is introduced.
|
|
296
296
|
|
|
297
|
+
### Ecosystem evaluation runners
|
|
298
|
+
|
|
299
|
+
The scripts under `scripts/spikes/` are operator-invoked research tools, not
|
|
300
|
+
runtime features or CI jobs. The MemFork runner uses only a caller-supplied
|
|
301
|
+
binary and a temporary local store. The CLM/Pi runner creates disposable
|
|
302
|
+
projects and may send their synthetic briefing text to the explicitly chosen
|
|
303
|
+
model provider. The Ollaya runner sends the public labelled fixture to a
|
|
304
|
+
caller-supplied local Ollaya runtime. No runner discovers credentials, installs
|
|
305
|
+
packages, changes harness configuration, or retains its temporary state by
|
|
306
|
+
default. `--keep-workspaces` and `--keep-data` are explicit debugging choices.
|
|
307
|
+
|
|
297
308
|
Claude local capture:
|
|
298
309
|
|
|
299
310
|
```text
|
|
@@ -308,7 +319,7 @@ OpenCode local capture:
|
|
|
308
319
|
|
|
309
320
|
```text
|
|
310
321
|
session.compacted session ID
|
|
311
|
-
→ local SDK get(session) +
|
|
322
|
+
→ local SDK get(session) + context(session)
|
|
312
323
|
→ select the pinned completed summary shape
|
|
313
324
|
→ recursive redaction
|
|
314
325
|
→ truncate / digest / index / write
|
|
@@ -316,7 +327,7 @@ session.compacted session ID
|
|
|
316
327
|
```
|
|
317
328
|
|
|
318
329
|
Only Claude Code `PostCompact` 2.1.219 and 2.1.220 and OpenCode
|
|
319
|
-
`session.compacted` 1.17.20 are pinned. Unknown versions/shapes fail open. Their payload is not logged, serialized,
|
|
330
|
+
`session.compacted` 1.17.20 and 2.0.24 are pinned. Unknown versions/shapes fail open. Their payload is not logged, serialized,
|
|
320
331
|
temporarily copied, digested, indexed, or retained; doctor receives only a
|
|
321
332
|
bounded value-free reason. Artifact candidates are redacted before digest, index, temporary storage, or any Cairnkeep write.
|
|
322
333
|
Truncation also happens only after redaction. The same recursive
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# Ecosystem evaluation evidence — October 2026
|
|
2
|
+
|
|
3
|
+
This report records narrow, reproducible experiments. It is not a general
|
|
4
|
+
benchmark, endorsement, or claim that one project is globally better. Every
|
|
5
|
+
external integration remains optional and outside Cairnkeep's runtime trust
|
|
6
|
+
boundary.
|
|
7
|
+
|
|
8
|
+
## Reproduction boundary
|
|
9
|
+
|
|
10
|
+
| Subject | Pin used | License relevant to the run |
|
|
11
|
+
|---|---|---|
|
|
12
|
+
| MemFork | release `v0.3.0`; source commit `dfc0fd0907b94fa2bfd5a2c5c989e556085ba7d5` | Apache-2.0 |
|
|
13
|
+
| Context Language Models | source commit `18dc11115f50f261233c5bba7937834491e307e8` | CC BY-NC 4.0 research repository |
|
|
14
|
+
| pi-clm | npm `@lolipopshock/pi-clm@1.0.0` | MIT |
|
|
15
|
+
| Ollaya | runtime `0.12.0`; source commit `e352e0ff7086e85f7f115487f77310c13203a7a6` | Apache-2.0 runtime; model licenses remain separate |
|
|
16
|
+
| Ollaya model | `decima:agent`, 321M parameters, fp32 | Apache-2.0 model family |
|
|
17
|
+
|
|
18
|
+
No external source or model is vendored. The CLM research repository's
|
|
19
|
+
non-commercial license is a further reason not to copy its implementation into
|
|
20
|
+
Cairnkeep. The separately published Pi extension was used only in a disposable
|
|
21
|
+
evaluation home.
|
|
22
|
+
|
|
23
|
+
## MemFork: branchable agent state
|
|
24
|
+
|
|
25
|
+
The MCP evaluation used two separately identified clients against one
|
|
26
|
+
temporary store. All six observations passed:
|
|
27
|
+
|
|
28
|
+
- a candidate branch remained isolated from `main`;
|
|
29
|
+
- a conflicting merge with policy `fail` was rejected;
|
|
30
|
+
- discarding the failed branch retained a one-line lesson;
|
|
31
|
+
- a second client resumed the first client's handoff;
|
|
32
|
+
- branch diff and historical `at` reads returned the expected state;
|
|
33
|
+
- the reviewed candidate merged into `main`.
|
|
34
|
+
|
|
35
|
+
This is a meaningful capability Cairnkeep does not provide: cheap branching
|
|
36
|
+
and rewind of mutable agent scratch state. It does not replace Cairnkeep's
|
|
37
|
+
review, provenance, immutable artifacts, project identity, or durable-memory
|
|
38
|
+
authority rules. The recommended boundary is an optional future capability for
|
|
39
|
+
experimental agent state only.
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
MEMFORK_BIN=/path/to/memfork \
|
|
43
|
+
node scripts/spikes/run-memfork-evaluation.mjs --version 0.3.0
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## CLM plus Cairnkeep: one long-horizon Pi task
|
|
47
|
+
|
|
48
|
+
One six-turn release-ledger task ran twice with the same Sol/medium model,
|
|
49
|
+
project setup, Cairnkeep Pi bridge, disappearing source briefings, and hidden
|
|
50
|
+
15-check verifier. The candidate arm added pi-clm with an artificial 12,000
|
|
51
|
+
token budget and its bundled steering brief.
|
|
52
|
+
|
|
53
|
+
| Arm | Hidden verifier | Total model tokens | Reported cost |
|
|
54
|
+
|---|---:|---:|---:|
|
|
55
|
+
| Cairnkeep | 15/15 | 142,212 | $0.021117 |
|
|
56
|
+
| Cairnkeep + pi-clm | 15/15 | 51,530 | $0.0543623 |
|
|
57
|
+
|
|
58
|
+
CLM preserved quality and used about 63.8% fewer total tokens. It cost about
|
|
59
|
+
2.58 times as much on this single OpenRouter run because context editing and
|
|
60
|
+
cache-write behavior added provider-specific overhead. This is promising for
|
|
61
|
+
hard context limits, not evidence for default installation or lower cost.
|
|
62
|
+
The sample size is one task, the budget is artificial, and the comparison must
|
|
63
|
+
be repeated across providers and longer workloads before a product decision.
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
node scripts/spikes/run-clm-pi-evaluation.mjs \
|
|
67
|
+
--cairn /absolute/path/to/bin/cairn \
|
|
68
|
+
--pi /absolute/path/to/pi \
|
|
69
|
+
--home /disposable/pi-home \
|
|
70
|
+
--cairn-extension /disposable/pi-home/.pi/agent/extensions/cairnkeep-memory.ts \
|
|
71
|
+
--clm-extension /disposable/pi-home/.pi/agent/npm/node_modules/@lolipopshock/pi-clm/index.ts \
|
|
72
|
+
--clm-steering /disposable/pi-home/.pi/agent/npm/node_modules/@lolipopshock/pi-clm/steering/house-brief.md \
|
|
73
|
+
--model openrouter/~openai/gpt-sol-latest
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
The runner makes paid/network model calls when the chosen provider requires
|
|
77
|
+
them. It never runs during package tests or CI.
|
|
78
|
+
|
|
79
|
+
## Ollaya: learned decisions versus explicit policy
|
|
80
|
+
|
|
81
|
+
The frozen dataset contains 32 balanced Cairnkeep cases: read-only
|
|
82
|
+
observations, mutations requiring approval, forbidden actions, and tasks where
|
|
83
|
+
memory is irrelevant. Labels are derived from four explicit boolean signals.
|
|
84
|
+
|
|
85
|
+
| Classifier | Correct | Accuracy |
|
|
86
|
+
|---|---:|---:|
|
|
87
|
+
| Deterministic policy | 32/32 | 100% |
|
|
88
|
+
| `decima:agent` | 19/32 | 59.4% |
|
|
89
|
+
|
|
90
|
+
The learned model averaged about 8.0 ms warm, but it missed four of eight
|
|
91
|
+
approval cases and four of eight denial cases. Some wrong answers were
|
|
92
|
+
confident, including treating deletion as no Cairnkeep action. That is
|
|
93
|
+
unacceptable for an authorization boundary.
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
OLLAYA_BIN=/path/to/ollaya \
|
|
97
|
+
node scripts/spikes/run-ollaya-decision-evaluation.mjs \
|
|
98
|
+
--runtime-version 0.12.0 --model decima:agent
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
The evidence supports a narrow use: a local decision model may rank fuzzy,
|
|
102
|
+
non-authoritative suggestions whose result is independently checked. MCP tool
|
|
103
|
+
exposure, mutation approval, secret handling, project isolation, memory
|
|
104
|
+
promotion, and release gates remain deterministic code.
|
|
105
|
+
|
|
106
|
+
## Product decision
|
|
107
|
+
|
|
108
|
+
1. Keep Cairnkeep focused on reviewed, local-first context and documentation.
|
|
109
|
+
2. Retain the optional read-only OpenViking provider already shipped.
|
|
110
|
+
3. Do not add MemFork, CLM, pi-clm, Ollaya, or model weights as dependencies.
|
|
111
|
+
4. Keep the three runners as opt-in evidence tools with disposable state.
|
|
112
|
+
5. Revisit a MemFork provider only if real multi-agent work needs speculative
|
|
113
|
+
state branches; revisit CLM only after multi-provider cost and quality runs.
|
package/examples/eval/README.md
CHANGED
|
@@ -40,3 +40,10 @@ Run only in a disposable checkout when exercising the fixture directly. The
|
|
|
40
40
|
result is always scoped as `offline-framework`; it cannot authorize a README or
|
|
41
41
|
release claim. No provider, endpoint, credential, model service, or network
|
|
42
42
|
default is included.
|
|
43
|
+
|
|
44
|
+
`ecosystem-decisions.json` is a separate balanced, labelled fixture for the
|
|
45
|
+
operator-invoked Ollaya comparison in
|
|
46
|
+
`scripts/spikes/run-ollaya-decision-evaluation.mjs`. It is not a coordinator
|
|
47
|
+
task set and contains no provider or endpoint. The offline
|
|
48
|
+
`scripts/test-ecosystem-spikes.sh` contract recomputes every expected label
|
|
49
|
+
from explicit policy signals before an optional learned model is measured.
|