klypix-mcp 1.58.0 → 1.60.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/BENCHMARKS.md +48 -0
- package/README.md +19 -0
- package/bin/klypix-install.mjs +1 -1
- package/bin/klypix-mcp.mjs +16 -2
- package/package.json +4 -1
- package/src/bench.mjs +323 -0
package/BENCHMARKS.md
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# KLYPIX brain — benchmark
|
|
2
|
+
|
|
3
|
+
Run `npx klypix-mcp bench` to reproduce this table on your own machine.
|
|
4
|
+
|
|
5
|
+
- klypix-mcp **1.58.0** · Node v24.13.1 · win32 10.0.26200
|
|
6
|
+
- Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz · 12 cores · 31.9 GB RAM
|
|
7
|
+
- 2026-08-06T08:42:53.111Z · mode: full · 24s
|
|
8
|
+
|
|
9
|
+
## Does this harness detect data loss at all?
|
|
10
|
+
|
|
11
|
+
Before trusting any "nothing lost" number, the same harness runs writers that BYPASS the lock — the behaviour that existed before the lock protocol. 10 unlocked writers attempted 22 cards and **17 were lost**.
|
|
12
|
+
|
|
13
|
+
✅ Loss is detectable, so the result below means something.
|
|
14
|
+
|
|
15
|
+
## Concurrent writers
|
|
16
|
+
|
|
17
|
+
| writers | cards written | cards surviving | lost | wall clock |
|
|
18
|
+
|---|---|---|---|---|
|
|
19
|
+
| 10 | 46 | 46 | **0** | 7292 ms |
|
|
20
|
+
|
|
21
|
+
Verdict: **no-loss**. Separate OS processes call the same `brain_note` every agent uses, released simultaneously by a file barrier.
|
|
22
|
+
|
|
23
|
+
## Coordination latency
|
|
24
|
+
|
|
25
|
+
| calls | p50 | p95 | max |
|
|
26
|
+
|---|---|---|---|
|
|
27
|
+
| 300 | 10 ms | 18 ms | 26 ms |
|
|
28
|
+
|
|
29
|
+
One call is what an agent pays to declare its task, receive peers, overlap warnings, messages and task-relevant brain context.
|
|
30
|
+
|
|
31
|
+
## Soak
|
|
32
|
+
|
|
33
|
+
| queries | p50 | p95 | first-decile p50 | last-decile p50 | drift | RSS |
|
|
34
|
+
|---|---|---|---|---|---|---|
|
|
35
|
+
| 1000 | 8 ms | 15 ms | 8 ms | 9 ms | +1 ms | 145.1 → 192.1 MB |
|
|
36
|
+
|
|
37
|
+
Drift compares the last tenth of the run against the first: a brain that slows down as a session grows would show it here.
|
|
38
|
+
|
|
39
|
+
## Crash safety
|
|
40
|
+
|
|
41
|
+
4 writers were SIGKILLed mid-write. Brain still parses: **yes** · cards before 41 → after 43 · pre-existing cards intact: **yes**.
|
|
42
|
+
|
|
43
|
+
## What these numbers are not
|
|
44
|
+
|
|
45
|
+
- They measure the local brain engine, not any hosted service, and not model quality.
|
|
46
|
+
- Latency depends on brain size and disk; a bigger brain on slower storage will be slower.
|
|
47
|
+
- Scenarios this machine could not run are reported as skipped, never as passes.
|
|
48
|
+
|
package/README.md
CHANGED
|
@@ -351,6 +351,25 @@ The app is a separate, proprietary Windows product. The format, this server and
|
|
|
351
351
|
Apache-2.0 and work with no app installed. The app's interface is available in English and Arabic
|
|
352
352
|
(some newer panels are still English-only).
|
|
353
353
|
|
|
354
|
+
## Measure it yourself
|
|
355
|
+
|
|
356
|
+
Claims about a shared brain — "nothing is lost", "it stays fast" — are unfalsifiable until a
|
|
357
|
+
stranger can re-run them, so the benchmark ships in the box:
|
|
358
|
+
|
|
359
|
+
```bash
|
|
360
|
+
npx klypix-mcp bench # ~25s, or --quick for a smaller run
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
It measures concurrent-write safety across real OS processes, coordination latency, a 1,000-query
|
|
364
|
+
soak with drift, and crash safety under SIGKILL — then prints the machine it ran on.
|
|
365
|
+
|
|
366
|
+
**It runs a negative control first.** Writers that bypass the lock go in before the real ones,
|
|
367
|
+
because a "0 lost" number means nothing unless the same harness can *see* a loss. On the reference
|
|
368
|
+
machine those unlocked writers lost 17 of 22 cards; the same contention through the lock protocol
|
|
369
|
+
lost 0 of 46. If the control ever loses nothing, the run reports **inconclusive** instead of a pass.
|
|
370
|
+
|
|
371
|
+
Latest results, with hardware and date: [BENCHMARKS.md](BENCHMARKS.md).
|
|
372
|
+
|
|
354
373
|
## Git and concurrency
|
|
355
374
|
|
|
356
375
|
One file in your repo, committed with your code — versioned, branchable, portable. So two
|
package/bin/klypix-install.mjs
CHANGED
|
@@ -293,7 +293,7 @@ try {
|
|
|
293
293
|
// canvas-view-app.html is the canvas_view MCP App UI — staged raw (an HTML
|
|
294
294
|
// file must never get a JS-comment banner) beside the flat server, which
|
|
295
295
|
// resolves it via its ./canvas-view-app.html candidate path.
|
|
296
|
-
for (const f of ['global-brain-hook.mjs', 'brain-semantic.mjs', 'semantic-memory.mjs', 'brain-note.mjs', 'brain-git-hook.mjs', 'klypix-format.mjs', 'klypix-core.mjs', 'brain-write-lock.mjs', 'agent-rules.mjs', 'brain-doctor.mjs', 'agent-presence.mjs', 'mcp-presence.mjs', 'finding-routing.mjs', 'mcp-supervisor.mjs', 'mcp-auto-update.mjs', 'runtime-inspector.mjs', 'project-graph.mjs', 'codex-brain-hook.mjs', 'codex-hooks.mjs', 'canvas-view-app.html']) {
|
|
296
|
+
for (const f of ['global-brain-hook.mjs', 'brain-semantic.mjs', 'semantic-memory.mjs', 'brain-note.mjs', 'brain-git-hook.mjs', 'klypix-format.mjs', 'klypix-core.mjs', 'brain-write-lock.mjs', 'agent-rules.mjs', 'brain-doctor.mjs', 'agent-presence.mjs', 'mcp-presence.mjs', 'finding-routing.mjs', 'mcp-supervisor.mjs', 'mcp-auto-update.mjs', 'runtime-inspector.mjs', 'project-graph.mjs', 'bench.mjs', 'codex-brain-hook.mjs', 'codex-hooks.mjs', 'canvas-view-app.html']) {
|
|
297
297
|
const s = path.join(SRC, f); if (exists(s)) staged.push({ dst: f, content: fs.readFileSync(s, 'utf8') });
|
|
298
298
|
}
|
|
299
299
|
for (const [src, dst] of [
|
package/bin/klypix-mcp.mjs
CHANGED
|
@@ -19,7 +19,7 @@ const PKG_VERSION = (() => {
|
|
|
19
19
|
}
|
|
20
20
|
})();
|
|
21
21
|
|
|
22
|
-
const DIRECT = new Set(['install', 'link', 'doctor', 'runtime', 'conformance', 'garden-code', 'init', 'git-driver', 'diff', 'pr-brief', 'uninstall']);
|
|
22
|
+
const DIRECT = new Set(['install', 'link', 'doctor', 'runtime', 'conformance', 'garden-code', 'init', 'git-driver', 'diff', 'pr-brief', 'uninstall', 'bench']);
|
|
23
23
|
|
|
24
24
|
const USAGE = [
|
|
25
25
|
`klypix-mcp ${PKG_VERSION} — shared project brain + MCP coordination server.`,
|
|
@@ -30,6 +30,7 @@ const USAGE = [
|
|
|
30
30
|
' doctor [--npm] [--all] [--json] read-only self-check; exits 1 on drift',
|
|
31
31
|
' runtime [--json] [--watch seconds] passive MCP process/RAM attribution; never terminates a process',
|
|
32
32
|
' conformance [--json] launch two real MCP clients against this build',
|
|
33
|
+
' bench [--quick] [--json] [--out F] reproducible benchmark: concurrent-write safety, latency, soak, crash',
|
|
33
34
|
' init seed a starter ./brain.klypix + print an MCP config',
|
|
34
35
|
' garden-code [brain] print the human approval code for brain_garden',
|
|
35
36
|
' uninstall [--check|--yes|unlink] remove this install from the machine (--check inventories first; never deletes a .klypix)',
|
|
@@ -55,7 +56,20 @@ if (verb && !verb.startsWith('-') && !DIRECT.has(verb)) {
|
|
|
55
56
|
process.exit(2);
|
|
56
57
|
}
|
|
57
58
|
|
|
58
|
-
if (verb === '
|
|
59
|
+
if (verb === 'bench') {
|
|
60
|
+
const { runBenchmark, formatBenchmark } = await import('../src/bench.mjs');
|
|
61
|
+
const quick = process.argv.includes('--quick');
|
|
62
|
+
const report = await runBenchmark({ quick });
|
|
63
|
+
const outIdx = process.argv.indexOf('--out');
|
|
64
|
+
const markdown = formatBenchmark(report);
|
|
65
|
+
if (outIdx >= 0 && process.argv[outIdx + 1]) {
|
|
66
|
+
const { writeFileSync } = await import('node:fs');
|
|
67
|
+
writeFileSync(process.argv[outIdx + 1], `${markdown}\n`);
|
|
68
|
+
}
|
|
69
|
+
console.log(process.argv.includes('--json') ? JSON.stringify(report, null, 2) : markdown);
|
|
70
|
+
const c = report.scenarios?.concurrentWriters;
|
|
71
|
+
process.exit(c?.verdict === 'no-loss' ? 0 : 1);
|
|
72
|
+
} else if (verb === 'runtime') {
|
|
59
73
|
process.argv.splice(2, 1);
|
|
60
74
|
await import('./klypix-runtime.mjs');
|
|
61
75
|
} else if (DIRECT.has(verb)) {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "klypix-mcp",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.60.0",
|
|
4
4
|
"description": "Shared project brain and MCP coordination server for multi-agent coding.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "Apache-2.0",
|
|
@@ -66,6 +66,7 @@
|
|
|
66
66
|
"examples",
|
|
67
67
|
"index.mjs",
|
|
68
68
|
"FORMAT.md",
|
|
69
|
+
"BENCHMARKS.md",
|
|
69
70
|
"README.md",
|
|
70
71
|
"A2A.md",
|
|
71
72
|
"LICENSE",
|
|
@@ -76,6 +77,8 @@
|
|
|
76
77
|
},
|
|
77
78
|
"scripts": {
|
|
78
79
|
"test:project-graph": "node test/project-graph.mjs",
|
|
80
|
+
"bench": "node bin/klypix-mcp.mjs bench",
|
|
81
|
+
"test:bench": "node test/bench.mjs",
|
|
79
82
|
"test": "node test/project-graph.mjs && node test/project-map-cli.mjs && node test/mcp-auto-update.mjs && node test/mcp-supervisor.mjs && node test/runtime-inspector.mjs && node test/codex-hooks.mjs && node test/agent-presence.mjs && node test/finding-routing.mjs && node test/finding-routing-hook.mjs && node test/presence-relay.mjs && node test/context-gateway.mjs && node test/conformance.mjs && node test/brain-doctor.mjs && node test/version-currency.mjs && node test/ship-capture.mjs && node test/lane-message.mjs && node test/brain-quality.mjs && node test/brain-connect-orphans.mjs && node test/brief-and-recall.mjs && node test/layout-cluster.mjs && node test/brain-ask.mjs && node test/field-report-2026-07-04.mjs && node test/autoprop.mjs && node test/overlay-recency-2026-07-12.mjs && node test/brain-challenge.mjs && node test/brain-lens.mjs && node test/brain-kind.mjs && node test/rule-drafts.mjs && node test/claim-engine.mjs && node test/skill-staleness.mjs && node test/canvas-view.mjs && node test/status-completeness.mjs && node test/semantic-gate.mjs && node test/memory-runtime.mjs && node test/semantic-cache.mjs && node test/decay-status.mjs && node test/decay-hook.mjs && node test/evidence-anchors.mjs && node test/presence-visibility.mjs && node test/merge-brains.mjs && node test/concurrent-writes.mjs && node test/lock-interop.mjs && node test/a2a-smoke.mjs && node test/cli-args.mjs && node test/format-guard.mjs && node test/git-tools.mjs && node test/uninstall.mjs",
|
|
80
83
|
"test:memory": "node test/memory-runtime.mjs",
|
|
81
84
|
"test:memory:soak": "node --expose-gc test/memory-soak.mjs",
|
package/src/bench.mjs
ADDED
|
@@ -0,0 +1,323 @@
|
|
|
1
|
+
// klypix-mcp bench — the public, reproducible benchmark.
|
|
2
|
+
//
|
|
3
|
+
// WHY THIS EXISTS. Every claim about a shared agent brain ("nothing is lost",
|
|
4
|
+
// "it stays fast") is unfalsifiable until a stranger can re-run it. This
|
|
5
|
+
// harness measures the properties KLYPIX actually sells, on real processes
|
|
6
|
+
// against a real brain file, and prints the command that reproduces it.
|
|
7
|
+
//
|
|
8
|
+
// HOW IT STAYS HONEST — the rules that make the numbers worth reading:
|
|
9
|
+
//
|
|
10
|
+
// 1. A NEGATIVE CONTROL RUNS FIRST. A no-loss result only means something if
|
|
11
|
+
// the same harness can DETECT loss. Control writers deliberately bypass the
|
|
12
|
+
// advisory lock; if they do not lose cards, contention was too weak and the
|
|
13
|
+
// whole no-loss result is reported INCONCLUSIVE rather than as a pass.
|
|
14
|
+
// 2. REAL PROCESSES, REAL OPS. Writers are separate node processes calling the
|
|
15
|
+
// same opBrainNote every agent uses, released together by a file barrier —
|
|
16
|
+
// in-process promises cannot prove cross-process safety.
|
|
17
|
+
// 3. NOTHING IS ROUNDED IN OUR FAVOUR. Latency is reported p50/p95/max, soak
|
|
18
|
+
// drift compares the last decile against the first, and any scenario that
|
|
19
|
+
// cannot be measured on this machine is reported as "skipped", never as a
|
|
20
|
+
// pass.
|
|
21
|
+
// 4. THE ENVIRONMENT IS PART OF THE RESULT. Version, platform, CPU, RAM and
|
|
22
|
+
// date are stamped into every report, because a number without a machine is
|
|
23
|
+
// a marketing asset, not a measurement.
|
|
24
|
+
import fs from 'node:fs';
|
|
25
|
+
import os from 'node:os';
|
|
26
|
+
import path from 'node:path';
|
|
27
|
+
import { spawn } from 'node:child_process';
|
|
28
|
+
import { buildKlypixMap, parseKlypix } from './klypix-format.mjs';
|
|
29
|
+
|
|
30
|
+
const nowMs = () => Number(process.hrtime.bigint() / 1_000_000n);
|
|
31
|
+
const delay = (ms) => new Promise((r) => setTimeout(r, ms));
|
|
32
|
+
const flat = (card) => String(card?.text || '').replace(/\s+/g, ' ').trim();
|
|
33
|
+
|
|
34
|
+
function percentile(sorted, p) {
|
|
35
|
+
if (!sorted.length) return null;
|
|
36
|
+
const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1));
|
|
37
|
+
return Math.round(sorted[index] * 10) / 10;
|
|
38
|
+
}
|
|
39
|
+
function stats(samples) {
|
|
40
|
+
const sorted = [...samples].sort((a, b) => a - b);
|
|
41
|
+
return {
|
|
42
|
+
n: sorted.length,
|
|
43
|
+
p50: percentile(sorted, 50),
|
|
44
|
+
p95: percentile(sorted, 95),
|
|
45
|
+
max: sorted.length ? Math.round(sorted[sorted.length - 1] * 10) / 10 : null,
|
|
46
|
+
};
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
// One writer process. `locked` false is the NEGATIVE CONTROL: it performs the
|
|
50
|
+
// same read-modify-write WITHOUT the advisory lock, which is what every writer
|
|
51
|
+
// did before the lock protocol existed.
|
|
52
|
+
const WRITER_SOURCE = String.raw`
|
|
53
|
+
import fs from 'node:fs';
|
|
54
|
+
const [coreUrl, formatUrl, lockUrl, brain, ready, barrier, tag, countRaw, lockedRaw] = process.argv.slice(2);
|
|
55
|
+
const count = Number(countRaw);
|
|
56
|
+
const locked = lockedRaw === '1';
|
|
57
|
+
const core = await import(coreUrl);
|
|
58
|
+
const fmt = await import(formatUrl);
|
|
59
|
+
const lock = await import(lockUrl);
|
|
60
|
+
fs.writeFileSync(ready, 'ready');
|
|
61
|
+
while (!fs.existsSync(barrier)) await new Promise(r => setTimeout(r, 5));
|
|
62
|
+
let wrote = 0;
|
|
63
|
+
for (let i = 0; i < count; i++) {
|
|
64
|
+
const text = 'BENCH ' + tag + ' card ' + i + ' unique-' + tag + '-' + i;
|
|
65
|
+
try {
|
|
66
|
+
if (locked) {
|
|
67
|
+
const res = await core.opBrainNote({ vault: '.', canvas: brain, text });
|
|
68
|
+
if (!res?.isError) wrote++;
|
|
69
|
+
} else {
|
|
70
|
+
// Unlocked read-modify-write — the pre-lock behaviour, kept ONLY as the
|
|
71
|
+
// control that proves this harness can see a lost update.
|
|
72
|
+
const captured = await fmt.captureIntoBrain(fs.readFileSync(brain), {
|
|
73
|
+
cards: [{ text, color: '#e8e8ed', createdVia: 'bench-control' }],
|
|
74
|
+
});
|
|
75
|
+
await fmt.atomicWrite(brain, captured.buffer);
|
|
76
|
+
wrote++;
|
|
77
|
+
}
|
|
78
|
+
} catch { /* counted as not-written */ }
|
|
79
|
+
}
|
|
80
|
+
process.stdout.write(JSON.stringify({ tag, wrote }));
|
|
81
|
+
`;
|
|
82
|
+
|
|
83
|
+
async function runWriters({ brain, writers, cardsEach, locked, root }) {
|
|
84
|
+
const writerFile = path.join(root, 'bench-writer.mjs');
|
|
85
|
+
fs.writeFileSync(writerFile, WRITER_SOURCE);
|
|
86
|
+
const barrier = path.join(root, `barrier-${locked ? 'locked' : 'control'}-${Date.now()}`);
|
|
87
|
+
const urls = {
|
|
88
|
+
core: new URL('./klypix-core.mjs', import.meta.url).href,
|
|
89
|
+
format: new URL('./klypix-format.mjs', import.meta.url).href,
|
|
90
|
+
lock: new URL('./brain-write-lock.mjs', import.meta.url).href,
|
|
91
|
+
};
|
|
92
|
+
const children = [];
|
|
93
|
+
const readies = [];
|
|
94
|
+
for (let w = 0; w < writers; w++) {
|
|
95
|
+
const tag = `${locked ? 'L' : 'C'}${w}`;
|
|
96
|
+
const ready = path.join(root, `ready-${tag}-${Date.now()}`);
|
|
97
|
+
readies.push(ready);
|
|
98
|
+
const child = spawn(process.execPath, [
|
|
99
|
+
writerFile, urls.core, urls.format, urls.lock, brain, ready, barrier, tag,
|
|
100
|
+
String(cardsEach), locked ? '1' : '0',
|
|
101
|
+
], { stdio: ['ignore', 'pipe', 'ignore'], cwd: root });
|
|
102
|
+
let out = '';
|
|
103
|
+
child.stdout.on('data', (chunk) => { out += chunk; });
|
|
104
|
+
children.push(new Promise((resolve) => child.on('exit', () => {
|
|
105
|
+
try { resolve(JSON.parse(out)); } catch { resolve({ tag, wrote: 0 }); }
|
|
106
|
+
})));
|
|
107
|
+
}
|
|
108
|
+
// Release every writer at the same instant — contention is the point.
|
|
109
|
+
const until = Date.now() + 30_000;
|
|
110
|
+
while (Date.now() < until && !readies.every((f) => fs.existsSync(f))) await delay(10);
|
|
111
|
+
const startedAt = nowMs();
|
|
112
|
+
fs.writeFileSync(barrier, 'go');
|
|
113
|
+
const results = await Promise.all(children);
|
|
114
|
+
const durationMs = Math.round(nowMs() - startedAt);
|
|
115
|
+
const attempted = writers * cardsEach;
|
|
116
|
+
const reportedWritten = results.reduce((sum, r) => sum + Number(r?.wrote || 0), 0);
|
|
117
|
+
const { struct } = await parseKlypix(fs.readFileSync(brain));
|
|
118
|
+
const present = new Set(struct.cards.map(flat).filter((t) => t.includes('unique-')));
|
|
119
|
+
let survived = 0;
|
|
120
|
+
for (const r of results) {
|
|
121
|
+
for (let i = 0; i < cardsEach; i++) {
|
|
122
|
+
const needle = `unique-${r.tag}-${i}`;
|
|
123
|
+
for (const text of present) { if (text.includes(needle)) { survived++; break; } }
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
return { writers, cardsEach, attempted, reportedWritten, survived, lost: reportedWritten - survived, durationMs };
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
async function seedBrain(file, cards = 200) {
|
|
130
|
+
const areas = [{
|
|
131
|
+
title: 'Bench',
|
|
132
|
+
cards: Array.from({ length: cards }, (_, i) => ({
|
|
133
|
+
text: `Seed decision ${i}: the retrieval path must stay correct under load, area ${i % 7}, topic ${['auth', 'canvas', 'sync', 'brain', 'release', 'drive', 'index'][i % 7]}`,
|
|
134
|
+
})),
|
|
135
|
+
}];
|
|
136
|
+
fs.writeFileSync(file, await buildKlypixMap({ title: 'bench-brain', areas }));
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
export async function runBenchmark({ quick = false, log = console.error } = {}) {
|
|
140
|
+
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'klypix-bench-'));
|
|
141
|
+
const started = new Date();
|
|
142
|
+
const scale = quick ? { writers: 4, cardsEach: 3, queries: 100, soak: 200 } : { writers: 10, cardsEach: 5, queries: 300, soak: 1000 };
|
|
143
|
+
const results = { scenarios: {} };
|
|
144
|
+
try {
|
|
145
|
+
// ── 1. NEGATIVE CONTROL — can this harness see a lost update at all? ─────
|
|
146
|
+
log(`[bench] negative control: ${scale.writers} unlocked writers …`);
|
|
147
|
+
const controlBrain = path.join(root, 'control.klypix');
|
|
148
|
+
await seedBrain(controlBrain, 40);
|
|
149
|
+
const control = await runWriters({ brain: controlBrain, writers: scale.writers, cardsEach: scale.cardsEach, locked: false, root });
|
|
150
|
+
results.scenarios.negativeControl = {
|
|
151
|
+
...control,
|
|
152
|
+
detectsLoss: control.lost > 0,
|
|
153
|
+
note: control.lost > 0
|
|
154
|
+
? 'Unlocked writers lost cards, so this harness demonstrably detects lost updates.'
|
|
155
|
+
: 'Unlocked writers lost nothing — contention was too weak on this machine, so the no-loss result below is INCONCLUSIVE, not a pass.',
|
|
156
|
+
};
|
|
157
|
+
|
|
158
|
+
// ── 2. CONCURRENT WRITERS through the real lock protocol ────────────────
|
|
159
|
+
log(`[bench] concurrent writers: ${scale.writers} locked writers …`);
|
|
160
|
+
const brain = path.join(root, 'brain.klypix');
|
|
161
|
+
await seedBrain(brain, 40);
|
|
162
|
+
const concurrent = await runWriters({ brain, writers: scale.writers, cardsEach: scale.cardsEach, locked: true, root });
|
|
163
|
+
results.scenarios.concurrentWriters = {
|
|
164
|
+
...concurrent,
|
|
165
|
+
verdict: !results.scenarios.negativeControl.detectsLoss
|
|
166
|
+
? 'inconclusive'
|
|
167
|
+
: (concurrent.lost === 0 ? 'no-loss' : 'loss-detected'),
|
|
168
|
+
};
|
|
169
|
+
|
|
170
|
+
// ── 3. COORDINATION LATENCY — the per-call cost every agent pays ────────
|
|
171
|
+
log(`[bench] coordination latency: ${scale.queries} task-context calls …`);
|
|
172
|
+
const { opBrainTaskContext } = await import('./klypix-core.mjs');
|
|
173
|
+
const coordination = [];
|
|
174
|
+
for (let i = 0; i < scale.queries; i++) {
|
|
175
|
+
const t = nowMs();
|
|
176
|
+
await opBrainTaskContext({ vault: root, canvas: brain, intent: `bench intent ${i % 11} sync auth canvas`, files: [`src/file-${i % 5}.ts`], k: 5 });
|
|
177
|
+
coordination.push(nowMs() - t);
|
|
178
|
+
}
|
|
179
|
+
results.scenarios.coordinationLatencyMs = stats(coordination);
|
|
180
|
+
|
|
181
|
+
// ── 4. SOAK — does retrieval drift or leak over a long session? ─────────
|
|
182
|
+
log(`[bench] soak: ${scale.soak} queries …`);
|
|
183
|
+
const rssStart = process.memoryUsage().rss;
|
|
184
|
+
const soak = [];
|
|
185
|
+
for (let i = 0; i < scale.soak; i++) {
|
|
186
|
+
const t = nowMs();
|
|
187
|
+
await opBrainTaskContext({ vault: root, canvas: brain, intent: `soak query ${i} topic ${['auth', 'canvas', 'sync', 'brain'][i % 4]}`, k: 5 });
|
|
188
|
+
soak.push(nowMs() - t);
|
|
189
|
+
}
|
|
190
|
+
const decile = Math.max(1, Math.floor(soak.length / 10));
|
|
191
|
+
const firstDecile = stats(soak.slice(0, decile));
|
|
192
|
+
const lastDecile = stats(soak.slice(-decile));
|
|
193
|
+
results.scenarios.soak = {
|
|
194
|
+
queries: soak.length,
|
|
195
|
+
overall: stats(soak),
|
|
196
|
+
firstDecileP50: firstDecile.p50,
|
|
197
|
+
lastDecileP50: lastDecile.p50,
|
|
198
|
+
driftP50Ms: Math.round(((lastDecile.p50 ?? 0) - (firstDecile.p50 ?? 0)) * 10) / 10,
|
|
199
|
+
rssStartMb: Math.round((rssStart / 1048576) * 10) / 10,
|
|
200
|
+
rssEndMb: Math.round((process.memoryUsage().rss / 1048576) * 10) / 10,
|
|
201
|
+
};
|
|
202
|
+
|
|
203
|
+
// ── 5. CRASH SAFETY — kill writers mid-write, is the brain still whole? ─
|
|
204
|
+
log('[bench] crash safety: SIGKILL during concurrent writes …');
|
|
205
|
+
const crashBrain = path.join(root, 'crash.klypix');
|
|
206
|
+
await seedBrain(crashBrain, 40);
|
|
207
|
+
const { struct: beforeStruct } = await parseKlypix(fs.readFileSync(crashBrain));
|
|
208
|
+
const writerFile = path.join(root, 'bench-writer.mjs');
|
|
209
|
+
const barrier = path.join(root, `barrier-crash-${Date.now()}`);
|
|
210
|
+
const kids = [];
|
|
211
|
+
for (let w = 0; w < 4; w++) {
|
|
212
|
+
kids.push(spawn(process.execPath, [
|
|
213
|
+
writerFile,
|
|
214
|
+
new URL('./klypix-core.mjs', import.meta.url).href,
|
|
215
|
+
new URL('./klypix-format.mjs', import.meta.url).href,
|
|
216
|
+
new URL('./brain-write-lock.mjs', import.meta.url).href,
|
|
217
|
+
crashBrain, path.join(root, `ready-crash-${w}`), barrier, `K${w}`, '6', '1',
|
|
218
|
+
], { stdio: 'ignore', cwd: root }));
|
|
219
|
+
}
|
|
220
|
+
await delay(400);
|
|
221
|
+
fs.writeFileSync(barrier, 'go');
|
|
222
|
+
await delay(250);
|
|
223
|
+
for (const kid of kids) { try { kid.kill('SIGKILL'); } catch { /* already gone */ } }
|
|
224
|
+
await delay(600);
|
|
225
|
+
let parsedAfterCrash = false;
|
|
226
|
+
let cardsAfter = 0;
|
|
227
|
+
try {
|
|
228
|
+
const { struct } = await parseKlypix(fs.readFileSync(crashBrain));
|
|
229
|
+
parsedAfterCrash = true;
|
|
230
|
+
cardsAfter = struct.cards.length;
|
|
231
|
+
} catch { parsedAfterCrash = false; }
|
|
232
|
+
results.scenarios.crashSafety = {
|
|
233
|
+
killedWriters: kids.length,
|
|
234
|
+
brainStillParses: parsedAfterCrash,
|
|
235
|
+
cardsBefore: beforeStruct.cards.length,
|
|
236
|
+
cardsAfter,
|
|
237
|
+
seedCardsIntact: cardsAfter >= beforeStruct.cards.length,
|
|
238
|
+
note: 'Writers are SIGKILLed mid-write. The brain must still parse and must never lose cards that existed before the crash.',
|
|
239
|
+
};
|
|
240
|
+
} finally {
|
|
241
|
+
try { fs.rmSync(root, { recursive: true, force: true }); } catch { /* temp */ }
|
|
242
|
+
}
|
|
243
|
+
|
|
244
|
+
const cpus = os.cpus();
|
|
245
|
+
return {
|
|
246
|
+
schemaVersion: 1,
|
|
247
|
+
tool: 'klypix-mcp bench',
|
|
248
|
+
startedAt: started.toISOString(),
|
|
249
|
+
durationSec: Math.round((Date.now() - started.getTime()) / 1000),
|
|
250
|
+
mode: quick ? 'quick' : 'full',
|
|
251
|
+
environment: {
|
|
252
|
+
klypixMcp: (() => { try { return JSON.parse(fs.readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version; } catch { return 'unknown'; } })(),
|
|
253
|
+
node: process.version,
|
|
254
|
+
platform: `${os.platform()} ${os.release()}`,
|
|
255
|
+
cpu: cpus[0]?.model?.trim() || 'unknown',
|
|
256
|
+
cores: cpus.length,
|
|
257
|
+
totalRamGb: Math.round((os.totalmem() / 1073741824) * 10) / 10,
|
|
258
|
+
},
|
|
259
|
+
...results,
|
|
260
|
+
};
|
|
261
|
+
}
|
|
262
|
+
|
|
263
|
+
export function formatBenchmark(report) {
|
|
264
|
+
const env = report.environment || {};
|
|
265
|
+
const c = report.scenarios?.concurrentWriters || {};
|
|
266
|
+
const ctrl = report.scenarios?.negativeControl || {};
|
|
267
|
+
const lat = report.scenarios?.coordinationLatencyMs || {};
|
|
268
|
+
const soak = report.scenarios?.soak || {};
|
|
269
|
+
const crash = report.scenarios?.crashSafety || {};
|
|
270
|
+
const lines = [
|
|
271
|
+
'# KLYPIX brain — benchmark',
|
|
272
|
+
'',
|
|
273
|
+
`Run \`npx klypix-mcp bench\` to reproduce this table on your own machine.`,
|
|
274
|
+
'',
|
|
275
|
+
`- klypix-mcp **${env.klypixMcp}** · Node ${env.node} · ${env.platform}`,
|
|
276
|
+
`- ${env.cpu} · ${env.cores} cores · ${env.totalRamGb} GB RAM`,
|
|
277
|
+
`- ${report.startedAt} · mode: ${report.mode} · ${report.durationSec}s`,
|
|
278
|
+
'',
|
|
279
|
+
'## Does this harness detect data loss at all?',
|
|
280
|
+
'',
|
|
281
|
+
`Before trusting any "nothing lost" number, the same harness runs writers that BYPASS the lock — the behaviour that existed before the lock protocol. ${ctrl.writers} unlocked writers attempted ${ctrl.reportedWritten} cards and **${ctrl.lost} were lost**.`,
|
|
282
|
+
'',
|
|
283
|
+
ctrl.detectsLoss
|
|
284
|
+
? '✅ Loss is detectable, so the result below means something.'
|
|
285
|
+
: '⚠️ No loss appeared in the control, so contention was too weak on this machine and the result below is **inconclusive**.',
|
|
286
|
+
'',
|
|
287
|
+
'## Concurrent writers',
|
|
288
|
+
'',
|
|
289
|
+
'| writers | cards written | cards surviving | lost | wall clock |',
|
|
290
|
+
'|---|---|---|---|---|',
|
|
291
|
+
`| ${c.writers} | ${c.reportedWritten} | ${c.survived} | **${c.lost}** | ${c.durationMs} ms |`,
|
|
292
|
+
'',
|
|
293
|
+
`Verdict: **${c.verdict}**. Separate OS processes call the same \`brain_note\` every agent uses, released simultaneously by a file barrier.`,
|
|
294
|
+
'',
|
|
295
|
+
'## Coordination latency',
|
|
296
|
+
'',
|
|
297
|
+
`| calls | p50 | p95 | max |`,
|
|
298
|
+
'|---|---|---|---|',
|
|
299
|
+
`| ${lat.n} | ${lat.p50} ms | ${lat.p95} ms | ${lat.max} ms |`,
|
|
300
|
+
'',
|
|
301
|
+
'One call is what an agent pays to declare its task, receive peers, overlap warnings, messages and task-relevant brain context.',
|
|
302
|
+
'',
|
|
303
|
+
'## Soak',
|
|
304
|
+
'',
|
|
305
|
+
`| queries | p50 | p95 | first-decile p50 | last-decile p50 | drift | RSS |`,
|
|
306
|
+
'|---|---|---|---|---|---|---|',
|
|
307
|
+
`| ${soak.queries} | ${soak.overall?.p50} ms | ${soak.overall?.p95} ms | ${soak.firstDecileP50} ms | ${soak.lastDecileP50} ms | ${soak.driftP50Ms >= 0 ? '+' : ''}${soak.driftP50Ms} ms | ${soak.rssStartMb} → ${soak.rssEndMb} MB |`,
|
|
308
|
+
'',
|
|
309
|
+
'Drift compares the last tenth of the run against the first: a brain that slows down as a session grows would show it here.',
|
|
310
|
+
'',
|
|
311
|
+
'## Crash safety',
|
|
312
|
+
'',
|
|
313
|
+
`${crash.killedWriters} writers were SIGKILLed mid-write. Brain still parses: **${crash.brainStillParses ? 'yes' : 'NO'}** · cards before ${crash.cardsBefore} → after ${crash.cardsAfter} · pre-existing cards intact: **${crash.seedCardsIntact ? 'yes' : 'NO'}**.`,
|
|
314
|
+
'',
|
|
315
|
+
'## What these numbers are not',
|
|
316
|
+
'',
|
|
317
|
+
'- They measure the local brain engine, not any hosted service, and not model quality.',
|
|
318
|
+
'- Latency depends on brain size and disk; a bigger brain on slower storage will be slower.',
|
|
319
|
+
'- Scenarios this machine could not run are reported as skipped, never as passes.',
|
|
320
|
+
'',
|
|
321
|
+
];
|
|
322
|
+
return lines.join('\n');
|
|
323
|
+
}
|