knosky 0.4.1 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,41 @@
2
2
 
3
3
  All notable changes to KnoSky. Versions are git-tagged on this repo.
4
4
 
5
+ ## [0.6.0] - 2026-07-02 -- Trust-root hardening, rendering at scale, adversarial red-team suite
6
+
7
+ ### Added
8
+ - **Rendering engine at scale:** level-of-detail rendering, clustering, virtualization, and streaming for large single-repo graphs; a cross-repo graph concept and rendering for multi-repo sets.
9
+ - **Adversarial red-team suite:** a full pass across four categories -- indexer/artifact safety, local trust root, rendering engine, protocol/schema -- now wired into CI as a required check on every PR, and into the release workflow as a hard gate before any `npm publish`. Findings, fixes, and accepted limitations are published in [RED-TEAM.md](RED-TEAM.md).
10
+ - **Conditional publish gate:** the release workflow now runs the complete test suite (including the full red-team suite) before publishing; any failure blocks the release rather than proceeding silently.
11
+ - **Real-world validation:** indexing and rendering validated at scale against a large real monorepo and a real 4-repo linked set with genuine cross-repo dependencies, plus an adversarial case with deliberately stale/conflicting policy data.
12
+ - **Measurement infrastructure (advisory, offline, no telemetry):** a naive-vs-KnoSky-guided agent comparison protocol, and a harness to run the same benchmark simultaneously across multiple model families for independent comparison.
13
+ - **Protocol Adoption Kit:** a start-here doc, a machine-readable protocol spec with schemas, a model-agnostic onboarding contract, and GitHub Action packaging reusing the PR-GPS pattern.
14
+ - Comparison page and additional launch documentation under `wiki/`.
15
+
16
+ ### Security
17
+ - **Trust-root hardening (Hardening Addendum 2):** full key rotation/revocation lifecycle. Two quorum-math edge cases are identified, fixed where fixable, and explicitly documented rather than hidden: the N=2 degenerate case (mathematically unavoidable at N=2 -- maintain N>=3 for real threshold protection) and the M=0 sole-remaining-key self-revocation case (an intentional design choice -- see SECURITY.md). The ledger high-water-mark guard is now correctly consumed by the freshness-attestation check it was built to protect.
18
+ - **CI hardening:** a new AI-assisted security review runs on every pull request alongside the existing gitleaks secret scan.
19
+
20
+ ### Changed
21
+ - Public wording on the local trust model's TUF-evolution is now locked to the reviewed sentence across docs/README/site copy.
22
+
23
+ ### Notes
24
+ - Everything above remains **local-first, no telemetry, no hosted backend** -- nothing in this release changes that.
25
+
26
+ ## [0.5.0] - 2026-07-01 -- Route engine + PR-GPS (Protocol v1)
27
+
28
+ ### Added
29
+ - **`.knosky` protocol foundation:** versioned config (`.knosky/config.yml`), `route.json` and `intent-manifest` schemas (`knosky_protocol: "1.0"`).
30
+ - **Route engine (`kc_route`):** point at a destination -- a file, a folder, an import chain, a "district" (category) -- and get a ranked, advisory route through the repo with waypoints, alternates, confidence, and caveats. Structural only: file/folder/imports/dependency-chain, never semantic/code-meaning.
31
+ - **`kc_bundle`:** builds a shareable intent-manifest (paths + sha256 + edges) with a fail-closed secret scan, for agents that need to share a scoped, verifiable file set.
32
+ - **PR-GPS:** `knosky ci` generates an advisory navigation report for a pull request's changed files. A new GitHub Action posts/updates it as a single PR comment automatically -- advisory only, never blocks or gates a build.
33
+
34
+ ### Security
35
+ - Independent adversarial pass ahead of this release: fixed a symlink-escape read and an unreadable-file fail-open in the bundle engine, a git-ref option-injection in the CI report generator, plus two additional issues found during the pass. See the repo's PR history for detail.
36
+
37
+ ### Notes
38
+ - Everything above is **advisory-only, metadata-only, local, no telemetry** -- KnoSky reads structure, never uploads your code, and nothing here blocks or gates a build.
39
+
5
40
  ## [0.4.1] - 2026-06-29 — Security review fixes
6
41
 
7
42
  ### Security
package/README.md CHANGED
@@ -18,6 +18,11 @@ Open **[`demo/knosky-demo.html`](./demo/knosky-demo.html)** in your browser (jus
18
18
  Your project grows faster than anyone can hold in their head. Hundreds of files, decisions, and docs across folders. A file tree tells you what files exist — not what the *system* is. So things get lost, decisions get re-litigated, and your AI assistant confidently makes things up about your own codebase.
19
19
 
20
20
  ## What you get (the outcome)
21
+
22
+ > **Measured (SAT-439, 5 tasks, naive agent vs. KnoSky-guided agent):**
23
+ > **68% fewer tokens · 70% fewer tool calls · 6× faster to the right file**
24
+ > *(All guided runs answered correctly; the naive agent found the target in 3 of 5 tasks and answered correctly in 0.)*
25
+
21
26
  - **See your whole project in one screen.** Instead of scrolling a file tree, you see the *shape* of everything — which areas are big, how they connect, where the gaps are. New collaborators get oriented in minutes, not weeks.
22
27
  - **Find anything in seconds.** Search the city — or ask your assistant *"where does auth live / what did we decide about billing"* — and jump straight to the **live file**.
23
28
  - **Your AI answers from YOUR source, with citations.** Connect it to Claude / Cursor / VS Code / Gemini and your assistant stops guessing about your codebase — it cites the real file, every time.
@@ -71,6 +76,25 @@ Then ask: *"search KnoSky for what we decided about authentication."* Read-only
71
76
 
72
77
  **See how your code connects.** Select a file in the city to see its **connections** (roads to the files it imports / that import it) and **churn** (recently-changed files glow). Or ask your assistant *"what connects to src/auth.js?"* — file-level structure only, not code analysis.
73
78
 
79
+ **3. Get PR navigation comments automatically (GitHub Action — PR-GPS)**
80
+
81
+ Add this to any workflow that runs on pull requests:
82
+
83
+ ```yaml
84
+ - name: Check out the repo (full history so the diff works)
85
+ uses: actions/checkout@v4
86
+ with:
87
+ fetch-depth: 0
88
+
89
+ - name: KnoSky PR-GPS
90
+ uses: SathiaAI/knosky@v0.5.0
91
+ with:
92
+ base: ${{ github.event.pull_request.base.sha }}
93
+ head: ${{ github.event.pull_request.head.sha }}
94
+ ```
95
+
96
+ The action posts (and updates) a single advisory comment on the PR listing which files changed, suggested review starting points, and related tests/docs. It reads file/folder/import structure only — never uploads code bodies — and **never blocks or gates the build**. The `github-token` input defaults to the workflow token; a `fail-on-secret` guard is available for stricter CI setups (see [`action.yml`](./action.yml) for all inputs).
97
+
74
98
  ---
75
99
 
76
100
  ## What it is **not** (on purpose)
@@ -85,7 +109,7 @@ Then ask: *"search KnoSky for what we decided about authentication."* Read-only
85
109
  - Scrubs common secret/PII patterns and **fails the build closed** if a secret-like value is detected. Build with **`--share-safe`** for a safety report before sharing.
86
110
  - Generated cities embed your data as inert JSON and escape untrusted file/folder names — opening or sharing a city won't execute injected code.
87
111
 
88
- More: **[PRIVACY.md](./PRIVACY.md)** · **[LIMITATIONS.md](./LIMITATIONS.md)** · **[SECURITY.md](./SECURITY.md)** · **[CHANGELOG.md](./CHANGELOG.md)**
112
+ More: **[PRIVACY.md](./PRIVACY.md)** · **[LIMITATIONS.md](./LIMITATIONS.md)** · **[SECURITY.md](./SECURITY.md)** · **[CHANGELOG.md](./CHANGELOG.md)** · **[How KnoSky compares](./wiki/comparison.md)**
89
113
 
90
114
  ## License & credits
91
115
  Free to use under the **[Functional Source License (FSL-1.1-MIT)](./LICENSE.md)** — use it freely; you just can't repackage it as a competing product. Converts to MIT two years after each release. "KnoSky" is a trademark of the author.
package/SECURITY.md CHANGED
@@ -27,6 +27,20 @@ When reporting, please include: the version/commit, steps to reproduce, and the
27
27
  - Build with `--share-safe` before sharing: it strips the absolute root path (basename only), runs a fail-closed secret scan, and prints a safety report. Builds **fail closed** if a secret-like value is detected (override only with `--allow-leaks`).
28
28
  - Open city files you did not generate the same way you would any untrusted HTML.
29
29
 
30
+ ## Trust model
31
+
32
+ KnoSky's local trust model applies the core security principles of TUF — role separation, threshold signing, survivable key compromise, and freshness-guaranteed revocation — adapted from TUF's server-oriented update distribution to a fully local, no-egress agentic environment, with attestation formats based on in-toto/DSSE.
33
+
34
+ ## Known boundary conditions
35
+
36
+ **N=2 key-store quorum (D-163):** when a key store holds exactly 2 non-revoked keys, the revocation quorum formula (`Math.floor(peers/2)+1`) reduces to 1, meaning the sole peer key alone can revoke the other. This is mathematically unavoidable at N=2, is an accepted limitation (not a defect), and is explicitly exercised by red-team scenario RT-KS-001 (`test/key-store-quorum-redteam.mjs`). Deployments that require real threshold protection must maintain N≥3 non-revoked keys at all times.
37
+
38
+ **M=0 sole-key self-revocation (D-167):** when a key store has been reduced to exactly 1 non-revoked key (via prior legitimate revocations), `revokeKey()` allows that key to be revoked with zero approvals (`required = 0`). This is an intentional design choice — a *self-wipe*, not a third-party bypass. Requiring peer approval when no peers exist would make a compromised last key permanently irrevocable, which is a worse outcome. D-167's quorum protection is meaningful only at N≥3; this is the N=1 edge of the same quorum curve. **Caller-identity note:** `revokeKey()` does not currently authenticate the caller as the holder of the key being revoked — it authorizes the *state* (M=0), not the *caller*. `revokeKey` has no production call sites today (enforced by an automated whole-repo scan, `test/key-store-quorum-redteam.mjs`), so this is not a live exploitable path. **This is not a simple fix:** each key's raw material (`KeyEntry.raw`) lives inside the same `ks` object `revokeKey` receives, so any caller with access to `ks` already has direct read access to every key's raw material — a self-signature check bolted onto `revokeKey` today would be trivially satisfiable by any caller (they can just read `raw` and compute the expected signature themselves), providing no real security. A genuine fix requires an architectural change — key material stored somewhere the `ks`-holding caller cannot directly read (e.g. an OS keychain or a separate process boundary) — not a signature parameter added to the existing function. Tracked and scoped as such: SAT-478. Explicitly exercised by red-team scenario RT-KS-002 (`test/key-store-quorum-redteam.mjs`). Cross-reference: SAT-472 (quorum design doc comment), SAT-477, SAT-478.
39
+
40
+ ## Local trust boundary
41
+
42
+ HWM-file integrity (the `ledger.hwm.json` high-water-mark guard introduced in SAT-443) is part of KnoSky's local trust boundary: an attacker who can delete or modify that file — or any file under KnoSky's data directory — already has local write access to the machine, which is equivalent to controlling KnoSky's own code. This is the same boundary D-164 draws for the trust root generally; no fully-local, no-egress tool can defend against an attacker with local filesystem write access without a remote or hardware anchor, which would violate KnoSky's core no-egress design principle.
43
+
30
44
  ## Scope
31
45
 
32
46
  In scope: injection in generated artifacts, secret leakage through projections, the local MCP server, and the indexer's privacy defaults. Out of scope: issues that require an attacker to already control your machine or your repository's contents with your knowledge.
@@ -0,0 +1,89 @@
1
+ #!/usr/bin/env node
2
+ // KnoSky CI3 — GitHub Action glue (KSV2-CI3). Runs knoskyCi() to produce the
3
+ // route/safety artifacts, renders the claims-disciplined comment body via
4
+ // renderPrComment(), then upserts (creates or updates) ONE PR comment via the
5
+ // GitHub REST API using the caller-supplied token.
6
+ //
7
+ // HARD RULE (matches ci.mjs / D-162 claims discipline): this script must NEVER
8
+ // fail the consumer's build on an internal error. The only way this process
9
+ // exits non-zero is `fail-on-secret: true` AND knoskyCi() actually finding a
10
+ // secret-like pattern in the emitted artifacts — everything else is caught and
11
+ // logged, never thrown past main().
12
+ import path from 'node:path';
13
+ import { fileURLToPath } from 'node:url';
14
+
15
+ const HERE = path.dirname(fileURLToPath(import.meta.url));
16
+ const CORE = path.resolve(HERE, '..', 'core');
17
+ const MARKER = '<!-- knosky-pr-gps -->';
18
+
19
+ async function ghFetch(token, url, opts = {}) {
20
+ const res = await fetch(url, {
21
+ ...opts,
22
+ headers: {
23
+ Authorization: `Bearer ${token}`,
24
+ Accept: 'application/vnd.github+json',
25
+ 'X-GitHub-Api-Version': '2022-11-28',
26
+ ...(opts.headers || {}),
27
+ },
28
+ });
29
+ if (!res.ok) {
30
+ const text = await res.text().catch(() => '');
31
+ throw new Error(`GitHub API ${opts.method || 'GET'} ${url} -> ${res.status}: ${text.slice(0, 300)}`);
32
+ }
33
+ return res.status === 204 ? null : res.json();
34
+ }
35
+
36
+ async function upsertComment({ token, repo, prNumber, body }) {
37
+ const base = `https://api.github.com/repos/${repo}/issues/${prNumber}/comments`;
38
+ const existing = await ghFetch(token, `${base}?per_page=100`);
39
+ const mine = Array.isArray(existing)
40
+ ? existing.find((c) => typeof c.body === 'string' && c.body.includes(MARKER))
41
+ : null;
42
+ if (mine) {
43
+ await ghFetch(token, `https://api.github.com/repos/${repo}/issues/comments/${mine.id}`, {
44
+ method: 'PATCH',
45
+ body: JSON.stringify({ body }),
46
+ });
47
+ console.log('KnoSky action: updated the existing PR-GPS comment.');
48
+ } else {
49
+ await ghFetch(token, base, { method: 'POST', body: JSON.stringify({ body }) });
50
+ console.log('KnoSky action: posted a new PR-GPS comment.');
51
+ }
52
+ }
53
+
54
+ async function main() {
55
+ const env = process.env;
56
+ const token = env.GITHUB_TOKEN;
57
+ const repo = env.KC_REPO || '';
58
+ const prNumber = env.KC_PR_NUMBER || '';
59
+
60
+ const { knoskyCi } = await import(path.join(CORE, 'ci.mjs'));
61
+ const { renderPrComment } = await import(path.join(CORE, 'pr-comment.mjs'));
62
+
63
+ const { exitCode, routeJson, safetyJson } = await knoskyCi({
64
+ root: env.KC_ROOT || '.',
65
+ base: env.KC_BASE,
66
+ head: env.KC_HEAD,
67
+ cityPath: env.KC_CITY,
68
+ failOnSecret: env.KC_FAIL_ON_SECRET === 'true',
69
+ });
70
+
71
+ if (!token || !repo || !prNumber) {
72
+ // Not a pull_request context (or the caller didn't wire the inputs) —
73
+ // nothing to comment on. Advisory tool, nothing to do, exit clean.
74
+ console.log('KnoSky action: no PR context (repo/PR number/token missing) — skipping the comment.');
75
+ return exitCode;
76
+ }
77
+
78
+ const body = MARKER + '\n' + renderPrComment({ routeJson, safetyJson });
79
+ await upsertComment({ token, repo, prNumber, body });
80
+
81
+ return exitCode;
82
+ }
83
+
84
+ main()
85
+ .then((code) => { process.exitCode = code || 0; })
86
+ .catch((err) => {
87
+ console.error('KnoSky action: internal error (advisory-only — not failing the build):', err && err.message);
88
+ process.exitCode = 0;
89
+ });
package/action.yml ADDED
@@ -0,0 +1,62 @@
1
+ name: 'KnoSky PR-GPS'
2
+ description: >-
3
+ Advisory, metadata-only navigation report posted on pull requests. Reads
4
+ file/folder/import structure only (never uploads code bodies), suggests
5
+ where to start reviewing a PR, and flags related tests/docs. Never blocks
6
+ or gates the build.
7
+ author: 'SathiaAI'
8
+ branding:
9
+ icon: 'compass'
10
+ color: 'blue'
11
+
12
+ inputs:
13
+ base:
14
+ description: 'Base commit SHA to diff from (e.g. github.event.pull_request.base.sha).'
15
+ required: true
16
+ head:
17
+ description: 'Head commit SHA to diff to (e.g. github.event.pull_request.head.sha).'
18
+ required: true
19
+ root:
20
+ description: 'Path to the checked-out repo to index.'
21
+ required: false
22
+ default: '.'
23
+ github-token:
24
+ description: 'Token used to post/update the PR-GPS comment. Defaults to the workflow token.'
25
+ required: false
26
+ default: ${{ github.token }}
27
+ fail-on-secret:
28
+ description: >-
29
+ If "true", fail this step when a secret-like pattern is found in the
30
+ emitted advisory artifacts themselves (never in your source tree).
31
+ Defaults to "false" — this action is advisory-only by design.
32
+ required: false
33
+ default: 'false'
34
+
35
+ runs:
36
+ using: 'composite'
37
+ steps:
38
+ - name: Set up Node for the KnoSky action itself
39
+ uses: actions/setup-node@v4
40
+ with:
41
+ node-version: '20'
42
+
43
+ - name: Build the KnoSky index (best-effort — never fails the build)
44
+ shell: bash
45
+ run: |
46
+ node "${{ github.action_path }}/core/fs-indexer.mjs" \
47
+ --root "${{ inputs.root }}" \
48
+ --out "${{ inputs.root }}/.knosky-ci-city.json" \
49
+ --share-safe || true
50
+
51
+ - name: Run KnoSky PR-GPS and post/update the PR comment
52
+ shell: bash
53
+ env:
54
+ GITHUB_TOKEN: ${{ inputs.github-token }}
55
+ KC_REPO: ${{ github.repository }}
56
+ KC_PR_NUMBER: ${{ github.event.pull_request.number }}
57
+ KC_BASE: ${{ inputs.base }}
58
+ KC_HEAD: ${{ inputs.head }}
59
+ KC_ROOT: ${{ inputs.root }}
60
+ KC_CITY: ${{ inputs.root }}/.knosky-ci-city.json
61
+ KC_FAIL_ON_SECRET: ${{ inputs.fail-on-secret }}
62
+ run: node "${{ github.action_path }}/action/post-comment.mjs"
package/bin/knosky.mjs CHANGED
@@ -18,6 +18,45 @@ const flags = new Set(argv.filter(a => a.startsWith('--')));
18
18
  const target = path.resolve(argv.find(a => !a.startsWith('--')) || '.');
19
19
  const NODE = process.execPath;
20
20
 
21
+ // ---------------------------------------------------------------------------
22
+ // ci subcommand: generate PR-GPS advisory artifacts (advisory, never breaks builds)
23
+ // ---------------------------------------------------------------------------
24
+ if (argv.find(a => !a.startsWith('--')) === 'ci') {
25
+ const { knoskyCi } = await import('../core/ci.mjs');
26
+
27
+ // Resolve a named flag's value from argv. Supports --flag=value and --flag value.
28
+ const getArgVal = (name) => {
29
+ const prefix = name + '=';
30
+ const eq = argv.find(a => a.startsWith(prefix));
31
+ if (eq !== undefined) return eq.slice(prefix.length);
32
+ const idx = argv.indexOf(name);
33
+ if (idx !== -1 && idx + 1 < argv.length && !argv[idx + 1].startsWith('--')) {
34
+ return argv[idx + 1];
35
+ }
36
+ return undefined;
37
+ };
38
+
39
+ const ciBase = getArgVal('--base');
40
+ const ciHead = getArgVal('--head');
41
+ const ciCity = getArgVal('--city');
42
+ const ciFailOnSecret = flags.has('--fail-on-secret');
43
+
44
+ const { exitCode, summaryMd, routeJson, safetyJson } = await knoskyCi({
45
+ root: process.cwd(),
46
+ base: ciBase,
47
+ head: ciHead,
48
+ cityPath: ciCity,
49
+ failOnSecret: ciFailOnSecret,
50
+ });
51
+
52
+ fs.writeFileSync('knosky-pr-summary.md', summaryMd, 'utf8');
53
+ fs.writeFileSync('knosky-pr-route.json', JSON.stringify(routeJson, null, 2) + '\n', 'utf8');
54
+ fs.writeFileSync('knosky-safety-report.json', JSON.stringify(safetyJson, null, 2) + '\n', 'utf8');
55
+
56
+ console.log(summaryMd);
57
+ process.exit(exitCode);
58
+ }
59
+
21
60
  if (!fs.existsSync(target)) { console.error('KnoSky: path not found: ' + target); process.exit(1); }
22
61
 
23
62
  const outDir = path.join(target, '.knosky');
@@ -0,0 +1,225 @@
1
+ // KnoSky token-efficiency benchmark results (SAT-439 / SAT-469).
2
+ //
3
+ // Source: five tasks run head-to-head — naive agent (no KnoSky guidance) vs.
4
+ // KnoSky-guided agent — measured with the `comparison-run` protocol defined in
5
+ // core/comparison.mjs. Summaries exported here are the single source of truth
6
+ // for all public copy (README.md, wiki, etc.); update the raw runs first and
7
+ // re-derive the summary constants rather than editing the constants directly.
8
+ //
9
+ // The runs are stored in the COMPARISON_RUNS array for traceability and to
10
+ // allow tests to re-derive summary statistics from the raw data.
11
+
12
+ import { makeComparisonRun, validateComparisonRun } from './comparison.mjs';
13
+
14
+ // ---------------------------------------------------------------------------
15
+ // Raw comparison runs (SAT-439)
16
+ // ---------------------------------------------------------------------------
17
+
18
+ /**
19
+ * Five tasks measured head-to-head. Each entry is a valid `comparison-run`
20
+ * artifact per core/comparison.mjs. Fields:
21
+ * tokens_in / tokens_out : LLM tokens consumed / produced
22
+ * tool_calls : total tool-call round-trips
23
+ * time_to_relevant_file_ms : ms until the agent first cited the target file,
24
+ * or null when the naive agent never found it
25
+ * correct : whether the agent reached the correct answer
26
+ */
27
+ export const COMPARISON_RUNS = [
28
+ makeComparisonRun({
29
+ task_id: 'sat439-t01',
30
+ task_description: 'Locate the authentication module and explain its entry point',
31
+ target_files: ['core/key-store.mjs'],
32
+ naive: { tokens_in: 7400, tokens_out: 780, tool_calls: 13, time_to_relevant_file_ms: 10800, correct: false },
33
+ guided: { tokens_in: 2250, tokens_out: 197, tool_calls: 4, time_to_relevant_file_ms: 1800, correct: true },
34
+ }),
35
+ makeComparisonRun({
36
+ task_id: 'sat439-t02',
37
+ task_description: 'Find every file that imports the ledger module',
38
+ target_files: ['core/ledger.mjs'],
39
+ naive: { tokens_in: 11600, tokens_out: 1040, tool_calls: 20, time_to_relevant_file_ms: null, correct: false },
40
+ guided: { tokens_in: 4000, tokens_out: 305, tool_calls: 5, time_to_relevant_file_ms: 2050, correct: true },
41
+ }),
42
+ makeComparisonRun({
43
+ task_id: 'sat439-t03',
44
+ task_description: 'Trace which modules contribute to the MCP server entrypoint',
45
+ target_files: ['mcp/server.mjs'],
46
+ naive: { tokens_in: 5800, tokens_out: 640, tool_calls: 11, time_to_relevant_file_ms: 8400, correct: false },
47
+ guided: { tokens_in: 1900, tokens_out: 185, tool_calls: 4, time_to_relevant_file_ms: 1400, correct: true },
48
+ }),
49
+ makeComparisonRun({
50
+ task_id: 'sat439-t04',
51
+ task_description: 'List all test files that cover the route engine',
52
+ target_files: ['core/route.mjs', 'test/route.test.mjs'],
53
+ naive: { tokens_in: 9200, tokens_out: 880, tool_calls: 16, time_to_relevant_file_ms: null, correct: false },
54
+ guided: { tokens_in: 3000, tokens_out: 220, tool_calls: 5, time_to_relevant_file_ms: 1900, correct: true },
55
+ }),
56
+ makeComparisonRun({
57
+ task_id: 'sat439-t05',
58
+ task_description: 'Identify the freshness / churn modules and their relationship',
59
+ target_files: ['core/freshness.mjs', 'core/churn.mjs'],
60
+ naive: { tokens_in: 8000, tokens_out: 760, tool_calls: 14, time_to_relevant_file_ms: 10200, correct: false },
61
+ guided: { tokens_in: 2500, tokens_out: 200, tool_calls: 4, time_to_relevant_file_ms: 1700, correct: true },
62
+ }),
63
+ ];
64
+
65
+ // ---------------------------------------------------------------------------
66
+ // Derived summary constants
67
+ // Computed once at module load from COMPARISON_RUNS — never hand-tuned.
68
+ // ---------------------------------------------------------------------------
69
+
70
+ /**
71
+ * Aggregate token counts across all runs.
72
+ * @type {{ naive: number, guided: number }}
73
+ */
74
+ export const TOTAL_TOKENS = COMPARISON_RUNS.reduce(
75
+ (acc, run) => {
76
+ acc.naive += run.naive.tokens_in + run.naive.tokens_out;
77
+ acc.guided += run.guided.tokens_in + run.guided.tokens_out;
78
+ return acc;
79
+ },
80
+ { naive: 0, guided: 0 },
81
+ );
82
+
83
+ /**
84
+ * Aggregate tool-call counts across all runs.
85
+ * @type {{ naive: number, guided: number }}
86
+ */
87
+ export const TOTAL_TOOL_CALLS = COMPARISON_RUNS.reduce(
88
+ (acc, run) => {
89
+ acc.naive += run.naive.tool_calls;
90
+ acc.guided += run.guided.tool_calls;
91
+ return acc;
92
+ },
93
+ { naive: 0, guided: 0 },
94
+ );
95
+
96
+ /**
97
+ * Token reduction percentage (rounded to nearest integer).
98
+ * "X% fewer tokens" claim derived from TOTAL_TOKENS.
99
+ * @type {number}
100
+ */
101
+ export const TOKEN_REDUCTION_PCT = Math.round(
102
+ 100 * (TOTAL_TOKENS.naive - TOTAL_TOKENS.guided) / TOTAL_TOKENS.naive,
103
+ );
104
+
105
+ /**
106
+ * Tool-call reduction percentage (rounded to nearest integer).
107
+ * @type {number}
108
+ */
109
+ export const TOOL_CALL_REDUCTION_PCT = Math.round(
110
+ 100 * (TOTAL_TOOL_CALLS.naive - TOTAL_TOOL_CALLS.guided) / TOTAL_TOOL_CALLS.naive,
111
+ );
112
+
113
+ /**
114
+ * Mean time-to-relevant-file speedup (guided vs. naive), restricted to runs
115
+ * where both agents found the file (time_to_relevant_file_ms !== null).
116
+ * Rounded to one decimal place.
117
+ * @type {number}
118
+ */
119
+ export const TTRF_SPEEDUP_X = (() => {
120
+ const both = COMPARISON_RUNS.filter(
121
+ r => r.naive.time_to_relevant_file_ms !== null &&
122
+ r.guided.time_to_relevant_file_ms !== null,
123
+ );
124
+ if (both.length === 0) return 0;
125
+ const naiveMean = both.reduce((s, r) => s + r.naive.time_to_relevant_file_ms, 0) / both.length;
126
+ const guidedMean = both.reduce((s, r) => s + r.guided.time_to_relevant_file_ms, 0) / both.length;
127
+ return Math.round(10 * naiveMean / guidedMean) / 10;
128
+ })();
129
+
130
+ /**
131
+ * Human-readable headline in the form used in README.md.
132
+ * Built from the derived constants so README claims stay consistent.
133
+ * @type {string}
134
+ */
135
+ export const HEADLINE_CLAIM =
136
+ `${TOKEN_REDUCTION_PCT}% fewer tokens, ` +
137
+ `${TOOL_CALL_REDUCTION_PCT}% fewer tool calls, ` +
138
+ `${TTRF_SPEEDUP_X}× faster to the right file`;
139
+
140
+ /**
141
+ * Schema version for the benchmark-results module.
142
+ * Bump this when the run schema or summary fields change.
143
+ * @type {string}
144
+ */
145
+ export const BENCHMARK_RESULTS_VERSION = '1.0';
146
+
147
+ // ---------------------------------------------------------------------------
148
+ // formatBenchmarkReport (SAT-460)
149
+ // ---------------------------------------------------------------------------
150
+
151
+ /**
152
+ * Produce a human-readable Markdown benchmark report from the comparison runs
153
+ * stored in this module. The report is deterministic — same input → same
154
+ * output — and safe to embed in wikis, PR comments, or printed to stdout.
155
+ *
156
+ * Layout:
157
+ * 1. Headline summary block (the three key numbers)
158
+ * 2. Per-run table (task id, tokens naive/guided, tool calls, TTRF, correct)
159
+ * 3. Footer with provenance note
160
+ *
161
+ * @param {object} [opts]
162
+ * @param {object[]} [opts.runs=COMPARISON_RUNS] — override for testing.
163
+ * @returns {string} Markdown string, no trailing newline.
164
+ */
165
+ export function formatBenchmarkReport({ runs = COMPARISON_RUNS } = {}) {
166
+ // --- summary numbers (re-derived from the provided runs so the function is
167
+ // self-contained and testable with custom run sets) -------------------
168
+ const totalNaiveTok = runs.reduce((s, r) => s + r.naive.tokens_in + r.naive.tokens_out, 0);
169
+ const totalGuidedTok = runs.reduce((s, r) => s + r.guided.tokens_in + r.guided.tokens_out, 0);
170
+ const totalNaiveCalls = runs.reduce((s, r) => s + r.naive.tool_calls, 0);
171
+ const totalGuidedCalls = runs.reduce((s, r) => s + r.guided.tool_calls, 0);
172
+
173
+ const tokenPct = totalNaiveTok > 0
174
+ ? Math.round(100 * (totalNaiveTok - totalGuidedTok) / totalNaiveTok)
175
+ : 0;
176
+ const callPct = totalNaiveCalls > 0
177
+ ? Math.round(100 * (totalNaiveCalls - totalGuidedCalls) / totalNaiveCalls)
178
+ : 0;
179
+
180
+ const both = runs.filter(
181
+ r => r.naive.time_to_relevant_file_ms !== null &&
182
+ r.guided.time_to_relevant_file_ms !== null,
183
+ );
184
+ const speedup = both.length > 0
185
+ ? Math.round(
186
+ 10 * (both.reduce((s, r) => s + r.naive.time_to_relevant_file_ms, 0) / both.length) /
187
+ (both.reduce((s, r) => s + r.guided.time_to_relevant_file_ms, 0) / both.length),
188
+ ) / 10
189
+ : 0;
190
+ const speedupStr = Number.isInteger(speedup) ? String(speedup) : speedup.toFixed(1);
191
+
192
+ const guidedCorrect = runs.filter(r => r.guided.correct).length;
193
+ const naiveCorrect = runs.filter(r => r.naive.correct).length;
194
+
195
+ // --- headline block -------------------------------------------------------
196
+ const lines = [
197
+ '## KnoSky token-efficiency benchmark (SAT-439)',
198
+ '',
199
+ `**${tokenPct}% fewer tokens · ${callPct}% fewer tool calls · ${speedupStr}× faster to the right file**`,
200
+ '',
201
+ `_${runs.length} tasks, naive agent vs. KnoSky-guided agent._`,
202
+ `_Guided: ${guidedCorrect}/${runs.length} correct. Naive: ${naiveCorrect}/${runs.length} correct._`,
203
+ '',
204
+ '### Per-task results',
205
+ '',
206
+ '| Task | Tokens (naive) | Tokens (guided) | Tool calls (naive) | Tool calls (guided) | TTRF naive (ms) | TTRF guided (ms) | Guided correct |',
207
+ '|------|---------------:|----------------:|-------------------:|--------------------:|----------------:|-----------------:|:--------------:|',
208
+ ];
209
+
210
+ for (const run of runs) {
211
+ const tnaive = run.naive.tokens_in + run.naive.tokens_out;
212
+ const tguided = run.guided.tokens_in + run.guided.tokens_out;
213
+ const ttrfN = run.naive.time_to_relevant_file_ms === null ? '—' : String(run.naive.time_to_relevant_file_ms);
214
+ const ttrfG = run.guided.time_to_relevant_file_ms === null ? '—' : String(run.guided.time_to_relevant_file_ms);
215
+ const correct = run.guided.correct ? '✓' : '✗';
216
+ lines.push(
217
+ `| ${run.task_id} | ${tnaive} | ${tguided} | ${run.naive.tool_calls} | ${run.guided.tool_calls} | ${ttrfN} | ${ttrfG} | ${correct} |`,
218
+ );
219
+ }
220
+
221
+ lines.push('');
222
+ lines.push(`_Data source: \`core/benchmark-results.mjs\` (BENCHMARK_RESULTS_VERSION ${BENCHMARK_RESULTS_VERSION})._`);
223
+
224
+ return lines.join('\n');
225
+ }