savant-code 0.0.17 → 0.0.19

package/dev/fids/archive/FID-2026-0803-006-code-map-audit-hygiene.md

@@ -0,0 +1,243 @@
+# FID: Code-Map Package Audit — Guard Fix, Loader Resilience, Hygiene (CM-1…CM-9)
+
+**Filename:** `FID-2026-0803-006-code-map-audit-hygiene.md`
+**ID:** FID-2026-0803-006
+**Severity:** medium
+**Status:** verified
+**Created:** 2026-08-03
+**Author:** Savant
+
+**Summary:**
+Audit of `packages/code-map` (tree-sitter indexing + language detection; ~900 lines read 0-EOF). Two
+MEDIUM findings — (CM-1) a dead `call in {}` prototype-key guard in `buildTokenCallers` that lets a
+real-world `toString`/`valueOf`/`hasOwnProperty` token collision crash the whole code map (SDK
+degrades to empty scores with only a warning), and (CM-2) the module-level `UnifiedLanguageLoader`
+caches a `Parser.init` rejection forever, silently disabling tree-sitter for the process lifetime with
+no surfaced diagnostic — plus seven LOW hygiene/robustness findings (CM-3 dead wasm-path fallback loop,
+CM-4 pointless rethrow, CM-5 lazy-init race, CM-6 language-table gaps, CM-7 TOCTOU + misclassified read
+errors, CM-8 unnecessary cast, CM-9 eval-style `new Function` shim). Baseline: 50/50 tests, typecheck
+clean, reachable via `sdk/src/run-state.ts` (Law 4 verified).
+
+---
+
+## Environment (RED — findings with evidence)
+
+All in `packages/code-map/src/`; baseline verified: `bun test` 50 pass / 0 fail, `bun run typecheck`
+exit 0, reachability `getFileTokenScores` → `sdk/src/run-state.ts:167` (dynamic import, wrapped in
+try/catch at `:171-174`) + SDK index re-export + `sdk/smoke-test-dist.ts:126`.
+
+### CM-1 — MEDIUM — dead `call in {}` guard can crash the code map
+
+`parse.ts:311` — `if (!definingFile || callingFile === definingFile || call in {}) continue`. The guard
+was meant to skip Object.prototype-colliding tokens, but `'toString' in {}` is **false** (only
+inherited `__proto__` is caught). Trigger path: a project where `toString`/`valueOf`/`hasOwnProperty`
+appears as both a defined identifier (a class overriding `toString` — common) and a call
+(`obj.toString()`). Then `callersByToken[call]` returns the *truthy* `Object.prototype.toString`
+function → `callerFiles.length (1) < 25` passes → `callerFiles.includes(...)` → **TypeError:
+callerFiles.includes is not a function** → `getFileTokenScores` rejects → SDK catches
+(`run-state.ts:171-174`) → entire code map silently empty with a warning. `constructor` is only
+shielded incidentally because `IGNORE_TOKENS` excludes it from scoring.
+
+### CM-2 — MEDIUM — loader caches a rejection; tree-sitter silently disabled forever
+
+`languages.ts:209-216,294` — `unifiedLoader.parserReady` is created eagerly at module scope
+(`this.parserReady = initTreeSitterForNode()`). If `Parser.init` rejects (wasm missing/unreadable —
+including the self-heal download failing once), every subsequent `getLanguageConfig` rejects forever,
+even if the wasm becomes available, and `getLanguageConfig`'s catch (`:301-308`) returns `undefined`
+logging only under `DEBUG_PARSING`. Net: symbol scoring silently disabled for the process lifetime with
+no surfaced cause; plus a possible unhandled-rejection race at module scope.
+
+### CM-3 — LOW — dead fallback loop in `resolveWasmPath`
+
+`languages.ts:174-181` — `for (const wasmPath of possiblePaths) { try { return wasmPath } catch {
+continue } }`: `return` can never throw, so the loop always returns `possiblePaths[0]` and the
+"try each path" intent never runs (the `catch` is dead scaffolding).
+
+### CM-4 — LOW — pointless rethrow
+
+`languages.ts:227` — `catch (err) { throw err }` around the entire `createLanguageConfig` init body is
+a no-op; remove the wrapper.
+
+### CM-5 — LOW — lazy-init race in `createLanguageConfig`
+
+`languages.ts:243-258` — concurrent calls both observe `!cfg.parser` and double-run
+`Language.load` + `new Parser()` + `new Query()`. Currently serial inside `getFileTokenScores`'s loop,
+but `getLanguageConfig` is public API (hosts could `Promise.all` files). Dedupe with a per-config
+in-flight promise.
+
+### CM-6 — LOW — language-table gaps
+
+`languages.ts` `languageTable` — `.mjs`/`.cjs` (ESM/CJS modules — ubiquitous in modern TS/JS projects)
+and `.mts`/`.cts` are unmapped, so those files are silently unindexed. `.c`/`.cc`/`.cxx`/`.h`, Kotlin,
+Swift, PHP, Scala, CSS etc. are absent — but `@vscode/tree-sitter-wasm/wasm/` (verified) ships only
+cpp/c-sharp/css/javascript/typescript, and the repo has no C/CSS queries, so those are correctly
+documented as out of scope rather than half-added.
+
+### CM-7 — LOW — TOCTOU + misclassified read errors
+
+`parse.ts:255-262` — `fs.statSync(filePath).size` then `fs.readFileSync(filePath, 'utf8')`: the file can
+change or vanish between the two calls (uncaught ENOENT lands in `parseTokensWithLimits`'s catch →
+`emptyParsedTokens(false)` at `:243`). The readFile-provider rejection path returns
+`emptyParsedTokens(false)` at `:183` too — a failed read is classified as *parsed*, consuming a
+`MAX_PARSE_FILES` slot with zero tokens. Should be `skipped: true`.
+
+### CM-8 — LOW — unnecessary cast
+
+`parse.ts:387` — `(tree as { delete?: () => void }).delete?.()`. `Tree.delete(): void` exists in
+web-tree-sitter 0.25.10 types (verified `node_modules/web-tree-sitter/tree-sitter.d.ts:338`). Use
+`tree.delete()` directly; keep the `finally` (memory hygiene is correct).
+
+### CM-9 — LOW — eval-style `__dirname` shim
+
+`utils.ts:2` — `new Function('try { return __dirname; } ...')`. Works in the private runtime but is
+CSP/sandbox-hostile and unwrapped at the call site (`resolveWasmPath` calls it outside any try).
+Add an outer try/catch and keep the documented eslint-disable.
+
+---
+
+## Root Cause
+
+Two classes: (1) a broken guard expression (`call in {}` — always-false except `__proto__`) that was
+never exercised by the test corpus (no test uses Object.prototype-colliding tokens), and (2) a
+module-level singleton that treats an init failure as terminal state without surfacing it. The rest are
+accumulated small dead-code/robustness warts in a package that predates the current quality program.
+
+## Proposed Solution (after approval — audit-only now)
+
+1. **CM-1** — `parse.ts:311`: `call in {}` → `call in Object.prototype` (one token; covers
+   `constructor`/`toString`/`valueOf`/`hasOwnProperty`/`isPrototypeOf`/`propertyIsEnumerable`/
+   `toLocaleString`/`__proto__`).
+2. **CM-2** — `languages.ts`: rework `UnifiedLanguageLoader` — lazy `initParser` that stores the
+   in-flight promise (concurrent callers share it) but clears it on rejection so a later call retries;
+   no eager module-scope promise (removes the unhandled-rejection risk). Add a one-time `console.warn`
+   in `getLanguageConfig`'s catch (with eslint-disable) so a silently-degrading subsystem surfaces
+   its cause once per process.
+3. **CM-3** — `languages.ts:174-181`: collapse the dead loop to `return possiblePaths[0]` with a
+   comment (behavior-identical — the loop always returned the first path).
+4. **CM-4** — `languages.ts`: remove the `try { … } catch (err) { throw err }` wrapper in
+   `createLanguageConfig`.
+5. **CM-5** — `languages.ts`: add optional `initPromise?: Promise<void>` to `LanguageConfig`; in
+   `createLanguageConfig`, `cfg.initPromise ??= (async () => { …init… })()` then `await` it, clearing
+   it in a catch so a failed init can retry.
+6. **CM-6** — `languages.ts` `languageTable`: add `.mjs`/`.cjs` → javascript wasm/query and
+   `.mts`/`.cts` → typescript wasm/query. Document the out-of-scope boundary (C-family, Kotlin, Swift,
+   PHP, CSS) in a table comment.
+7. **CM-7** — `parse.ts`: readFile-provider rejection → `emptyParsedTokens(true)`; wrap the stat-path
+   `readFileSync` in try/catch → `null` (skipped) on failure, closing the TOCTOU window.
+8. **CM-8** — `parse.ts:387`: `tree.delete()` directly.
+9. **CM-9** — `utils.ts`: wrap the `new Function` invocation in try/catch returning `undefined` on
+   failure (callers already fall back to `process.cwd()`).
+
+**Non-goals:** adding C/Kotlin/Swift/PHP/CSS grammars (wasm/query authoring is a separate feature
+track); worker-thread parallel parsing (design note, larger change); changing the SDK's degraded-mode
+handling of empty scores.
+
+## Files To Be Changed (implementation stage)
+
+- `packages/code-map/src/parse.ts` (CM-1, CM-7, CM-8)
+- `packages/code-map/src/languages.ts` (CM-2, CM-3, CM-4, CM-5, CM-6)
+- `packages/code-map/src/utils.ts` (CM-9)
+- Tests: `packages/code-map/__tests__/parse.test.ts` or `integration.test.ts` (CM-1 regression —
+  `toString`-collision corpus must not throw), `__tests__/languages.test.ts` (CM-6 — new extensions
+  resolve), optional concurrent-init identity test (CM-5)
+
+## Verification
+
+- [x] Recon: 5 source files read 0-EOF; tests 50/50; typecheck clean; reachability verified
+      (`run-state.ts:167`, SDK index export, smoke-test asserts `getFileTokenScores`)
+- [x] `call in {}` semantics confirmed (`'toString' in {}` === false); `Tree.delete(): void` confirmed
+      in web-tree-sitter types (d.ts:338); wasm manifest confirmed (`@vscode/tree-sitter-wasm/wasm/`)
+- [x] No implementation files modified during this audit (audit-only)
+- [x] Implementation: code-map 51 pass / 0 fail (incl. 1 new CM-1 regression), `bun run typecheck` clean,
+      zero-warning ESLint (full-repo `--max-warnings 0` exit 0), Prettier clean on changed files,
+      `bun run lint:md` exit 0
+- [x] Implementation: SDK smoke caveat noted (smoke-test-dist requires a built dist absent from the
+      tree; CJS tree-sitter harness failure pre-existing/environmental — `getFileTokenScores` export
+      verified at source level via typecheck); independent AUDIT via code-reviewer (clean); CHANGELOG
+      entry added; FID archived
+
+## Perfection Loop
+
+### Loop 1
+
+- **RED:** Completed 2026-08-03 — findings CM-1…CM-9 catalogued with file:line evidence and runtime
+  verification (50/50 tests, clean typecheck, `Tree.delete` exists, wasm manifest inspected, SDK
+  caller + degradation path confirmed at `run-state.ts:171-174`).
+- **GREEN:** Sequential thinking over each finding (see Summary). Missed-question check ("What
+  questions should I have asked when this FID was created, but failed to?"):
+  1. *Is the `call in {}` crash actually reachable with real tokens?* — Yes: `toString`/`valueOf`/
+     `hasOwnProperty` are both ordinary identifiers and Object.prototype keys; `constructor` is only
+     shielded by `IGNORE_TOKENS` scoring exclusion. The `callersByToken[call]` truthy-function path is
+     confirmed (`length 1 < 25` then missing `.includes` → TypeError).
+  2. *Does the crash surface loudly?* — No: the SDK catches it and continues with empty scores
+     (warning only). So CM-1 is feature-loss, not a hard crash — severity MEDIUM, not CRITICAL.
+  3. *Is the CM-2 retry safe against double-init?* — Yes: concurrent callers share one in-flight
+     promise (stored before await); only a rejection clears it for retry.
+  4. *Does collapsing `resolveWasmPath` change behavior?* — No: the loop always returned
+     `possiblePaths[0]` (its try/catch could never throw); collapse is behavior-identical.
+- **AUDIT:** Document-audit before implementation: CM-1 fix is a one-token expression change
+  verified against `Object.prototype` member semantics; CM-2/CM-5 fixes follow the loader's existing
+  promise-dedup pattern (`parserReady`) and are race-checked (single in-flight promise, clear-on-
+  rejection); CM-6 additions reuse existing wasm/query pairs (no new grammars); CM-7's `skipped: true`
+  classification is consistent with `getFileTokenScores`'s `if (parsed.skipped) continue` guard;
+  CM-8 verified against the shipped type declarations. Call-graph evidence in Verification. Zero new
+  public exports beyond the optional `LanguageConfig.initPromise` field (backward-compatible).
+- **SELF-CORRECT:** Initial scan flagged CM-6 as "language coverage gaps" broadly; corrected during
+  GREEN: the wasm manifest inspection bounded the fix to the four free JS/TS-family extensions and
+  moved C-family/Kotlin/Swift/etc. to documented non-goals (writing new grammars is a feature track,
+  not hygiene). CM-1 severity held at MEDIUM after confirming the SDK's catch-and-degrade path.
+- **SELF-CORRECT (implementation):** CM-8 deviated from the FID's literal "use `tree.delete()`
+  directly" — the optional call was retained (`tree.delete?.()`): web-tree-sitter's Tree declares
+  `delete()`, but structurally-compatible test mocks omit it (8 mock tests failed on the direct call),
+  and the cast (the actual Law-6 violation) is gone. An editing slip during CM-8 briefly dropped the
+  call entirely (leaving only a comment — a production memory-leak regression); caught and restored in
+  the immediate follow-up edit. CM-7 was completed in response to the independent AUDIT: the TOCTOU
+  try/catch now wraps `statSync` as well as `readFileSync`, so a file that vanishes before the stat
+  also classifies as skipped rather than parsed. SDK `smoke-test-dist` was found to require a built
+  dist absent from the working tree — its CJS tree-sitter failure (`import_code_map is not defined`)
+  is pre-existing/environmental, not caused by this FID.
+
+## Lessons Learned
+
+1. A guard expression that is always-false (`call in {}`) is indistinguishable from no guard at all —
+   prototype-key collisions (`toString`, `valueOf`) are a real, common-input crash class. When
+   writing "skip Object.prototype keys" logic, verify with `Object.prototype` explicitly.
+2. Module-level singletons that cache a rejected init promise turn a transient failure into permanent
+   silent disablement; clear-on-rejection + a one-time warn turns it into a retryable, diagnosable
+   condition.
+
+## Resolution
+
+- **Fixed By:** Savant (operator-approved implementation — FID presented for approval, approved, then
+  implemented)
+- **Fixed Date:** 2026-08-03
+- **Fix Description:** CM-1 — `parse.ts` `buildTokenCallers` guard `call in {}` → `call in
+  Object.prototype` (covers constructor/toString/valueOf/hasOwnProperty/etc.; the old guard only
+  caught `__proto__` and let a real `toString` collision crash the code map). CM-2 —
+  `UnifiedLanguageLoader` rewritten: lazy `initParser` sharing one in-flight promise across concurrent
+  callers, clearing it on rejection so a later call retries (no more permanent silent disablement),
+  plus a one-time `console.warn` in `getLanguageConfig` surfacing the cause; no eager module-scope
+  promise (removes the unhandled-rejection risk). CM-3 — dead wasm-path fallback loop collapsed to
+  `return possiblePaths[0]` (behavior-identical). CM-4 — no-op `catch (err) { throw err }` removed.
+  CM-5 — `LanguageConfig.initPromise` dedupes concurrent `createLanguageConfig` inits, cleared on
+  failure for retry. CM-6 — `.mjs`/`.cjs` → javascript and `.mts`/`.cts` → typescript rows added to
+  `languageTable` (same grammars; C-family/Kotlin/Swift/CSS documented out-of-scope). CM-7 — read
+  failures (provider-reject and fs paths) classify as `skipped: true` instead of consuming a parse
+  slot; the fs path now does stat+read in one try/catch window (TOCTOU closed, including a vanished-
+  before-stat file). CM-8 — the `as { delete?: … }` cast removed; `tree.delete?.()` retained (the
+  declared type has `delete()`, the optional call guards structurally-compatible mocks). CM-9 —
+  `getDirnameDynamically` wrapped in an outer try/catch (CSP/sandbox-safe, callers already fall back
+  to `process.cwd()`).
+- **Tests Added:** `integration.test.ts` +1 — CM-1 regression: a `toString`-defined + `toString`-
+  called corpus must resolve without throwing and produce no `toString` caller entry (fails on the old
+  code). `languages.test.ts` — `languageTable.length` 10→12 and the `expectedLanguages` list extended
+  for `.mjs`/`.cjs`/`.mts`/`.cts`. Existing 8 mock parse tests exercised the CM-8 guard (they fail on
+  a direct `tree.delete()`), confirming the retained `?.`.
+- **Verified By:** Savant — independent AUDIT via code-reviewer (clean — no CRITICAL/HIGH/MEDIUM;
+  CM-7 statSync window completed in response; CM-8 `?.` deviation documented). Gate suite: code-map
+  51 pass / 0 fail, code-map typecheck clean, full-repo `bun x eslint . --max-warnings 0` exit 0,
+  `bun run lint:md` exit 0, Prettier clean. SDK smoke caveat: `smoke-test-dist` requires a built dist
+  (absent in the tree); the CJS tree-sitter failure is pre-existing/environmental; the
+  `getFileTokenScores` export assertion holds at source level (typecheck).
+- **Commit/PR:** None (working tree)
+- **Archived:** Yes

package/dev/fids/archive/FID-2026-0803-007-evals-benchmark-audit-hygiene.md

@@ -0,0 +1,296 @@
+# FID: Evals Benchmark Runner Audit (EV-1…EV-11)
+
+**Filename:** `FID-2026-0803-007-evals-benchmark-audit-hygiene.md`
+**ID:** FID-2026-0803-007
+**Severity:** high
+**Status:** verified
+**Created:** 2026-08-03
+**Author:** Savant
+
+**Summary:**
+Audit of the evals benchmark harness (`evals/benchmark/*` + `evals/logger.ts`) for
+correctness, error handling, and type safety. Headline: **the evals package fails
+typecheck** (`tsc --noEmit` exits 2) with two errors that are invisible to the
+CI hard gate (AGENTS.md gates only sdk/common/agent-runtime/cli). Both broken
+lines are botched "fixes" from a prior audit that were never verified to compile.
+
+---
+
+## Findings
+
+### EV-1 (HIGH) — evals package typecheck is broken, invisible to CI
+
+`bun run typecheck` exits **2** with two errors. Neither surfaced because evals is
+not in the AGENTS.md hard gate, and both were introduced by the
+FID-2026-0802-006 "fix" pass without a compile check.
+
+### EV-1a (HIGH) — lying cast in trace-analyzer that fails tsc
+
+`evals/benchmark/trace-analyzer.ts:222`:
+
+```ts
+return output.value as unknown as {
+  overallAnalysis: string
+  agentFeedback: unknown[]
+}
+```
+
+The cast target `agentFeedback: unknown[]` **does not satisfy the declared return
+type** (`Array<{ agentId; strengths; weaknesses; recommendations }>`) → TS2322.
+The FID-0802-006 DEBT1 comment claims this was the "concrete cast instead of
+`any`" fix, but it both weakens the type and does not compile. Downstream,
+`format-output.ts:159-193` (`formatTraceAnalysis`) reads `feedback.agentId` and
+`feedback.strengths.length` directly — malformed output would TypeError.
+
+**Fix:** define `TraceAnalyzerResultSchema` with zod (mirroring `JudgingResultSchema`
+in judge.ts) and `safeParse` the value; on failure fall through to the existing
+error branch (which already returns the correct fallback shape).
+
+### EV-1b (HIGH) — pino "fix" that never compiled
+
+`evals/logger.ts:21`:
+
+```ts
+}) => ReturnType<(typeof pino)['destination']>
+```
+
+`destination` does not exist on `typeof pino` in the installed pino 9.6 types →
+TS2339:21:34. pino 9 removed static `destination`/`transport` from its type
+declarations (they exist at runtime). The `PinoWithStaticDestination` intersection
+references the missing member in its own return type.
+
+**Fix:** import `DestinationStream` (exported — verified `pino.d.ts:282`) and type
+the return as `DestinationStream`:
+
+```ts
+type PinoWithStaticDestination = typeof pino & {
+  destination: (opts: {
+    dest: string
+    mkdir: boolean
+    sync: boolean
+  }) => DestinationStream
+}
+```
+
+### EV-2 (MEDIUM) — JudgingResultSchema defined but never used
+
+`evals/benchmark/judge.ts:12` defines and exports `JudgingResultSchema`; the zod
+schema is **dead code** — `runSingleJudge` at `:203` does
+`judgeResult.output.value as JudgingResult` with zero runtime validation.
+Malformed model output (missing `analysis`, string `overallScore`) flows straight
+into score averaging → NaN aggregates, or a TypeError in
+`format-output.ts:183-187`. The schema was clearly built for this purpose.
+
+**Fix:** `const parsed = JudgingResultSchema.safeParse(judgeResult.output.value)`;
+on failure log + `return null` (falls into the existing all-failed branch).
+
+### EV-3 (MEDIUM) — timeouts don't abort the underlying work
+
+`withTimeout` (`common/src/util/promise.ts:68`) races and rejects on timeout, but
+the inner `client.run` promise **keeps executing** — a timed-out 20-min judge
+keeps burning API dollars and keeps the event loop alive (process won't exit).
+Call sites: `judge.ts:160` (20 min), `trace-analyzer.ts:187` (20 min),
+`meta-analyzer.ts:198` (30 min), `agent-runner.ts:64` (60 min, wraps repo setup +
+runner).
+
+The SDK already supports abort: `sdk/src/run.ts:181` (`signal?: AbortSignal`),
+`:294` and `:546` check `signal?.aborted`.
+
+**Fix:**
+
+- `judge.ts`, `trace-analyzer.ts`, `meta-analyzer.ts`: drop the `withTimeout`
+  wrapper; pass `signal: AbortSignal.timeout(<ms>)` in the `client.run` options.
+  On abort, `run()` rejects with an AbortError → existing catch branches handle it.
+- `agent-runner.ts`: keep the outer `withTimeout` as the overall budget, and
+  additionally thread an `AbortSignal` into `SavantCodeRunner` (new optional
+  constructor opt) → its `client.run` call, so the inner LLM loop actually stops.
+- **Out of scope (documented):** external CLI runners (claude/codex/opencode) —
+  those spawn child processes; kill semantics are a separate mechanism.
+
+### EV-4 (MEDIUM) — dead, mislabeled judge agent
+
+`judge.ts:134-136`: `judgeAgents['judge-sonnet']` has `id: 'judge-claude'`
+(key/id mismatch), and only gpt+gemini are ever invoked (`:211-215`). The
+"2 judges in parallel" comment is accurate — the sonnet entry is unreachable dead
+code with a confusing name. `Object.values(judgeAgents)` also passes it as an
+available definition on every run.
+
+**Fix:** delete the `'judge-sonnet'` entry.
+
+### EV-5 (MEDIUM) — median picker always favors the higher scorer
+
+`judge.ts:276-277`: with 2 valid results sorted ascending,
+`medianIndex = floor(2/2) = 1` → the analysis text always comes from the
+*higher*-scoring judge, never the lower — a systematic bias when judges disagree.
+
+**Fix:** `const medianIndex = Math.floor((sortedResults.length - 1) / 2)` — true
+lower median (identical for odd n, conservative for even n; text cannot be
+averaged).
+
+### EV-6 (LOW) — `catch (error: any)` on exec failure
+
+`agent-runner.ts:185`: the caught error from `execAsync` has
+`code/stdout/stderr`. `any` bypasses the error contract.
+
+**Fix:** `catch (error)` + narrow:
+`const execError = error as { code?: number; stdout?: string; stderr?: string }`.
+
+### EV-7 (LOW) — 9× `any[]` where `AgentDefinition[]` fits
+
+`run-benchmark.ts:55,59`, `trace-analyzer.ts:118`, `agent-runner.ts:40`,
+`lessons-extractor.ts:21`, `meta-analyzer.ts:151`, `eval-task-generator.ts:107`,
+`runners/savant.ts:15,25` — all just pass agent definitions into `client.run`,
+which already types them as `AgentDefinition[]` (SDK export, already imported in
+every one of these files). Also `setup-test-repo.ts:44` (`options: any`) and
+`gen-repo-eval.ts:30` (`(c: any)`).
+
+**Fix:** replace with `AgentDefinition[]` / `{ sha: string }` / typed options.
+Compiler-arbitrated (typecheck fails if any site needs more than the import).
+
+### EV-8 (LOW) — untyped map callbacks in truncateTrace
+
+`trace-utils.ts:14,18,38`: `(item: any)` / `(file: any)` in `truncateTrace` map
+callbacks. The tool-result output union exists in the codebase.
+
+**Fix:** type as the tool-result output item shape (narrowed local type) with the
+`?.` access already present.
+
+### EV-9 (LOW) — meta-analyzer cast feeds manual validation (downgraded)
+
+`meta-analyzer.ts:278`: `output.value as unknown as Record<string, unknown>`.
+0-EOF read shows the downstream code **already defensively validates** (typeof /
+Array.isArray guards, string filters) — so this is NOT a crash path like EV-2.
+
+**Fix:** for consistency with EV-1a/EV-2, define `MetaAnalyzerResultSchema` (zod)
+and parse; or document the manual validation as intentional. Implement the zod
+variant only if it lands cleanly with the EV-1a schema work; otherwise leave with
+a confirming comment.
+
+### EV-10 (LOW) — bare `catch {` swallows diagnostics
+
+`analyze-task-scores.ts:98,165`, `filter-supplemental-files.ts:23,70`:
+`catch {` with no error binding.
+
+**Fix:** `catch (error)` + `console.warn`/`getErrorObject` (pattern already used
+elsewhere in these files).
+
+### EV-11 (LOW) — partial agent trace lost on timeout/error
+
+`agent-runner.ts:141-149`: on abort/exception, `trace` stays empty and `cost`
+stays 0 — up to 60 min of work discarded with no record of what the agent did
+before dying. `SavantCodeRunner` collects `steps` locally and only returns them
+on success.
+
+**Fix:** add optional `traceSink?: AgentStep[]` constructor opt to
+`SavantCodeRunner`; push each event to both local `steps` and the sink.
+`agent-runner` passes its `trace` array → partial trace survives an abort, and
+the existing error string is recorded.
+
+---
+
+## Perfection Loop trace (Loop 1)
+
+- **RED** — Four-pass scan. All findings verified against live code: `tsc` run
+  twice (exit 2, both errors reproduced); `JudgingResultSchema` usage grep (only
+  definition + type-infer); `AbortSignal` support verified in `sdk/src/run.ts`
+  (181/294/546); pino `DestinationStream` export verified in `pino.d.ts:282`;
+  `Runner` interface + `SavantCodeRunner.run()` read 0-EOF; `localAgentDefinitions`
+  dataflow traced main.ts → run-benchmark → agent-runner → savant.ts (all `any[]`,
+  all feeding `client.run`'s typed param); `format-output.ts:159-193` consumer
+  confirmed. Zero CRITICAL; nothing data-destroying.
+- **GREEN** — Missed-questions answered: *is the typecheck failure real or a
+  harness artifact?* (real — plain `tsc --noEmit -p .` exits 2); *would EV-2
+  crash or just produce NaN?* (both possible — NaN averages or a TypeError in
+  format-output); *is EV-3 worth SDK surgery?* (no — SDK already supports
+  `signal`); *does removing judge-sonnet change the 2-judge ensemble?* (no —
+  only gpt+gemini are ever invoked); *is EV-5's lower-median unbiased?* (no
+  selection scheme is unbiased for n=2; lower-median is deterministic and
+  conservative, and identical to current behavior for odd n).
+- **AUDIT** — EV-1a/1b are compiler-arbitrated (typecheck fails if the fix is
+  wrong); EV-2 mirrors the already-present fallback branch (all-failed → zeroed
+  result); EV-3 abort path lands in existing catch branches (no new control
+  flow); EV-11's `traceSink` is a backward-compatible optional constructor opt
+  (existing callers unaffected); zod schemas follow the judge.ts pattern already
+  in the file. Zero new public SDK/config surface.
+- **SELF-CORRECT** — EV-9 downgraded from "lying cast" to "defensive-validated,
+  optional schema" after the 0-EOF read of meta-analyzer.ts; EV-3 scoped to the
+  three direct `client.run` sites + SavantCodeRunner (external CLI runners
+  documented out of scope); EV-5 fix chosen as lower-median index rather than
+  "average the two analyses" (text cannot be averaged); EV-11 fix shaped as a
+  trace sink rather than restructuring `RunnerResult`.
+
+**Files touched on approval:** 10 files — `judge.ts`, `trace-analyzer.ts`,
+`meta-analyzer.ts`, `agent-runner.ts`, `trace-utils.ts`, `logger.ts`,
+`run-benchmark.ts`, `eval-task-generator.ts`, `lessons-extractor.ts`,
+`runners/savant.ts` (+ `setup-test-repo.ts`/`gen-repo-eval.ts` if EV-7's one-off
+anys are in scope).
+
+**Verification on approval:** evals typecheck (must be exit 0 — the headline),
+evals tests (67 v2 tests), sdk/common typechecks (touched files import from
+them), full-repo ESLint `--max-warnings 0`, Prettier, markdownlint, independent
+code-reviewer pass, CHANGELOG entry + LEARNINGS + session summary, archive.
+
+---
+
+## Resolution
+
+**Implemented:** 2026-08-03, all EV-1…EV-11 as specified.
+
+- **EV-1a** — `trace-analyzer.ts`: added `TraceAnalyzerResultSchema` (zod,
+  mirroring `JudgingResultSchema`) and `safeParse`; malformed output falls
+  through to the existing error branch. Replaces the `unknown[]` cast that
+  failed TS2322.
+- **EV-1b** — `logger.ts`: `PinoWithStaticDestination.destination` now returns
+the exported `DestinationStream` type (verified `pino.d.ts:861`; `pino()
+accepts it at :828`). Fixes TS2339:21:34.
+- **EV-2** — `judge.ts`: `JudgingResultSchema.safeParse` in `runSingleJudge`;
+  failure → `return null` → existing all-failed branch.
+- **EV-3** — `judge.ts`/`trace-analyzer.ts`/`meta-analyzer.ts`/`lessons-extractor.ts`:
+  `withTimeout(client.run(...))` replaced with `signal: AbortSignal.timeout(...)`
+  (SDK already supports abort — `sdk/src/run.ts:181/294/546`); `agent-runner.ts`
+  keeps the outer `withTimeout` and additionally threads a 60-min
+  `AbortSignal.timeout` + `traceSink` into `SavantCodeRunner` → its `client.run`.
+  **Recorded extension:** `lessons-extractor.ts:197` had the identical
+  `withTimeout` pattern (missed in the scan) and was fixed for consistency.
+  **Recorded nuance:** the abort error message replaces the descriptive
+  "timed out after N minutes" text in logs — acceptable.
+- **EV-4** — `judge.ts`: `'judge-sonnet'` entry deleted (`Object.values` now
+  yields the 2 used judges).
+- **EV-5** — `judge.ts`: `Math.floor((n-1)/2)` (true lower median).
+  **Recorded behavior change:** for n=2 the analysis text now comes from the
+  lower scorer — deliberate per the approved spec.
+- **EV-6** — `agent-runner.ts`: `catch (error: any)` → narrowed
+  `{ code?; stdout?; stderr?; message? }`.
+- **EV-7** — `any[]` → `AgentDefinition[]` at 9 sites + `options: any` →
+  `ExecFileSyncOptions` + `(c: any)` → `{ sha: string }`. Boundary verified:
+  `LoadedAgentDefinition extends AgentDefinition`.
+- **EV-8** — `trace-utils.ts`: typed with `ToolResultOutput` +
+  `isJsonObject` guard; truncation semantics preserved.
+- **EV-9** — `meta-analyzer.ts`: confirming comment only (manual defensive
+  validation is intentional — the FID's documented fallback branch).
+- **EV-10** — `analyze-task-scores.ts:98,165`: `catch (error)` +
+  `console.warn`. **Recorded deviation:** the two `filter-supplemental-files.ts`
+  sites are expected-failure probes (file-not-at-commit / commit-may-already-
+exist) — a warn per missing file would spam, so bare `catch {` retained with
+  intent comments instead of the FID's blanket `catch (error) + warn`.
+- **EV-11** — `SavantCodeRunner` gains optional `traceSink?: AgentStep[]`;
+  `agent-runner.ts` passes its `trace` so partial steps survive an abort, and
+  skips the success-path push for savant (no double-push — keyed on
+  `externalAgentType`).
+
+**Verification (all green):**
+
+- **evals typecheck: exit 0** (was 2 — the headline fix)
+- evals tests: 67 pass / 0 fail · sdk + common typechecks: 0 errors
+- full-repo ESLint `--max-warnings 0`: exit 0 · Prettier: clean · markdownlint: exit 0
+- Independent code review (code-reviewer-deepseek-flash): clean — no
+  CRITICAL/HIGH/MEDIUM; three recorded notes above all captured in this Resolution.
+
+**SELF-CORRECT during implementation:**
+- EV-10 `filter-supplemental-files.ts` narrowed from "add warn everywhere" to
+  "diagnostics for real failures, intent comments for expected-failure probes".
+- EV-3 scope extended to `lessons-extractor.ts` (identical pattern discovered).
+- EV-11 shaped as a `traceSink` (backward-compatible optional ctor opt) with a
+  double-push guard instead of restructuring `RunnerResult`.
+
+**Fixed Date:** 2026-08-03 · **Author:** Savant

package/dev/fids/archive/FID-2026-0803-009-echo-enforcement-doc-drift.md

@@ -0,0 +1,112 @@
+# FID: ECHO Enforcement Layer Documentation Drift (EC-1..EC-4)
+
+**Filename:** `FID-2026-0803-009-echo-enforcement-doc-drift.md`
+**ID:** FID-2026-0803-009
+**Severity:** low
+**Status:** analyzed
+**Created:** 2026-08-03
+**Author:** Savant
+
+**Summary:**
+Audit of the ECHO Protocol enforcement layer (`agents/savant/savant.ts`, `protocol.config.yaml`,
+`ARCHITECTURE.md`, `ECHO.md` v0.2.0) found the *runtime* enforcement fully ECHO-compliant (FSM gating in
+`tool-executor.ts:386-467`, SoD holds — Forge has no bash, Detective has no write tools, Verifier is zero-tool) with
+**4 LOW documentation-drift items** in the bootstrap doc `ECHO.md` and one config comment. No CRITICAL/HIGH/MEDIUM.
+
+---
+
+## RED — Findings
+
+### EC-1 (LOW) — ECHO.md Researcher row is stale
+
+`ECHO.md:64` lists the Researcher tools as `web_search, read_url` only. There are **two** researcher agents:
+`researcher-web` (`toolNames: ['web_search', 'read_url']`, researcher-web.ts:21) and `researcher-docs`
+(`toolNames: ['read_docs']`, researcher-docs.ts:22). Since ECHO.md is the bootstrap doc the Orchestrator reads, it
+never learns `read_docs` exists. `ARCHITECTURE.md:33` already documents this correctly
+(`web_search, read_url (web); read_docs (docs)`).
+
+### EC-2 (LOW) — ECHO.md "9 specialized agents" omits the 4 infra spawnables
+
+`ECHO.md:49-51` states the harness "enforces the Perfection Loop through 9 specialized agents", but `savant.ts:123-137`
+`spawnableAgents` actually lists **13**: the 9 canonical + `basher`, `tmux-cli`, `browser-use`, `context-pruner`
+(infrastructure helpers). `ARCHITECTURE.md:212-238` covers this ("Helper Tool Libraries (Filesystem-Only)") and
+explicitly warns "Future checklists/audits should not confuse them." ECHO.md is silent on the distinction.
+
+### EC-3 (LOW) — ECHO.md Forge "Restricted Tools" cell lists bash it never has
+
+`ECHO.md:59` Forge restricted cell reads `spawn_agents, bash (destructive), ask_user`. `forge.ts:33` is
+`toolNames: ['write_file', 'str_replace', 'set_output']` with **zero** `bash` references in the file — bash is never
+granted, so listing it as "restricted" is misleading for a bootstrap doc. The SoD table (ECHO.md:72) correctly says
+"No bash (test) access".
+
+### EC-4 (LOW) — protocol.config.yaml `commands.build` is a partial build
+
+`protocol.config.yaml:28` `build: 'bun run ci'` compiles only SDK + Savant-Free (`package.json:29`:
+`ci: "bun run build:sdk && bun run build:savant-free"`) — the CLI, agent-runtime, code-map, database, and
+llm-providers are not compiled. The inline comment acknowledges this ("use type_check + lint_md for quality gates"),
+but ECHO Law 1 defines `commands.build` as "your compile/build command". Verified: **no code reads `commands.build`**
+— it is a pure documentation surface for agents, so a comment fix is safe and sufficient.
+
+---
+
+## GREEN — Proposed Fixes (minimal, docs/config only)
+
+| # | File | Change |
+|---|---|---|
+| **EC-1** | `ECHO.md:64` | Researcher tools cell: `web_search, read_url` → `web_search, read_url (web); read_docs (docs)` (match ARCHITECTURE.md:33) |
+| **EC-2** | `ECHO.md:49-51` | Add footnote after roster intro: 4 infra spawnables (`basher`, `tmux-cli`, `browser-use`, `context-pruner`) are helper/infra agents not in the 9-agent roster — see ARCHITECTURE.md → Helper Tool Libraries |
+| **EC-3** | `ECHO.md:59` | Forge restricted cell: `spawn_agents, bash (destructive), ask_user` → `spawn_agents, ask_user` (bash never granted) |
+| **EC-4** | `protocol.config.yaml:28` | Rewrite comment: build = release-artifact build (SDK + Savant-Free); full 9-workspace compile gate is `type_check` |
+
+**Rejected alternatives (recorded):**
+
+- Changing `commands.build` value to the full `type_check` chain — rejected: `bun run ci` is the release-pipeline
+  entrypoint used by CI; the semantic split (build artifacts vs compile gate) is the correct model, it only needed
+  documentation. Comment-only fix, zero runtime risk.
+- Deleting/rewriting the ECHO.md roster table wholesale — rejected: 7/9 rows are accurate; only Researcher (EC-1) and
+  Forge restricted cell (EC-3) need edits. Minimal-change principle (Law: MINIMAL changes).
+
+---
+
+## AUDIT — Verification Plan
+
+1. **markdownlint** on `ECHO.md` (`bun run lint:md`) — table edits must not break MD tables (MD055/MD056).
+2. **YAML parse** of `protocol.config.yaml` (`bun -e` with `yaml` pkg) — comment rewrite must not break the file.
+3. **Table alignment** — re-grep the roster table: Researcher row contains `read_docs`, Forge cell no longer lists
+   `bash`, footnote present after intro.
+4. **No stale refs** — confirm no other tracked `.md` repeats the stale `web_search, read_url` Researcher cell
+   (known: only ECHO.md:64; ARCHITECTURE.md:33 is already correct).
+5. **No code change** — this FID touches docs/config only; typecheck not required (comment-only + markdown edits).
+
+**Double-audit requirement:** static (grep verification) + runtime (markdownlint + YAML parse) — two independent
+methods per AUDIT phase.
+
+---
+
+## Resolution
+
+**Status:** verified — **Fixed Date:** 2026-08-03 · **Verified By:** Savant
+(double audit: static grep + markdownlint/YAML parse)
+
+**Changes applied (all 4, per approved GREEN section):**
+
+- **EC-1** — `ECHO.md:69` Researcher row tools cell → `web_search, read_url (web); read_docs (docs)`
+  (matches ARCHITECTURE.md:33 and the two real researcher agents).
+- **EC-2** — `ECHO.md:55-58` added roster-intro footnote: 4 infra spawnables (`basher`, `tmux-cli`,
+  `browser-use`, `context-pruner`) are NOT independent ECHO conversation roles → ARCHITECTURE.md →
+  "Helper Tool Libraries".
+- **EC-3** — `ECHO.md:64` Forge restricted cell → `spawn_agents, ask_user` (bash never granted;
+  forge.ts:33 has zero bash). SoD table row (ECHO.md:77 "No bash (test) access") now consistent with the roster.
+- **EC-4** — `protocol.config.yaml:28` build comment rewritten: release-artifact build (SDK + Savant-Free)
+  vs. full 9-workspace compile gate (`type_check`).
+
+**Verification (double audit — two independent methods):**
+
+1. **Static grep** — each edit verified in place: Researcher row carries `read_docs (docs)`, Forge cell no
+   longer lists `bash`, footnote present after roster intro, `commands.build` comment rewritten. No other
+   tracked `.md` carries the stale Researcher cell (only ARCHITECTURE.md:33, already correct).
+2. **Runtime** — `bun run lint:md` exits 0 after FID spacing fixes; `protocol.config.yaml` parses via `yaml`
+   pkg (`YAML_OK commands.build = bun run ci`).
+
+**Scope guard:** doc/config only — no TypeScript touched, so typecheck not required (per AUDIT plan item 5).
+No code changed; nothing to call-graph verify (Law 4 N/A).