deadbyte-mcp 0.8.9 → 0.9.0

package/docs/superpowers/plans/2026-09-17-r35-production-successor.md

@@ -0,0 +1,408 @@
+# DEADBYTE MCP R35 Production Successor Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build R35 / V0.9.0 as a compatibility-preserving successor to sealed R34 with machine-checked tool-surface archaeology, four required-language semantic backends plus optional Go evidence, provenance-aware context, durable critic/repair loops, signed provider circuit state, reproducible capability benchmarks, resumable remote sessions, and a rollback/restore Final closure to exact R34.
+
+**Architecture:** Copy the exact sealed R34 package tree into a separate R35 workspace, preserve existing public tool names, and add narrowly bounded modules behind existing seams. Every slice is test-first and releases through the same immutable slot/evidence machinery, with new R35 authorities deriving from canonical registries rather than duplicated arrays.
+
+**Tech Stack:** Node.js ESM, `@modelcontextprotocol/*`, Zod v4, TypeScript language service, JSON-RPC/LSP over stdio for external language servers, Windows PowerShell 5.1 controller/release gates, native Windows containment/process hosts, Ed25519 evidence receipts.
+
+**Spec:** `docs/superpowers/specs/2026-09-17-r35-production-successor-design.md`
+
+## Global Constraints
+
+- Parent release is exactly `v0.8.11-6c01e5285e065220` with manifest `6c01e5285e065220db07d7719f78bb6aa4fde88365419281b33be1b1d59dd3c5`.
+- Never mutate the sealed R34 release slot or the R34 development/final workspace in place.
+- R35 version is `0.9.0`; release label is `R35` until Final closure proves a stronger label.
+- Preserve R34's 73 public tool names unless an explicit versioned R35 surface test approves an addition.
+- Process, coding, autonomous, and host authority never auto-arm.
+- No raw privileged shell command strings and no silent semantic grep fallback.
+- No runtime auto-download of language servers.
+- External LSP executables must be configured and SHA-256 pinned before launch.
+- Full audit evidence remains durable; compact terminal UX remains the default.
+- New code is TDD: RED must be observed before the minimal production change.
+- R35 Final requires Windows gate PASS, immutable activation, public parity, rollback to exact sealed R34, exact R35 restore, all privileged planes DISARMED, signed closure, and independent verifier PASS.
+
+---
+
+### Task 1: Materialize the R35 successor and canonical tool-surface ledger
+
+**Files:**
+- Create successor tree: `D:\Projects\deadbyte-mcp-r35-release-v0.9.0` from exact R34 package bytes, preserving the already-written R35 spec/plan documents after the copy.
+- Create: `src/tool-surface-ledger.mjs`
+- Create: `scripts/tool-surface-snapshot.mjs`
+- Create: `test/tool-surface-ledger.test.mjs`
+- Modify: `src/mcp-server.mjs`
+- Modify: `src/version.mjs`
+- Modify: `package.json`
+- Modify: `CONTEXT.md`
+
+**Interfaces:**
+- Produces `canonicalSchemaHash(schema): string`.
+- Produces `snapshotToolSurface(toolDescriptors): {schema:'deadbyte.tool-surface.v1', tools:Array<{name,input_sha256,output_sha256}>, surface_sha256:string}`.
+- Produces `diffToolSurface(baseline,current): {unchanged:string[],added:string[],removed:string[],schema_changed:string[]}`.
+- Snapshot generation consumes the MCP protocol `tools/list` result produced by the real registered server surface. Release tooling never reconstructs names or schemas independently; this avoids a second descriptor registry and measures exactly what clients observe.
+
+- [ ] **Step 1: Verify parent bytes before copying**
+
+Create an external PowerShell verifier under `D:\Projects` that reads the R34 final result and verifies the immutable slot `C:\Users\DEADBYTE\.deadbyte-mcp\releases\v0.8.11-6c01e5285e065220` against manifest `6c01e528...`, including exactly 1560 listed files and every listed digest. Explicitly reject `D:\Projects\deadbyte-mcp-r34-release-v0.8.11` as parent authority because it is a mutable working tree. Run the verifier and require `R34_PARENT_EXACT_PASS`.
+
+- [ ] **Step 2: Materialize successor without mutating parent**
+
+Copy R34 package files into the R35 path with `Copy-Item`/`robocopy` read-only source semantics, exclude R34 mutable runtime outputs, then restore the R35 spec/plan documents. Verify a pre-change comparison showing all inherited package bytes equal to R34 except the R35 planning documents.
+
+- [ ] **Step 3: Write the failing ledger tests**
+
+`test/tool-surface-ledger.test.mjs` must assert:
+
+```js
+const baseline=snapshotToolSurface([
+  {name:'alpha',inputSchema:{type:'object',properties:{x:{type:'string'}}},outputSchema:{type:'object',properties:{ok:{type:'boolean'}}}},
+  {name:'beta',inputSchema:{type:'object'},outputSchema:{type:'object'}}
+]);
+const reordered=snapshotToolSurface([...same descriptors in reverse property/order form...]);
+assert.equal(baseline.surface_sha256,reordered.surface_sha256);
+assert.throws(()=>snapshotToolSurface([{name:'alpha',...},{name:'alpha',...}]),/duplicate tool/i);
+const diff=diffToolSurface(baseline,changed);
+assert.deepEqual(diff.removed,['beta']);
+assert.deepEqual(diff.schema_changed,['alpha']);
+```
+
+Also add an integration assertion that the R35 registered surface initially contains exactly the R34 73 names.
+
+- [ ] **Step 4: Run RED**
+
+Run `node --test test/tool-surface-ledger.test.mjs`. Expected failure: module/export missing.
+
+- [ ] **Step 5: Implement deterministic ledger**
+
+Canonicalize objects recursively with lexicographically sorted keys, preserve array order, reject non-JSON schema values, hash UTF-8 canonical JSON with SHA-256, sort tool records by name, reject duplicate names, then hash the whole snapshot.
+
+- [ ] **Step 6: Make server descriptors canonical**
+
+Refactor only the registration descriptor construction needed to expose the exact descriptors to the ledger. Do not rename public tools. Add `scripts/tool-surface-snapshot.mjs` that writes/prints the snapshot without including secrets or remote endpoint values.
+
+- [ ] **Step 7: Cut R35 identity**
+
+Set `DEADBYTE_VERSION='0.9.0'`; update package version/banner/current docs from R34 successor identity to R35 while retaining the exact R34 parent manifest in `CONTEXT.md`.
+
+- [ ] **Step 8: Run GREEN and R34 compatibility gate**
+
+Run focused tests, then compare generated R35 surface with an R34 baseline snapshot generated from sealed R34. Expected: 73 names, `removed=[]`, `schema_changed=[]`; additions require explicit plan amendment.
+
+---
+
+### Task 2: Introduce engine-neutral semantic runtime and pinned LSP transport
+
+**Files:**
+- Create: `src/semantic-backend-registry.mjs`
+- Create: `src/lsp-stdio-client.mjs`
+- Create: `src/semantic-lsp-backend.mjs`
+- Create: `test/lsp-stdio-client.test.mjs`
+- Create: `test/semantic-backend-registry.test.mjs`
+- Modify: `src/semantic-code.mjs`
+- Modify: `src/coding-output-contracts.mjs` only if an existing semantic output contract needs an `engine` string broadened without shape drift.
+
+**Interfaces:**
+- `createSemanticBackendRegistry({backends})` -> `{backendForPath(relativePath), languages()}`.
+- A backend implements `{id, extensions:Set<string>, outline(ctx), definition(ctx), references(ctx), replaceSymbol(ctx), diagnostics(ctx)}`.
+- `createLspStdioClient({executablePath,expectedSha256,args,cwd,env,requestTimeoutMs,maxMessageBytes})` -> `{initialize,notify,request,shutdown}`.
+- LSP transport accepts no shell string and verifies executable digest before spawn.
+
+- [ ] **Step 1: Write failing framed-protocol tests**
+
+Use a test fixture child process that speaks LSP `Content-Length` framing. Assert split headers/bodies, multiple frames in one chunk, request-id correlation, timeout, oversized message rejection, unexpected process exit, and graceful shutdown.
+
+- [ ] **Step 2: Run RED**
+
+Run `node --test test/lsp-stdio-client.test.mjs test/semantic-backend-registry.test.mjs`. Expected missing modules.
+
+- [ ] **Step 3: Implement bounded stdio JSON-RPC transport**
+
+Use `spawn(executablePath,args,{shell:false,...})`; verify SHA before spawn; parse byte buffers, not line-delimited JSON; cap each message; maintain a request map; reject all outstanding requests on exit; do not retry automatically.
+
+- [ ] **Step 4: Implement backend registry**
+
+Resolve by normalized lower-case extension. Unknown extension throws `unsupported semantic language: <ext>`; no grep fallback.
+
+- [ ] **Step 5: Adapt existing TypeScript backend**
+
+Move current TypeScript service behavior behind a backend object while preserving current output fields and symbol-id stale-precondition semantics. Run existing `test/semantic-code.test.mjs` unchanged first; contract drift is a failure.
+
+- [ ] **Step 6: Run GREEN**
+
+Focused LSP/registry + existing semantic tests must pass.
+
+---
+
+### Task 3: Required semantic tracer corpus plus optional Go evidence
+
+**Files:**
+- Create: `src/semantic-lsp-config.mjs`
+- Create fixtures under: `test/fixtures/semantic/{typescript,python,rust,cpp,go}`
+- Create: `test/semantic-multilang.test.mjs`
+- Create: `scripts/semantic-preflight.ps1`
+- Modify: `src/semantic-code.mjs`
+- Modify: `controller/deadbyte-controller.config.psd1.example`
+- Modify: `docs/ARCHITECTURE.md`
+- Modify: `README.txt`
+
+**Interfaces:**
+- Config records per language: `{id,extensions,executable_path,executable_sha256,args,initialization_options}`.
+- Preflight reports each tracer language as `builtin`, `pinned`, or `missing`; missing required language blocks R35 Final.
+
+- [ ] **Step 1: Inventory local language-server executables**
+
+Use a read-only preflight script to locate configured/known executables and hash them. Do not install anything automatically. Record exact results outside the package tree.
+
+- [ ] **Step 2: Write failing acceptance tests for TypeScript/JavaScript, Python, Rust, and C/C++; keep Go optional**
+
+Each fixture defines a symbol in one file and references it from another. Tests call the same public semantic runtime methods and assert normalized definition/reference paths and symbol-bound replacement. TypeScript uses the builtin backend; the four external languages use configured pinned LSP backends.
+
+- [ ] **Step 3: Run RED**
+
+Expected failures identify missing backend config or unsupported language. If a required executable is absent, stop implementation of that adapter and resolve the dependency explicitly rather than fake semantics.
+
+- [ ] **Step 4: Implement LSP normalization**
+
+Map `DocumentSymbol`/`SymbolInformation`, locations, references, diagnostics, and `WorkspaceEdit`/text spans into DEADBYTE normalized locations. Reject edits outside allowed root and reject multi-file mutation from `replaceSymbol` unless the public contract explicitly supports it.
+
+- [ ] **Step 5: Implement stale preconditions**
+
+Symbol ids bind backend id, path, file SHA, symbol range/name. `replaceSymbol` verifies expected file SHA immediately before mutation and uses existing coding-plane receipt-backed text replacement.
+
+- [ ] **Step 6: Run GREEN corpus**
+
+The four required tracer languages (TypeScript/JavaScript, Python, Rust, C/C++) must pass outline/definition/references/replace tracer cases. Go is optional evidence only. Diagnostics must be non-empty for one deliberate syntax/type error fixture per required language where the server supports diagnostics.
+
+---
+
+### Task 4: Provenance-aware context and Looped Engineering V2
+
+**Files:**
+- Create: `src/context-provenance.mjs`
+- Create: `test/context-provenance.test.mjs`
+- Modify: `src/autonomous-context-engine.mjs`
+- Modify: `src/autonomous-loop-state.mjs`
+- Modify: `src/autonomous-runtime.mjs`
+- Modify: `src/autonomous-progress.mjs`
+- Modify: `src/autonomous-planner-contracts.mjs`
+- Modify: `test/autonomous-context-engine.test.mjs`
+- Modify: `test/autonomous-loop.test.mjs`
+- Modify: `test/autonomous-r34-e2e.test.mjs` or create R35 successor e2e test while retaining R34 regression coverage.
+
+**Interfaces:**
+- `provenanceFor({sourceKind,sourceId,bytes,observedAtUtc,trustClass,relevanceEpoch})` returns canonical provenance with byte count and safe source digest.
+- `dedupeEvidence(items)` keeps newest evidence for identical stable provenance identity.
+- Loop phases become exact enum: `PLAN|INSPECT|ACT|VERIFY|CRITIC|REPAIR|RELEASE|WAITING|SUCCEEDED|FAILED`.
+
+- [ ] **Step 1: Write provenance RED tests**
+
+Assert deterministic identity, payload elision preserving provenance, secret fields not becoming digest oracles, dedupe stability, and context byte budget.
+
+- [ ] **Step 2: Write loop RED tests**
+
+Assert failed VERIFY enters CRITIC, critic classification enters REPAIR, REPAIR must change plan/hypothesis fingerprint before another ACT, and journal replay produces byte-equivalent todo projection.
+
+- [ ] **Step 3: Run RED**
+
+Focused context/loop tests must fail on missing provenance and phases.
+
+- [ ] **Step 4: Implement provenance envelope**
+
+Attach provenance before compaction; compact payload independently; preserve provenance fields. Keep R34 secret sanitizer semantics.
+
+- [ ] **Step 5: Implement explicit critic/repair transitions**
+
+Do not add another agent. Extend durable state reducer and runtime with machine transitions. Planner remains the single decision source; critic payload is structured failure classification derived from evidence plus a planner repair decision.
+
+- [ ] **Step 6: Implement restart-safe todo projection**
+
+Derive todo items from active plan + journal state; never trust a standalone mutable todo file.
+
+- [ ] **Step 7: GREEN and crash replay**
+
+Run context/loop/e2e/adversarial tests including deliberate failed action, crash after mutation, restart, critic repair, and single final side effect.
+
+---
+
+### Task 5: Signed provider circuit state
+
+**Files:**
+- Create: `src/provider-circuit.mjs`
+- Create: `test/provider-circuit.test.mjs`
+- Modify: `src/autonomous-local-supervisor.mjs`
+- Modify: `src/autonomous-supervisor-host.mjs`
+- Modify: provider/client-sampling adapter seam used by R35 planner invocation.
+
+**Interfaces:**
+- `deriveCircuit(events,providerId,policy)` -> `{state,failure_count,opened_at_utc,next_probe_at_utc,last_success_at_utc}`.
+- `classifyProviderFailure(error)` returns bounded class: `timeout|transport|rate_limit|auth|invalid_response|other`.
+- Circuit policy contains `failure_threshold`, `open_ms`, `half_open_probe_limit`.
+
+- [ ] **Step 1: Write RED state-machine tests**
+
+Test CLOSED -> OPEN at threshold, OPEN rejects calls without invoking provider, HALF_OPEN after time boundary, one bounded probe, success -> CLOSED, failure -> OPEN, authenticated replay equivalence, and corrupted evidence -> unavailable.
+
+- [ ] **Step 2: Run RED**
+
+Missing module/state is expected.
+
+- [ ] **Step 3: Implement pure reducer first**
+
+No timers inside reducer; caller supplies observed UTC timestamp. State derives from signed journal events.
+
+- [ ] **Step 4: Integrate only around provider/network calls**
+
+Local coding/process/file operations bypass circuit state. Circuit open emits typed WAITING reason and never arms authority or selects an unconfigured fallback provider.
+
+- [ ] **Step 5: GREEN supervisor/restart tests**
+
+Crash/restart must preserve OPEN/HALF_OPEN state from signed evidence.
+
+---
+
+### Task 6: Resumable remote result cursor and deduplication
+
+**Files:**
+- Create: `src/remote-result-journal.mjs`
+- Create: `test/remote-result-journal.test.mjs`
+- Modify: `src/deadbyte-cli.mjs`
+- Modify: `src/remote-session-guardian.mjs`
+- Modify: Bridge sidecar only if server-to-client replay requires a transport hook; old R34 Bridge bytes remain untouched.
+- Modify: `test/remote-session-cli.test.mjs`
+
+**Interfaces:**
+- Journal event `{session_id,seq,request_id,tool,status,observed_at_utc,payload_sha256}` with bounded compact display metadata.
+- `replayAfter(cursor,{limit})` returns ordered unique results or typed `cursor_outside_retention`.
+- Result identity is `request_id + terminal result identity`; duplicate terminal results are suppressed.
+
+- [ ] **Step 1: Write RED journal tests**
+
+Assert monotonic sequence, restart adoption, duplicate suppression, bounded retention, cursor replay, stale cursor error, and no raw payload storage in compact journal.
+
+- [ ] **Step 2: Run RED**
+
+Expected module missing.
+
+- [ ] **Step 3: Implement journal and durable session identity**
+
+Use atomic append/replace patterns already proven by audit/session code. Session reconnect never changes authority state.
+
+- [ ] **Step 4: Add CLI reconnect/resume UX**
+
+Default remains compact. A resumed session prints only concise status and results after cursor; `--verbose` may show full audit/Bridge stream. Do not print SHA noise in compact mode.
+
+- [ ] **Step 5: GREEN reconnect tests**
+
+Simulate Bridge restart between result events and prove no duplicate terminal display or side effect.
+
+- [ ] **Step 6: Stress remote-session tests**
+
+Run `test/remote-session-cli.test.mjs` repeatedly. Every wait predicate must correspond to the exact later assertion; no proxy-condition race is accepted.
+
+---
+
+### Task 7: Reproducible capability benchmark corpus
+
+**Files:**
+- Create: `bench/fixtures/*`
+- Create: `src/capability-benchmark.mjs`
+- Create: `scripts/capability-benchmark.mjs`
+- Create: `test/capability-benchmark.test.mjs`
+- Create: `docs/R35-BENCHMARK-METHOD.md`
+- Modify: `README.txt` comparison section to reference measured evidence only.
+
+**Interfaces:**
+- Benchmark result schema `deadbyte.r35-capability-benchmark.v1`.
+- Each case records `{case_id,correct,round_trips,bytes_returned,elapsed_ms,stale_rejected,recovery_success,duplicate_side_effects}`.
+- Aggregate never collapses correctness into a speed score; dimensions remain separately inspectable.
+
+- [ ] **Step 1: Write RED schema/determinism tests**
+
+Assert fixture identity/hash, required metrics, deterministic correctness verdicts, and explicit failure when a case lacks evidence.
+
+- [ ] **Step 2: Implement offline cases**
+
+Cover filesystem bounded read, exact edit/stale reject, paged search, owned process lifecycle, the four required semantic tracer languages with optional Go evidence, context compaction/provenance, autonomous fail/repair, and remote replay dedupe.
+
+- [ ] **Step 3: Run benchmark twice**
+
+Correctness verdicts and fixture identities must match. Timing may vary and is reported, not used to manufacture a winner.
+
+- [ ] **Step 4: Document competitor methodology**
+
+Describe how equivalent public operations may be measured against external products without copying their code. Claims in README must cite local benchmark evidence and retain the semantic-breadth caveat where applicable.
+
+---
+
+### Task 8: R35 release authorities, Windows gate, rollback/restore, and Final closure
+
+**Files:**
+- Modify: `scripts/gate-windows.ps1`
+- Modify: `scripts/windows-gate-evidence.mjs`
+- Modify: `scripts/release-parity.mjs`
+- Create or update R35 parity client derived from canonical tool-surface authority.
+- Modify: `scripts/final-closure.mjs`
+- Modify: `scripts/final-closure-verify.mjs`
+- Modify: `src/runtime-update.mjs` only if R35 generation data needs semantic-server/circuit/replay configuration pins.
+- Create: `R35-BASELINE.json` only at proven candidate freeze.
+- Tests: release/version/parity/runtime-update/final-verifier tests plus all existing source tests.
+
+**Interfaces:**
+- Windows evidence schema becomes `deadbyte.r35-windows-gate.v1` and binds exact manifest, tool-surface snapshot, semantic preflight identities, benchmark evidence hash, source-test count, and gate marker.
+- Final closure binds R35 activation receipt, R34 rollback receipt, R35 restore receipt, public parity, tool-surface evidence, benchmark evidence, Windows evidence, all-DISARMED proof, and supervisor-running proof.
+
+- [ ] **Step 1: Write RED release-authority tests**
+
+Tests must reject R34 schema/marker identities, stale 73-tool arrays copied by hand, missing surface snapshot, missing benchmark hash, missing semantic preflight pins, and closure that did not roll back to exact `v0.8.11-6c01e5285e065220`.
+
+- [ ] **Step 2: Currentize release authorities**
+
+Derive expected public surface from `tool-surface-ledger.mjs`. Keep R34 rollback authority constant and explicit. Do not regenerate R34 identity from current source.
+
+- [ ] **Step 3: Full source regression**
+
+Run `npm test` and all focused stress suites. Zero fail is mandatory; preserve raw failure output externally for any flake.
+
+- [ ] **Step 4: Canonical Windows gate**
+
+Build native artifacts, refresh process policy pins, run public/machine/coding/process/observation/autonomous/semantic/benchmark smokes, freeze one exact manifest, verify exact tree, and emit manifest-bound R35 evidence outside the candidate tree.
+
+- [ ] **Step 5: Independent frozen verification**
+
+A separate verifier checks manifest entries/digests, version, surface snapshot, Windows evidence, benchmark evidence, semantic executable pins, and absence of unlisted package files.
+
+- [ ] **Step 6: Immutable R35 activation**
+
+Create a content-addressed R35 release/generation. Rebind only exact pinned Bridge/LSP/process/supervisor identities. Verify runtime ONLINE and supervisor `RUNNING [verified]`.
+
+- [ ] **Step 7: Public R35 parity**
+
+Verify exact R35 public tool surface and functional planes. No host authority is required for ordinary parity. Record signed parity evidence.
+
+- [ ] **Step 8: Roll back to sealed R34**
+
+Use signed activation/rollback receipts. Verify exact R34 release id, manifest, public surface, runtime, and disarmed authority state.
+
+- [ ] **Step 9: Restore exact R35**
+
+Restore the exact R35 manifest/generation with pinned Bridge/LSP identities; verify parity again and supervisor running.
+
+- [ ] **Step 10: Seal all authority**
+
+Revoke process, coding, autonomous, and host standing grants. Verify all four are DISARMED while runtime remains ONLINE.
+
+- [ ] **Step 11: Signed R35 final closure**
+
+Emit signed closure/summary outside the release tree, then run the independent verifier. Only an independent `status=passed`, exact R35 manifest, exact R34 rollback anchor, `all_disarmed=true`, and `supervisor_running=true` permit the label `R35 PRODUCTION FINAL BASELINE`.
+
+---
+
+## Self-review result
+
+- Spec coverage: Tasks 1-8 cover every R35-A through R35-H requirement.
+- Placeholder scan: no TBD/TODO/coming-soon implementation steps are permitted by this plan.
+- Type consistency: semantic registry/LSP interfaces are introduced in Task 2 and consumed by Task 3; provenance/loop interfaces are introduced together in Task 4; provider circuit is isolated in Task 5; remote journal is isolated in Task 6; benchmark schema is introduced in Task 7 and consumed by release evidence in Task 8.
+- Scope discipline: no dynamic plugin framework, provider swarm, vector database, auto-download, or host auto-arm is included.