@shipispec/tsfix 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -4,6 +4,65 @@ All notable changes to `@shipispec/tsfix` are documented here. Format follows [K
 
 ## [Unreleased]
 
+### Added (Layer 4 — stub-and-continue escape hatch)
+- **`stubAndContinue(opts)`** — new public API. Inserts `// @ts-expect-error - tsfix: <codes> — <message>` immediately above each unresolved error site so `tsc --noEmit` exits 0. Closes the "tsfix never leaves the workspace worse than it found it" property. Uses `@ts-expect-error` (not `@ts-ignore`) so directives self-destruct once the underlying issue is fixed by other means.
+- **`runMendLoop` opt-in flag** — new `stubOnFailure?: boolean` option (default `false`). When the LLM loop terminates with leftover errors and the flag is set, Layer 4 runs automatically. New `"stubbed"` stop reason; new `stubs?: AppliedStub[]` result field with what was applied.
+- **Idempotency** — re-running `stubAndContinue` on an already-stubbed workspace is a no-op. Detects existing `@ts-expect-error` / `@ts-ignore` directives on the line above and skips.
+- **Safe skips** — `node_modules/`, `.d.ts` files, missing files, and lines beyond file length are recorded as `skipped` (with reason) rather than crashing.
+- **Multi-error coalescing** — multiple diagnostics on the same line collapse into one stub comment listing all TS codes and joined messages.
+- **Indent + CRLF preservation** — comment matches the indentation of the line it's stubbing; CRLF line endings on Windows-authored files survive the rewrite.
+- **`dryRun`** support — same semantics as Layer 2: reports `stubsApplied` without writing.
+
+### Added (fixture engine — Day 2/3 mutators)
+- **5 new ts-morph mutators** covering codes the original 3-mutator set didn't reach:
+  - `ts2322-incompatible-return.mjs` — replaces a return expression with a wrong-typed primitive literal in a function with a primitive return type
+  - `ts2304-cannot-find-name.mjs` — renames a value-position identifier (variable, call, parameter usage) to a no-near-match string; Layer 0's auto-import abstains because there's no candidate
+  - `ts2345-arg-type-mismatch.mjs` — replaces a function-call argument with a wrong-typed primitive when the parameter's declared type is `string` / `number` / `boolean`
+  - `ts2554-arg-count-mismatch.mjs` — drops the trailing argument from a call that currently satisfies its signature
+  - `ts2365-operator-mismatch.mjs` — replaces one operand of a numeric binary expression (`<`, `>`, `<=`, `>=`, `-`, `*`, `/`, `%`) with a string literal
+- **50 new generated fixtures** (10 per new code × 8 codes total). Total Layer-2 fixture corpus: **85** (was 35) — 3 minimal + 2 realistic + 80 generated across 8 codes. Total fixture count across all layers: **99** (14 Layer-0 + 85 Layer-2).
+
+### Added (tests)
+- **19 new Layer-4 unit tests** — 16 in `stubAndContinue.test.ts` + 3 in `runMendLoop.test.ts` covering single error, multi-error-same-line, multi-code, indent preservation, descending-order processing, idempotency, node_modules skip, .d.ts skip, missing-file skip, dry-run, message truncation, CRLF preservation, first-line edge case, no-eligible case, warning/suggestion filtering, and the runMendLoop integration (stopReason flip, default-off behavior, dryRun interaction).
+
+### Changed
+- **Public surface** at `src/index.ts` extended with `stubAndContinue`, `StubAndContinueOptions`, `StubAndContinueResult`, `AppliedStub`, `SkippedStub`. Layer 0/1/2 surface unchanged.
+- **`RunMendLoopOptions`** gains `stubOnFailure?: boolean`. **`RunMendLoopResult`** gains optional `stubs?: AppliedStub[]`. **`StopReason`** union gains `"stubbed"`. All additive — old callers unaffected.
+- **`scripts/generate-fixtures.mjs`** now runs via `tsx` and imports from `src/index.ts` directly instead of `dist/index.js`. Reason: the v0.4.0 dist bundle inlines `@vercel/oidc` (transitive of `ai`), which uses dynamic `require()` patterns that fail under esbuild's ESM output at module-init time. The generator only needs Layer 0/1 entry points, so importing from source bypasses the AI SDK entirely. Side benefit: no `npm run build` prerequisite — `npm run generate-fixtures` works from a fresh clone.
+- Removed `pregenerate-fixtures: npm run build` hook from `package.json`.
+
+### Fixed
+- **`stubAndContinue` resolves relative paths** against `workspaceRoot`. Diagnostics from `runInProcessTsc` use relative paths; consumers may pass absolute. Both work.
+
+### Added (Layer-2 benchmark — Day 4)
+- **Parallelism** — `npm run benchmark:llm` now runs fixtures concurrently via an inline `pLimit(N)` semaphore (no new dep). Default concurrency is 8; configurable via `--concurrency=N`. 100 fixtures at ~1.5s/each: sequential ~2 min → parallel ~20s. Per-fixture workspaces are isolated (snapshot/restore is local) so parallelism is safe; tsfix's program cache thrashes harmlessly between fixtures.
+- **File-based response cache** — every LLM call is keyed by `sha256(systemBlock + userBlock + model)` and stored under `.benchmark-cache/<hash>.json`. Re-runs against unchanged fixtures replay cached responses for free. Any change to the system prompt template, fixture content, or model invalidates automatically (it's all in the hash). `--no-cache` bypasses; `--clear-cache` wipes and exits. `.benchmark-cache/` added to `.gitignore`.
+- **Cache module** — extracted to `benchmark/cache.ts` so the logic is unit-testable independent of the full benchmark. 16 new unit tests covering: deterministic keying, key sensitivity per input, hex format, separator-confusion resistance, round-trip read/write, corrupted-entry handling, miss → store → hit cycle, parameter discrimination, bypass behavior, apiKey NOT in the cache key (rotating keys doesn't invalidate), error propagation without poisoning the cache.
+- **Failure reporting** — when fixtures fail, the per-iteration LLM raw response dump is collected and printed in a single block at the end of the run (instead of inline during the loop, which would interleave under concurrency).
+- **Layer-2 fixture filter (LLM benchmark)** — the LLM benchmark now filters fixtures by `expected.json` shape (`costUsdMax` or `expectedErrorCode`), mirroring the Layer-0 benchmark's filter. Prevents accidentally running Layer-0 fixtures through the LLM.
+- **`benchmark/run-llm-benchmark.ts` rewritten** around the parallel + cached worker model. Per-fixture output gets a `[n/m]` progress prefix and prints in completion order; final summary sorted by name for deterministic output. Total wall time, total cost (cache misses only — hits are free), and cache hit rate are reported at the end.
+
+### Added (CLI — Layer 2 exposure)
+- **`--llm` flag** — opt-in escalation to Layer 2 (single-file LLM mend) for errors that survive Layer 0/1. Off by default; CLI default path remains zero-network.
+- **`--llm-model <name>`** — Anthropic model (default `claude-haiku-4-5`). Known-priced models hardcoded for cost estimation: `claude-haiku-4-5`, `claude-sonnet-4-5`, `claude-opus-4-7`. Unknown models warn and report cost as 0.
+- **`--llm-max-iterations <N>`** — cap on LLM retries (default 3).
+- **`--llm-budget-usd <amount>`** — soft cost cap. If exceeded, exits with code 3 (Layer 2 still ran; partial work persisted to disk).
+- **Exit code 3** added — Layer 2 budget exceeded.
+- **Validation:** `--llm` without `ANTHROPIC_API_KEY` → exit 2 with helpful error. `--llm + --dry-run` is rejected as mutually exclusive (Layer 2 writes patches to disk).
+- **JSON report extension** — `layer2: { ran, stopReason, errorsBefore, errorsAfter, iterations, totalInputTokens, totalOutputTokens, totalCostUsd, budgetExceeded, model } | null`. Layer 0/1 surface unchanged.
+- **Human report extension** — new "Layer 2 (LLM)" line in the per-run summary when `--llm` was used. Shows error count delta, iteration count, tokens, cost, and stopReason.
+- **13 new CLI integration tests** in `cli/run-stack.test.ts` covering argument validation, exit codes, no-key errors, mutual-exclusion checks, JSON report shape, and the no-Layer-2-when-Layer-0-clean path. Tests spawn the actual `tsx cli/run-stack.ts` process — catches integration issues that pure unit tests of `parseArgs` would miss.
+
+### Fixed (latent v0.4.0 bundle bug)
+- **Externalized `ai` and `@ai-sdk/anthropic`** from the esbuild bundle in `scripts/build.mjs`. v0.4.0 inlined them, which (1) bloated `dist/index.js` from ~25 KB to 1.3 MB, and (2) crashed under plain `node` execution at module-init time because `@vercel/oidc` (transitive) uses dynamic `require()` patterns that fail in ESM bundles. Both packages are declared in `dependencies` so npm install pulls them in automatically for consumers using Layer 2. **Bundle sizes after fix:** `dist/index.js` 1.3 MB → 36 KB; `dist/cli.js` ~22 KB → 45 KB. Library import via plain `node` now works (verified with `node -e 'import("./dist/index.js")'`).
+
+### Deferred (fixture engine)
+- **TS2532** (Object is possibly undefined) — seeds don't currently contain optional chains or `Map.get()`-style calls that would produce TS2532 deterministically. Mutator deferred until seeds expand or a real-failure capture provides better candidates.
+- **TS2551-negative** (LSP returns multiple equally-close fix candidates → abstains) — engineering a deterministic TS2551 case where Layer 0's `fixesAreEquivalent` check abstains is contrived. Defer until we see a real-world example.
+
+### Also changed (CLI)
+- **CLI public surface** at `cli/run-stack.ts` extended with the Layer 2 flags listed above. The bin entry (`dist/cli.js`) now imports `runMendLoop`, `runInProcessTsc` and the contract types from `src/index.ts` in addition to the previous `runValidationLoop` + `discoverTsFiles`. Tree-shaking keeps Layer 2 out of the bundle's code path unless `--llm` is set.
+
 ## [0.4.0] - 2026-05-14
 
 **Layer 2 LLM mend is now in-package.** The previously-planned sister package `@shipispec/tsmend` has been folded into `tsfix` so the deterministic Layer 0/1 stack and the LLM-driven Layer 2 stack ship as one. This reverses the v0.3.0 sister-package decision (D3) — see roadmap update.