seer-mcp 0.1.0 → 0.1.2

package/STRESS_TEST_REPORT.md

@@ -0,0 +1,219 @@
+# Seer-Core — Pre-Release Stress Test Report
+
+**Date:** 2026-05-30
+**Scope:** Release-hardening pass. Simulate realistic (and weird) repositories that
+exercise every Seer-Core feature, drive the MCP tools, find/fix real bugs, and
+re-validate the full suite. No new features were added.
+
+All scratch repos and harness scripts live under `.stress/` (gitignored). The
+reusable MCP stdio driver is `.stress/mcpClient.ts`.
+
+---
+
+## 1. Executive Summary
+
+- **Baseline:** full `npm test` suite green before changes.
+- **2 real bugs found and fixed**, each with a new regression test in
+  `tests/bug-regressions.ts` (bug 9 and bug 10).
+- **8 end-to-end stress scenarios** built from realistic repos, all green
+  (128 assertions), plus adversarial parser-robustness and a 245-file scale index.
+- **4 limitations/clarifications** documented (by-design or out-of-scope, not bugs).
+- Full `npm test` re-run green after the fixes.
+
+---
+
+## 2. Bugs Found and Fixed
+
+### Bug 9 — FastAPI route decorators mis-detected as outbound service calls
+- **Symptom:** A FastAPI service indexed its own route decorators
+  (`@app.post("/api/payments/charge")`, `@app.get("/api/payments/{payment_id}")`)
+  as phantom outbound `http-client` `service_calls`. The service-link resolver
+  then **self-linked a route to its own registration site** (e.g. the
+  `/api/payments/{payment_id}` route linked to a phantom client call of the same
+  path). An agent reading `seer_service_calls` would see a service "calling"
+  its own endpoints — misleading data.
+- **Root cause:** The walker's route/service-call de-duplication is *node-level*
+  (`src/parser/walker.ts`): it suppresses a service call only when the **same**
+  AST node also produced a route. For Express (`app.get("/x", h)`) the route and
+  the call are the same `call_expression`, so it works. For FastAPI the route is
+  emitted on the parent `decorator` node while `tryExtractServiceCalls` fires on
+  the inner `call` node — different nodes — so the guard never fired. (Flask
+  `@app.route` slipped past because `route` isn't an HTTP verb.)
+- **How it was found:** `.stress/micro.ts` — a 3-service repo (TS/Express gateway
+  → Python/FastAPI payments → Python/Flask notifications). `seer_service_calls`
+  returned 5 calls, 2 of which were FastAPI decorators; one route self-linked.
+- **Fix:** `src/parser/languages/python.ts` `tryExtractServiceCalls` returns
+  `null` when `node.parent?.type === 'decorator'` — a decorator call is a
+  registration, never an outbound client.
+- **Regression:** `tests/bug-regressions.ts` → `bug9_fastapiDecoratorNotServiceCall`.
+
+### Bug 10 — TS template-literal path params dropped, breaking route matching
+- **Symptom:** `axios.get(`http://svc/api/users/${id}`)` normalized to
+  `/api/users` — the `${id}` path segment vanished. The call then could not
+  match a `/api/users/:id` route (segment-count mismatch), so a real
+  cross-service link **silently failed to resolve**.
+- **Root cause:** `readTemplateString` in `src/parser/languages/typescript.ts`
+  dropped *every* `${...}` substitution. For a path parameter this collapsed the
+  segment count that `routePatternsMatch` relies on.
+- **How it was found:** `.stress/micro.ts` (gateway `getUser()` →
+  `/api/users/<id>`) and again in `.stress/external.ts` (shop `getInvoice()` →
+  billing `/api/invoices/<invoice_id>`).
+- **Fix:** non-env substitutions now emit a single `:param` placeholder segment
+  so the segment count is preserved; env-base substitutions
+  (`${process.env.X}/charge`) still drop (preserving `env_base` matching and the
+  recovered `envKey`).
+- **Regression:** `tests/bug-regressions.ts` → `bug10_tsTemplatePathParam`
+  (asserts both the `:param` preservation and the unchanged env-base path).
+
+Both fixes are conservative and node-local; the Track G suite (312 assertions)
+and SeerBench (precision/recall ≥ 0.9) remained green after the change.
+
+---
+
+## 2b. Enhancements Implemented (follow-up pass)
+
+Two of the documented limitations were promoted to fixes after review, each with
+a regression in `tests/bug-regressions.ts` (bug 11, bug 12). Full suite + all
+stress scenarios green afterward.
+
+### Rec 2 — C++ out-of-line method definitions now reconstruct the class scope
+- `T Vec<T>::dot(...)` defined at namespace scope used to qualify as `geo.dot`
+  (reads like a free function). It now folds the owner scope from the
+  `qualified_identifier` declarator → `geo.Vec.dot`. Also fixes a latent
+  collision: `Foo::bar` and `Baz::bar` at the same scope stayed distinct
+  (`Foo.bar`/`Baz.bar`) instead of collapsing to `bar`/`bar#1`.
+- Mechanism: new optional `SymbolDef.scopePath` (the one sanctioned scope hint;
+  extractors still return short names). The walker folds it into the qualified
+  name and keys overload disambiguation on `(scope + name)`.
+- Files: `src/types.ts`, `src/parser/walker.ts`, `src/parser/languages/cpp.ts`.
+- Verified on the Unreal fixture too: `APickupItem::OnPickedUp` → `APickupItem.OnPickedUp`.
+
+### Rec 3 — `seer_definition` `file` filter accepts a path-suffix on a boundary
+- The `file` disambiguator was exact-match only; a basename (`svc.ts`) couldn't
+  match rel_path `src/svc.ts` and silently returned nothing — a wasted
+  round-trip. It now matches an absolute path, the exact rel_path, OR a trailing
+  fragment aligned to a `/` boundary (so `auth.ts` never matches `oauth.ts`),
+  with LIKE metacharacters escaped (a `_` in a filename can't act as a wildcard).
+  Improves `seer_definition`/`seer_context`/`seer_risk`/`seer_continuity` at once.
+- Files: `src/db/store.ts` (`getDefinition`, `escapeLike`), `src/mcp/server.ts`
+  (tool description).
+
+## 2c. AI-Agent Optimization Spec — implemented (separate pass)
+
+A six-section "AI-agent optimization" proposal was critiqued and the surviving
+items implemented (§6 SCIP auto-download was rejected outright — it violates the
+local-first/zero-native-dep/deterministic contract and was NOT built). All are
+deterministic, local, zero-AI. Covered by `tests/optspec.ts` (35 assertions,
+wired into `test:mcp`).
+
+- **§3 `seer_skeleton`** — deterministic source elision. Renders a file with
+  every symbol signature kept and bodies collapsed to `{ … N lines … }` fold
+  markers (exact line counts). Nesting is reconstructed from line-range
+  containment, so it is language-agnostic. `focusSymbol` expands one body
+  verbatim. Byte-identical on re-render. New: `src/indexer/skeleton.ts`.
+- **§2 Did-you-mean** — when `seer_definition` / `seer_symbols` / `seer_callers`
+  / `seer_risk` / `seer_context` / `seer_behavior` / `seer_symbol_module` /
+  `seer_continuity` resolve to nothing, the response carries a `didYouMean`
+  array of up to 5 BM25/FTS suggestions. **Suggestion-only** — never
+  substituted, never used to compute risk on a guessed symbol.
+- **§1 Token budget** — optional `tokenBudget` on the high-volume list tools
+  (`seer_symbols`, `seer_definition`, `seer_callers`, `seer_callees`,
+  `seer_complexity`, `seer_service_calls`, `seer_service_links`). Deterministic
+  prefix-trim on serialized char length (~4 chars/token), keeps the
+  highest-ranked rows, flags `truncated`/`omitted`. Absent budget ⇒ byte-identical
+  to prior output. The high-fan-in caller path is **not** re-sorted.
+- **§5a Lazy lifecycle** — derived passes (modules / shape-hash / symbol-history)
+  now auto-build on first dependent query (extends the JIT-freshness philosophy).
+  The three `*_build` tools are rebranded "(Advanced — usually unnecessary)" but
+  kept registered for forced rebuilds.
+- **§5b `seer_trace` umbrella** — one scope-dispatch entry
+  (`callers|callees|path|file|module|service|service_path|module_service`) over
+  the existing `seer_trace_*` family (which remain for direct/power use).
+  Delegate throws are caught and returned as clean advisory errors.
+- **§4 `seer_batch`** — run up to 25 read-only tools in one call; sequential,
+  one failure never aborts the rest, cannot nest. Saves agent turns on known
+  fan-out.
+- Files: `src/mcp/server.ts` (handler registry + wrapper, helpers, new tools),
+  `src/indexer/skeleton.ts`, `tests/optspec.ts`, `package.json` (test wiring).
+
+## 3. Limitations / Clarifications (not bugs)
+
+1. **Server-side HTTP route extraction is framework-limited.** Routes are
+   extracted only for Java(Spring), Python(FastAPI/Flask), TS(Express/Fastify/
+   tRPC/GraphQL). **Go `net/http`/gin/chi, Rust axum/actix, C# ASP.NET, C++**
+   have no HTTP route extraction, so such a backend cannot be a cross-service
+   link *target* over HTTP (it can still be a client; Go/Java/C# are gRPC
+   clients and gRPC routes come from `.proto`). This matches the master guide's
+   documented route list but is worth surfacing for polyglot microservice maps.
+
+2. ~~C++ out-of-line method qualified names drop the class scope.~~ **FIXED —
+   see §2b Rec 2.**
+
+3. ~~`seer_definition` `file` filter is exact-match.~~ **FIXED — see §2b Rec 3.**
+
+4. **Continuity is a snapshot shape-matcher, not a historical-shape tracer.** It
+   links a rename/move only when both identities coexist in the working tree
+   (e.g. a kept alias) — a true cross-commit rename where the old symbol is gone
+   yields **no** continuity candidate (correctly; it never invents one).
+   Cross-commit lineage is delivered instead by `seer_history` (git `--follow` +
+   line-overlap), which was verified to trace across a file move **and** a rename
+   back to the pre-rename commit. The headline "trace refactoring across
+   renames/moves" is thus fulfilled by history; continuity is advisory snapshot
+   evidence. Honest behavior, no false positives.
+
+### Minor test-quality notes (correct but low-power assertions)
+`tests/tracke.ts:248`, `tests/trackf.ts:164`, `tests/trackg.ts:1332`,
+`tests/tracki-boundaries.ts:174`, `tests/tracki-external-bundles.ts:58`, and
+`tests/worker-pool.ts:224` assert `>= 0` on counts or `assert(true)` (i.e.
+"it ran / is a valid number"). Not wrong, but they cannot fail; could be
+strengthened to assert concrete values.
+
+---
+
+## 4. Scenarios Validated (all green)
+
+| # | Scenario | Harness | Result |
+|---|----------|---------|--------|
+| 1 | Polyglot weird syntax — 9 languages + edge cases (overloads, generics, decorators, unicode idents, metaclass, records, out-of-line methods, C body-gate, JSX/TSX, broken syntax, BOM/CRLF, empty, comments-only) | `.stress/polyglot.ts` | 35/35 |
+| 2 | Microservices cross-service links (HTTP literal + param across TS/FastAPI/Flask) | `.stress/micro.ts` | 14/14 |
+| 3 | Rename/move continuity over **real git history** (history traces across move+rename; continuity is honest) | `.stress/continuity.ts` | 7/7 |
+| 4 | Monorepo boundaries + cross-boundary crossing flags in risk | `.stress/boundaries.ts` | 11/11 |
+| 5 | Watcher + JIT freshness under 15-edit burst (instant, max 25ms, no stalls/leaks/dups, prune on delete) | `.stress/watcher.ts` | 24/24 |
+| 6 | SCIP compiler-precision overlay merges with AST (no duplicate clutter, idempotent, provenance) | `.stress/scip.ts` | 10/10 |
+| 7 | External bundle layers across **two repos** + idempotent/forced re-import + contract diff | `.stress/external.ts` | 12/12 |
+| 8 | Scale modify/reindex/git cycle on helix-core (45 Rust files: cache reuse, prune, move, delete, add, churn) | `.stress/scale-modify.ts` | 15/15 |
+
+**Plus:** adversarial parser robustness (300KB single line, 150-level nesting,
+random-byte `.ts`) — no crash, WASM runtime survived, sibling files still
+indexed; and a full helix-master scale index (245 files / 5,207 symbols /
+24,057 edges in ~14s) matching the documented scale matrix.
+
+---
+
+## 5. How to Reproduce
+
+```bash
+npm run build
+# individual scenarios:
+npx tsx .stress/polyglot.ts
+npx tsx .stress/micro.ts
+npx tsx .stress/continuity.ts
+npx tsx .stress/boundaries.ts
+npx tsx .stress/watcher.ts
+npx tsx .stress/scip.ts
+npx tsx .stress/external.ts
+npx tsx .stress/scale-modify.ts
+# regressions for the two fixes:
+npx tsx tests/bug-regressions.ts
+# full suite:
+npm test
+```
+
+---
+
+## 6. Files Changed
+
+- `src/parser/languages/python.ts` — bug 9 fix (decorator guard).
+- `src/parser/languages/typescript.ts` — bug 10 fix (template path-param placeholder).
+- `tests/bug-regressions.ts` — new regressions bug 9 + bug 10 (+ header).
+- `.gitignore` — ignore `.stress/` scratch.