@mknrt/autotests-overkill 1.2.2 → 1.2.3

package/docs/audits/2026-04-20-iteration-2-comprehensive.md

@@ -1,563 +0,0 @@
-# Audit Iteration 2
-
-Date: 2026-04-20
-
-## Goal
-
-Continue the first audit with a broader question:
-
-> If I had to work with this MCP server every day as a real assistant for test design, test drafting, coverage review, and failure debugging, what would I improve next?
-
-This iteration focuses on:
-
-- product capability coverage
-- trustworthiness of recommendations
-- engineering execution quality
-- operability and packaging quality
-- what to add next for maximum daily usefulness
-
-## Audit Method
-
-This iteration combined:
-
-- direct repository review
-- targeted code reading of domain, indexer, connector, MCP, CLI, and docs layers
-- verified command execution
-- parallel subagent audits for:
-  - product capabilities
-  - engineering quality and reliability
-  - operability, packaging, and onboarding
-
-## Commands Run
-
-The following commands were run during this iteration:
-
-```bash
-npm run verify
-npm run mcp:smoke
-npm run acceptance:coverage
-npm run acceptance:blueprint
-npm run acceptance:failure
-npm run acceptance:impact
-npm pack --dry-run
-```
-
-Observed outcomes:
-
-- baseline TypeScript, Vitest, and build checks passed
-- MCP smoke passed
-- acceptance commands passed
-- npm dry-run package creation succeeded
-
-Important note:
-
-Passing acceptance here does not mean all outputs are equally trustworthy. Some acceptance outputs already reveal heuristic overreach and explainability gaps, which are documented below.
-
-## Executive Summary
-
-The project already has the shape of a genuinely useful testing-intelligence MCP server.
-
-Its strongest qualities today are:
-
-- coherent end-to-end workflow coverage
-- evidence-first response structure
-- safe read-only posture toward consumer repositories
-- useful backend bridging through indexed API knowledge
-
-If I had to use it daily, I would trust it for:
-
-- discovering reusable tests and setup patterns
-- drafting initial test skeletons
-- identifying likely frontend/runtime coverage gaps
-- collecting triage context faster than manual grepping
-
-I would not yet fully trust it for:
-
-- authoritative rerun routing
-- precise impact mapping
-- long-lived server freshness
-- packaging and onboarding portability
-- “review this draft and tell me whether it fits project conventions” workflows
-
-The next major gains will come less from adding another narrow helper and more from strengthening:
-
-1. trust in output quality
-2. daily authoring ergonomics
-3. operability and release integrity
-
-## What Is Already Strong
-
-### 1. The workflow coverage is coherent
-
-This is not a random set of MCP tools. The current registry covers a real daily loop:
-
-- find existing assets
-- inspect frontend contract
-- inspect runtime logic
-- map current coverage
-- detect likely gaps
-- generate a blueprint
-- recommend selectors
-- propose setup/teardown
-- triage failures
-- build rerun scope
-- analyze diff impact
-
-Relevant files:
-
-- `src/mcp/registerTools.ts`
-- `README.md`
-- `docs/tool-catalog.md`
-
-### 2. The normalized output contract is a major strength
-
-The shared result envelope makes the server much more LLM-friendly than most internal helper tools:
-
-- `summary`
-- `evidence`
-- `recommended_actions`
-- `repo_paths`
-- `confidence`
-- `warnings`
-
-This is one of the strongest architectural choices in the repository.
-
-Relevant files:
-
-- `src/contracts/toolOutput.ts`
-- `src/contracts/confidence.ts`
-- `src/domain/shared.ts`
-
-### 3. The product boundary is correct
-
-The server is intentionally local-first and analysis-first:
-
-- it reads consumer repos
-- it indexes knowledge
-- it drafts and recommends
-- it does not silently edit consumer repositories
-
-That is the right safety boundary for a testing-intelligence assistant.
-
-Relevant files:
-
-- `README.md`
-- `docs/architecture/overview.md`
-- `skills/overkill-generate-test/SKILL.md`
-
-### 4. Backend-aware test assistance is stronger than average
-
-The `ApiDiscoveryCatalog` + `api-interaction` + `backend-endpoint` model is one of the best differentiators in the current product.
-
-It gives the server a meaningful way to reason about:
-
-- request flows
-- backend params
-- contract hints
-- API-heavy debugging
-
-without hardcoding a giant endpoint map.
-
-Relevant files:
-
-- `src/indexer/extractors/apiDiscoveryExtractor.ts`
-- `src/indexer/extractors/apiInteractionExtractor.ts`
-- `README.md`
-- `docs/tool-catalog.md`
-
-## What Would Frustrate Me As A Daily User
-
-### 1. There is no MCP tool for reviewing a draft test
-
-The server can help me generate a draft, but it cannot yet close the loop by answering:
-
-- does this draft really reuse project patterns?
-- is the setup correct for the area?
-- are the selectors stable?
-- where is the likely flake risk?
-- what in the draft duplicates existing project code?
-
-There is a bundled skill for this idea, but not a first-class MCP tool.
-
-Why this matters:
-
-- daily authoring is not only about generation
-- review quality is usually more valuable than first-draft generation
-- a “draft reviewer” would directly reduce flaky and off-style tests
-
-Relevant files:
-
-- `skills/overkill-review-test-draft/SKILL.md`
-- `src/mcp/registerTools.ts`
-
-### 2. Failure debugging is still too shallow
-
-Today the server can summarize CI context and extract artifact-backed hypotheses, which is useful.
-
-But if I were using this every day, I would also want:
-
-- log search
-- artifact clustering
-- screenshot comparison
-- API capture diffing
-- flaky fingerprint grouping
-- historical failure trend summaries
-
-Current coverage is helpful, but still feels like “triage starter” rather than “deep failure debugging assistant”.
-
-Relevant files:
-
-- `src/domain/triageFailedRun.ts`
-- `src/domain/buildRerunScope.ts`
-- `docs/tool-catalog.md`
-
-### 3. Batch and portfolio workflows are missing
-
-Current tools are mostly single-input and single-feature oriented.
-
-That is good for v1, but a daily operator will eventually want:
-
-- review coverage across a widget family
-- compare two snapshots or two revisions
-- generate a batch gap report for a feature area
-- track risk concentration across changed files
-- identify coverage drift over time
-
-This is one of the most promising expansion areas.
-
-### 4. The server sometimes looks more certain than it should
-
-The most important trust issue in the current product is not “it crashes”.
-It is “it can produce plausible-looking but over-broad results”.
-
-Two real examples from verified command output:
-
-1. `npm run acceptance:impact`
-
-For a diff on:
-
-- `C:/gitrep/caseplatform-web/vcm/jqVCM/widget/caption/CaptionSingleton.ts`
-
-the output includes clearly plausible caption-related specs, but also unrelated-looking items such as:
-
-- translation tests
-- constructor context tests
-
-That means the current impact logic can over-select because it is still strongly name/token based.
-
-2. `npm run acceptance:failure`
-
-The captured API evidence includes:
-
-- `DELETE /platform/rs/udmmessagequeue/...`
-
-but the linked high-confidence API/backend evidence in the same output includes:
-
-- `api interaction: sec`
-- multiple `sec` backend endpoints
-
-That mismatch is a strong signal that current failure-bridge heuristics can connect the user to the wrong backend context while still looking evidence-backed.
-
-This does not make the server useless.
-It does mean the product should become better at saying:
-
-- “this is exploratory”
-- “this is weakly matched”
-- “this is only name-based”
-- “do not treat this as rerun-routing truth”
-
-## Quality and Reliability Risks
-
-### 1. Knowledge freshness is too weak for long-lived MCP sessions
-
-`ensureIndexed()` currently treats “database is non-empty” as enough to stop refreshing for the lifetime of the current app context.
-
-That creates a real risk:
-
-- repo changes
-- branch switches
-- moved files
-- deleted repositories
-- updated artifacts
-
-may not be reflected during long-lived usage unless the process is restarted or indexing is explicitly rerun.
-
-Relevant files:
-
-- `src/domain/shared.ts`
-- `src/indexer/refreshPipeline.ts`
-- `src/knowledge/repositories.ts`
-
-What I would add:
-
-- refresh fingerprints per repo
-- TTL or file-hash invalidation
-- explicit refresh status reporting
-- purge behavior when configured repos disappear
-
-### 2. Extraction quality is still mostly heuristic
-
-This is already documented as a v1 limitation, and the code confirms it.
-
-The highest-risk parts are still based on:
-
-- regex extraction
-- line-based scanning
-- token overlap
-- basename/path heuristics
-
-That is acceptable for v1 exploration, but it becomes the main ceiling for:
-
-- contract accuracy
-- runtime branch accuracy
-- backend bridge accuracy
-- diff impact trustworthiness
-
-Relevant files:
-
-- `src/indexer/extractors/frontendContractExtractor.ts`
-- `src/indexer/extractors/frontendRuntimeExtractor.ts`
-- `src/indexer/extractors/apiDiscoveryExtractor.ts`
-- `src/domain/analyzeDiffImpact.ts`
-
-What I would add:
-
-- AST-aware parsing where impact is highest
-- clearer provenance for every extracted signal
-- stronger dedupe
-- more “fail closed” behavior on ambiguous matches
-
-### 3. The current test strategy under-protects the riskiest edges
-
-The test suite is good and broad for this stage, but it is still mostly stronger on positive-path coverage than on “trust erosion” cases.
-
-The highest-value missing tests are around:
-
-- stale cache behavior
-- repo disappearance
-- malformed API capture shapes
-- false-positive suppression
-- confidence downgrades for weak matches
-- path normalization consistency
-
-Relevant files:
-
-- `tests/domain/analyzeDiffImpact.test.ts`
-- `tests/domain/triageFailedRun.test.ts`
-- `tests/indexer/refreshPipeline.caseplatformIndex.test.ts`
-
-### 4. Scaling will likely hit refresh/indexing first
-
-The current refresh model still depends on broad globs and whole-file rescans.
-
-This is fine in small-to-medium local use, but if the target repos grow or the operator refreshes often, the cost will scale with repository size instead of changed surface.
-
-Relevant files:
-
-- `src/connectors/autotests2Connector.ts`
-- `src/connectors/caseplatformConnector.ts`
-- `src/indexer/refreshPipeline.ts`
-
-What I would add:
-
-- incremental indexing
-- file-hash caching
-- changed-file-only refresh modes
-- refresh timing metrics
-
-## Operability and Packaging Gaps
-
-### 1. Metadata drift is real
-
-Verified mismatch:
-
-- `package.json` version is `1.1.5`
-- `.codex-plugin/plugin.json` version is `1.1.0`
-- `src/mcp/server.ts` version is `1.1.0`
-
-The plugin manifest also still contains placeholder OpenAI identity fields instead of the actual project identity.
-
-Relevant files:
-
-- `package.json`
-- `.codex-plugin/plugin.json`
-- `src/mcp/server.ts`
-
-### 2. Published package shape and plugin story are not fully aligned
-
-Verified through `npm pack --dry-run`:
-
-- npm tarball includes `dist/`, `bin/`, `skills/`, `docs/`, templates, assets, and config example
-- npm tarball does **not** include `.codex-plugin/plugin.json`
-- npm tarball does **not** include `.mcp.json`
-
-That means the published package is useful as a CLI/MCP library artifact, but not as a fully bundled Codex plugin package in the same way the repository layout suggests.
-
-Relevant files:
-
-- `package.json`
-- `README.md`
-- `docs/plugin-packaging.md`
-
-### 3. Portability is still too machine-shaped
-
-The checked-in `.mcp.json`, multiple docs, and some tests still embed workstation-specific absolute paths.
-
-This makes the installation story look greener than it really is for a new environment.
-
-Relevant files:
-
-- `.mcp.json`
-- `docs/plugin-packaging.md`
-- `tests/docs/pluginManifest.test.ts`
-- `tests/helpers/testContext.ts`
-
-### 4. Verification gates should match the product promise better
-
-`verify` is currently useful, but it does not cover all the product-level trust surfaces highlighted by the docs.
-
-If this were my daily tool, I would want a release-oriented verification layer that checks:
-
-- metadata consistency
-- packaging expectations
-- MCP handshake
-- at least one acceptance-level usage scenario
-
-Relevant files:
-
-- `package.json`
-- `README.md`
-- `docs/operator-cookbook.md`
-
-## What I Would Add Next
-
-### Highest-impact additions
-
-1. `review_test_draft`
-
-Proposed MCP tool purpose:
-
-- accept a draft spec or draft snippet
-- compare it against existing project assets and conventions
-- flag duplication, flake risk, selector weakness, setup mistakes, and missed reuse
-
-Why it matters:
-
-- it closes the generation-to-review loop
-- it is immediately useful in daily authoring
-- it directly improves test quality, not only test speed
-
-2. richer artifact forensics
-
-Examples:
-
-- `compare_api_captures`
-- `cluster_failure_signatures`
-- `summarize_artifact_history`
-- `search_failure_logs`
-
-Why it matters:
-
-- failure debugging is where operators spend real time
-- this would upgrade the server from “context helper” to “debugging partner”
-
-3. batch coverage and drift tools
-
-Examples:
-
-- `summarize_area_coverage`
-- `compare_coverage_snapshots`
-- `find_riskiest_feature_clusters`
-
-Why it matters:
-
-- teams do not work on one widget forever
-- portfolio-level visibility is the next maturity step
-
-4. operator diagnostics
-
-Examples:
-
-- `doctor_installation`
-- `show_refresh_status`
-- `explain_confidence`
-
-Why it matters:
-
-- better onboarding
-- better trust
-- easier debugging of the server itself
-
-## Recommended Improvement Tracks
-
-### Track A: Trustworthiness First
-
-Goal:
-
-Make current output safer to rely on before adding many new tools.
-
-Priority work:
-
-1. Add refresh invalidation / freshness tracking
-2. Downgrade confidence more aggressively for name-only or weak matches
-3. Normalize all returned paths consistently
-4. Add tests for false-positive suppression
-5. Mark exploratory matches more explicitly in summaries and warnings
-
-### Track B: Daily Authoring Value
-
-Goal:
-
-Make the server meaningfully more useful during real test-writing sessions.
-
-Priority work:
-
-1. Add `review_test_draft`
-2. Enrich blueprint generation with stronger anti-flake checks
-3. Improve setup/reuse reasoning with better evidence links
-4. Add batch gap and batch coverage tools
-
-### Track C: Operability and Release Integrity
-
-Goal:
-
-Make the server easier to install, trust, publish, and maintain.
-
-Priority work:
-
-1. Remove metadata drift
-2. Make plugin/package story explicit and consistent
-3. Reduce machine-specific assumptions
-4. Add release verification for MCP/package/install surfaces
-
-## Recommended Order
-
-If the goal is “improve on all fronts, but in a way that compounds value”, I would do the next work in this order:
-
-1. `Track A: Trustworthiness First`
-2. `Track C: Operability and Release Integrity`
-3. `Track B: Daily Authoring Value`
-4. deeper semantic extraction and portfolio analytics after the above
-
-Why this order:
-
-- better trust increases the value of every existing tool
-- better operability makes the product easier to adopt and maintain
-- then new capabilities land on top of a stronger base
-
-## Short Conclusion
-
-Iteration 2 verdict:
-
-- the server is already useful
-- the architecture is stronger than the current outer shell
-- the product has real differentiation
-- the most important next step is not “more helpers”
-- the most important next step is making current intelligence more trustworthy and more operable
-
-If I had to work with this server every day, the first improvements I would personally want are:
-
-1. a real draft-review tool
-2. stronger confidence and freshness guarantees
-3. less heuristic overreach in impact and failure routing
-4. a cleaner installation and release story