@mknrt/autotests-overkill 1.1.4 → 1.2.0

package/docs/audits/2026-04-20-iteration-1.md

@@ -0,0 +1,251 @@
+# Audit Iteration 1
+
+Date: 2026-04-20
+
+## Goal
+
+Evaluate the current state of `autotests-overkill` as a strong assistant for:
+
+- test design
+- test drafting
+- test debugging
+- coverage and impact analysis
+
+This first iteration focuses on architectural quality, product readiness, verification coverage, portability, and obvious drift risks.
+
+## Scope
+
+Reviewed:
+
+- product docs and positioning
+- MCP and CLI entrypoints
+- domain and indexing architecture
+- bundled skills
+- package and plugin manifests
+- current verification commands
+- representative tests
+
+Key files reviewed:
+
+- `README.md`
+- `package.json`
+- `.codex-plugin/plugin.json`
+- `.mcp.json`
+- `src/mcp/registerTools.ts`
+- `src/mcp/server.ts`
+- `src/indexer/refreshPipeline.ts`
+- `src/contracts/toolOutput.ts`
+- `src/contracts/confidence.ts`
+- `src/cli/runAcceptance.ts`
+- `docs/architecture/overview.md`
+
+## Verified State
+
+The following command was run during the audit:
+
+```bash
+npm run verify
+```
+
+Observed result:
+
+- TypeScript lint passed
+- Vitest suite passed
+- Build passed
+- `75` tests passed across `40` test files
+
+This means the repository is currently in a healthy technical state at the baseline verification level.
+
+## Strong Areas
+
+### 1. Clear architecture
+
+The project already has a strong boundary model:
+
+- connectors gather evidence
+- indexers normalize and persist it
+- domain services compose outputs
+- MCP exposes narrow, composable tools
+
+This is one of the strongest parts of the repository because it matches the actual product goal: helping an agent reason about tests without silently editing consumer repositories.
+
+Relevant files:
+
+- `docs/architecture/overview.md`
+- `src/indexer/refreshPipeline.ts`
+- `src/mcp/registerTools.ts`
+
+### 2. Strong normalized tool contract
+
+The unified output shape is a major strength:
+
+- `summary`
+- `evidence`
+- `recommended_actions`
+- `repo_paths`
+- `confidence`
+- `warnings`
+
+This makes the project much more reliable as an LLM-facing tool platform because every MCP tool returns a predictable structure.
+
+Relevant files:
+
+- `src/contracts/toolOutput.ts`
+- `src/contracts/confidence.ts`
+- `src/domain/shared.ts`
+
+### 3. Good product stance
+
+The platform consistently keeps the right operating boundary:
+
+- analyze
+- explain
+- draft
+- suggest
+- do not silently mutate consumer repositories
+
+That principle is visible in README, skills, and domain behavior. This is important because it makes the tool safer and more trustworthy in real engineering workflows.
+
+Relevant files:
+
+- `README.md`
+- `skills/overkill-generate-test/SKILL.md`
+- `skills/overkill-onboard/SKILL.md`
+
+### 4. Better-than-average verification mindset
+
+The project does not stop at unit-level checks. It already includes:
+
+- docs checks
+- packaging checks
+- MCP handshake coverage
+- acceptance-style checks for coverage, blueprint, failure triage, and impact analysis
+
+That is a strong base for a tool whose value is mostly in workflow quality and output usefulness.
+
+Relevant files:
+
+- `tests/mcp/stdioHandshake.test.ts`
+- `tests/docs/pluginManifest.test.ts`
+- `tests/docs/packageManifest.test.ts`
+- `tests/cli/runAcceptance.test.ts`
+
+## Weak Areas
+
+### 1. Release metadata drift
+
+There is already visible drift between package, plugin, and MCP metadata.
+
+Observed examples:
+
+- `package.json` version: `1.1.5`
+- `.codex-plugin/plugin.json` version: `1.1.0`
+- `src/mcp/server.ts` version: `1.1.0`
+
+There is also identity drift in plugin metadata:
+
+- plugin manifest still points to `openai.com`
+- plugin manifest still points to `https://github.com/openai/plugins`
+- plugin manifest still uses `OpenAI Codex` as project author/developer identity
+
+Why this matters:
+
+- packaging looks less trustworthy
+- plugin identity is partially incorrect
+- release information can become misleading for users and tooling
+
+Relevant files:
+
+- `package.json`
+- `.codex-plugin/plugin.json`
+- `src/mcp/server.ts`
+
+### 2. Poor portability
+
+The project currently contains machine-specific assumptions that reduce reliability outside the current workstation.
+
+Observed examples:
+
+- `.mcp.json` contains hardcoded `cwd: C:\\gitrep\\autotests-overkill`
+- docs repeat that exact local path
+- tests assert that exact path
+- test helpers assume `C:/gitrep/autotests2` and `C:/gitrep/caseplatform-web`
+
+Why this matters:
+
+- local checkout usage is brittle
+- plugin packaging is less reusable
+- tests are tightly coupled to one environment shape
+- onboarding quality drops sharply outside the author machine
+
+Relevant files:
+
+- `.mcp.json`
+- `docs/plugin-packaging.md`
+- `tests/docs/pluginManifest.test.ts`
+- `tests/helpers/testContext.ts`
+
+### 3. Verify command does not fully protect the product promise
+
+The docs position `mcp:smoke` and acceptance commands as important proof of installation and behavior, but `npm run verify` currently includes only:
+
+- lint
+- test
+- build
+
+This is not broken, but it does leave a gap between:
+
+- what the product says matters
+- what the main verification gate actually enforces
+
+Why this matters:
+
+- release confidence depends on social discipline instead of automation
+- MCP/package workflow regressions may survive longer than necessary
+- product-level trust is slightly weaker than the current architecture deserves
+
+Relevant files:
+
+- `README.md`
+- `docs/operator-cookbook.md`
+- `package.json`
+
+## Assessment
+
+This repository is already a strong first-generation foundation for an AI-assisted testing platform. Its strongest qualities are:
+
+- disciplined architecture
+- evidence-first tool outputs
+- strong problem framing
+- good initial verification culture
+
+Its most important current risks are not algorithmic quality failures or test instability. The higher-priority risks are:
+
+- release/metadata drift
+- portability problems
+- incomplete automation around product-level verification
+
+In other words: the core is promising, but the outer delivery shell is weaker than the inner design.
+
+## Recommended Next Iteration
+
+Recommended focus for the next improvement cycle:
+
+`operability / release integrity`
+
+That iteration should prioritize:
+
+1. Synchronizing package, plugin, and MCP versions and identity metadata
+2. Reducing machine-specific assumptions in MCP packaging and test setup
+3. Adding anti-drift checks so metadata divergence is caught automatically
+4. Strengthening the verification gate around packaging and MCP operability
+
+## Short Conclusion
+
+Iteration 1 verdict:
+
+- the project is technically healthy
+- the architecture is strong
+- the product direction is correct
+- the most valuable next work is not feature expansion
+- the most valuable next work is making the platform more portable, more release-safe, and harder to drift out of sync