@agentskit/doc-bridge 1.6.4 → 1.7.44
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +243 -0
- package/action.yml +1 -1
- package/dist/cli/program.js +793 -137
- package/dist/cli/program.js.map +1 -1
- package/dist/config/index.d.ts +1 -1
- package/dist/config/index.js +38 -2
- package/dist/config/index.js.map +1 -1
- package/dist/{index-DudNuwI5.d.ts → index-C2PCQSrB.d.ts} +216 -25
- package/dist/index.d.ts +837 -67
- package/dist/index.js +858 -127
- package/dist/index.js.map +1 -1
- package/docs/PRD-enterprise-hardening.md +288 -0
- package/docs/adr/0001-enterprise-verification-contract.md +35 -0
- package/docs/knowledge-engine-runbook.md +18 -2
- package/docs/spec/analyzer-plugin-v1.md +24 -0
- package/docs/spec/benchmark-v1.md +30 -0
- package/docs/spec/config-v1.md +111 -0
- package/docs/validation-cycle-plan.md +236 -0
- package/docs/verification-harness.md +33 -4
- package/mcpb/manifest.json +1 -1
- package/package.json +1 -1
- package/scripts/report-visual-check.mjs +45 -10
- package/scripts/verification-harness.mjs +216 -13
- package/skills/doc-bridge-handoff/scripts/resolve-handoff.mjs +1 -1
- package/src/agents/registry-adapter.ts +31 -7
- package/src/cli/program.ts +44 -11
- package/src/config/index.ts +2 -0
- package/src/config/schema.ts +56 -0
- package/src/discovery/documentation.ts +46 -5
- package/src/discovery/repository.ts +95 -16
- package/src/index.ts +29 -0
- package/src/metrics/benchmark.ts +176 -0
- package/src/plugins/contract.ts +89 -0
- package/src/reconciliation/reconcile.ts +137 -3
- package/src/report/html.ts +302 -78
- package/src/schemas/knowledge.ts +16 -1
- package/src/version.ts +1 -1
- package/src/workflow/engine.ts +65 -9
|
@@ -0,0 +1,288 @@
|
|
|
1
|
+
# PRD: Enterprise Hardening for Doc Bridge
|
|
2
|
+
|
|
3
|
+
## Problem Statement
|
|
4
|
+
|
|
5
|
+
Doc Bridge already provides deterministic repository discovery, documentation discovery, relationship reconciliation, persistent workflow artifacts, CLI and MCP surfaces, Registry-agent assistance, and a read-only HTML report. The current implementation has proven that it can scan a large real repository, preserve evidence, expose documentation/code drift, reduce agent context, and render an interactive architecture view.
|
|
6
|
+
|
|
7
|
+
It is not yet an enterprise-grade knowledge bridge. The current validation still uses a `poc` dogfood profile, some analyzer surfaces are explicitly incomplete, semantic precision is not measured strongly enough, recovery and operational guarantees need broader failure validation, and the full product surface is not always exercised in one auditable contract. A large number of findings can also represent missing documentation rather than actionable defects, so users need confidence and quality metrics instead of an undifferentiated pass/fail result.
|
|
8
|
+
|
|
9
|
+
The product must help humans and agents establish a trustworthy shared model of a repository while remaining honest about uncertainty. It must never present inferred or heuristic architecture as observed fact, must never hide unsupported analysis, and must never claim completion without evidence from the real artifact.
|
|
10
|
+
|
|
11
|
+
## Solution
|
|
12
|
+
|
|
13
|
+
Harden Doc Bridge around a stable, language-neutral knowledge-engine contract with a first-class JS/TS analyzer implementation. Add explicit analyzer coverage, precision/recall evaluation, confidence and provenance, configurable runtime and generated-code adapters, reliable resumable execution, enterprise configuration profiles, Registry-agent guardrails, documentation quality signals, and complete real-surface validation.
|
|
14
|
+
|
|
15
|
+
The enterprise contract will be strict by default:
|
|
16
|
+
|
|
17
|
+
- Every requested capability is either validated or explicitly declared not applicable.
|
|
18
|
+
- No silent exemptions are allowed. Every exemption requires a reason and is recorded in the run.
|
|
19
|
+
- Unsupported analysis is visible, measurable, and included in quality reporting.
|
|
20
|
+
- Deterministic analysis is the source of truth. Registry agents may discover, classify, explain, and propose, but may not approve or silently mutate results.
|
|
21
|
+
- Reports remain read-only. Mechanical fixes require explicit human approval and a fresh post-apply verification run.
|
|
22
|
+
- The default Registry agent is `ecosystem-doc-bridge-corpus-scanner`; alternative AgentsKit Registry agents are configurable through the same contract.
|
|
23
|
+
- JS/TS and Markdown are the initial supported implementation scope. The extension contract must not require a future schema rewrite to add other languages.
|
|
24
|
+
|
|
25
|
+
## Goals
|
|
26
|
+
|
|
27
|
+
1. Improve architecture and documentation analysis coverage without inventing certainty.
|
|
28
|
+
2. Quantify semantic quality, efficiency, reliability, and user-visible behavior.
|
|
29
|
+
3. Make workflow execution resumable, idempotent, versioned, auditable, and failure-safe.
|
|
30
|
+
4. Provide an ESLint/tsc-like configuration and plugin experience with safe defaults and enterprise enforcement.
|
|
31
|
+
5. Use Registry agents as bounded, evidence-grounded assistants with provenance and human approval.
|
|
32
|
+
6. Validate the actual CLI, MCP, report, documentation, and applicable runtime surfaces end to end.
|
|
33
|
+
7. Produce anonymization-safe longitudinal data suitable for internal improvement and external case studies.
|
|
34
|
+
|
|
35
|
+
## Non-Goals
|
|
36
|
+
|
|
37
|
+
- Building a hosted SaaS dashboard or central repository database.
|
|
38
|
+
- Replacing source control, CI, package managers, language compilers, or observability platforms.
|
|
39
|
+
- Claiming complete semantic understanding of reflection, arbitrary code generation, or dynamic runtime behavior.
|
|
40
|
+
- Automatically applying documentation or code changes without human approval.
|
|
41
|
+
- Supporting every programming language in the first implementation.
|
|
42
|
+
- Requiring endpoint or database validation for a project that does not expose or use those surfaces.
|
|
43
|
+
|
|
44
|
+
## User Stories
|
|
45
|
+
|
|
46
|
+
### Analysis coverage and architecture
|
|
47
|
+
|
|
48
|
+
1. As a repository owner, I want every in-scope package and application classified, so that the architecture map reflects the real project boundaries.
|
|
49
|
+
2. As a repository owner, I want package, module, and file levels to be independently navigable, so that I can move from a system overview to exact evidence.
|
|
50
|
+
3. As a human reviewer, I want relation direction and relation kind preserved, so that the graph does not hide ownership or dependency direction.
|
|
51
|
+
4. As a human reviewer, I want dynamic imports and runtime wiring reported with their resolution status, so that unresolved behavior is visible rather than silently omitted.
|
|
52
|
+
5. As a human reviewer, I want generated code and source-map limitations identified, so that generated artifacts are not mistaken for fully analyzed source.
|
|
53
|
+
6. As an analyzer author, I want a language-neutral coverage model, so that future analyzers can report the same completeness semantics.
|
|
54
|
+
7. As a repository owner, I want configured framework/runtime adapters, so that known registration and loading patterns can be resolved without broad false-positive heuristics.
|
|
55
|
+
8. As a human reviewer, I want disconnected nodes, cycles, hotspots, and likely single points of failure surfaced with evidence and confidence, so that structural risks can be investigated.
|
|
56
|
+
9. As a human reviewer, I want inferred groupings separated from observed package boundaries, so that derived architecture remains distinguishable from source facts.
|
|
57
|
+
10. As an agent, I want stable entity and relation identities across equivalent rescans, so that I can cite knowledge without losing references after harmless file reordering.
|
|
58
|
+
|
|
59
|
+
### Semantic quality and metrics
|
|
60
|
+
|
|
61
|
+
11. As a product owner, I want precision and recall measured against known fixtures, so that a green run means more than compilation or unit tests.
|
|
62
|
+
12. As a product owner, I want 100% recall for the supported known-case fixture matrix, so that supported findings are not missed.
|
|
63
|
+
13. As a product owner, I want at least 95% precision for supported known-case fixtures, so that users are not overwhelmed by false positives.
|
|
64
|
+
14. As a human reviewer, I want every finding to include evidence, provenance, confidence, and remediation context, so that I can decide whether it is actionable.
|
|
65
|
+
15. As a human reviewer, I want observed, declared, inferred, heuristic, unresolved, and not-analyzed states separated, so that uncertainty is explicit.
|
|
66
|
+
16. As a product owner, I want undocumented, stale, conflicting, unresolved, and confirmed results quantified separately, so that documentation improvement can be tracked over time.
|
|
67
|
+
17. As a product owner, I want comparison between immutable snapshots, so that improvements and regressions are measurable.
|
|
68
|
+
18. As a product owner, I want agent latency, response size, token estimate, hit rate, and context reduction tracked, so that efficiency claims are evidence-based.
|
|
69
|
+
19. As a product owner, I want publication-safe aggregate metrics without repository secrets, paths, prompts, or contents, so that case studies can be prepared safely.
|
|
70
|
+
20. As an operator, I want benchmark thresholds and baseline provenance recorded, so that a baseline cannot be silently replaced to make a regression pass.
|
|
71
|
+
|
|
72
|
+
### Reliable execution and recovery
|
|
73
|
+
|
|
74
|
+
21. As an operator, I want every workflow stage persisted with input and output hashes, so that execution can be audited and resumed.
|
|
75
|
+
22. As an operator, I want interrupted runs to resume from the last valid stage, so that large repository scans do not need to restart unnecessarily.
|
|
76
|
+
23. As an operator, I want repeated execution with the same source, configuration, and pipeline version to reuse valid artifacts, so that the workflow is idempotent.
|
|
77
|
+
24. As an operator, I want concurrent runs to be isolated or rejected safely, so that artifacts cannot be mixed or corrupted.
|
|
78
|
+
25. As an operator, I want cancellation and failure states to preserve valid evidence, so that partial work is recoverable and clearly labeled.
|
|
79
|
+
26. As an operator, I want schema and analyzer version migrations explicit, so that old artifacts cannot be consumed as if they were current.
|
|
80
|
+
27. As an operator, I want resource limits, timeouts, and bounded output enforced at trust boundaries, so that hostile or oversized repositories cannot exhaust the process.
|
|
81
|
+
28. As an auditor, I want a complete state transition history and exact run identifier, so that I can reconstruct what happened.
|
|
82
|
+
29. As a human approver, I want approvals bound to the exact source revision, contract, and evidence hashes, so that an approval cannot be reused for different output.
|
|
83
|
+
30. As an operator, I want task-owned temporary artifacts cleaned after a run without deleting ambiguous user data, so that the workspace remains maintainable.
|
|
84
|
+
|
|
85
|
+
### Configuration and extensibility
|
|
86
|
+
|
|
87
|
+
31. As a new user, I want a safe default profile that works without extensive configuration, so that initial adoption has low friction.
|
|
88
|
+
32. As a project owner, I want strict, poc, custom, and enterprise profiles, so that enforcement can match project maturity without weakening the enterprise contract.
|
|
89
|
+
33. As a project owner, I want profile overrides to be explicit and reasoned, so that reduced rigor is visible in the report and audit log.
|
|
90
|
+
34. As a project owner, I want analyzer, reconciliation, report, workflow, agent, and validation settings in one versioned schema, so that behavior is reproducible.
|
|
91
|
+
35. As an analyzer author, I want a stable plugin contract with declared capabilities, supported constructs, version compatibility, and coverage output, so that new languages and frameworks can be added independently.
|
|
92
|
+
36. As a project owner, I want custom relation kinds, runtime methods, framework adapters, and documentation rules, so that the engine can represent local architecture conventions.
|
|
93
|
+
37. As a project owner, I want invalid configuration to fail before mutation or scanning, so that a typo cannot produce misleading evidence.
|
|
94
|
+
38. As a user, I want useful diagnostics for configuration errors, so that the correction path is clear.
|
|
95
|
+
39. As an enterprise operator, I want backward-compatible schema evolution and migration guidance, so that upgrading the library is safe.
|
|
96
|
+
40. As a maintainer, I want the common engine isolated from language-specific analyzers, so that future expansion does not destabilize current behavior.
|
|
97
|
+
|
|
98
|
+
### Registry-agent assistance
|
|
99
|
+
|
|
100
|
+
41. As a user, I want the default discovery assistant to come from the AgentsKit Registry, so that the agent identity and capabilities are discoverable.
|
|
101
|
+
42. As a project owner, I want to configure another Registry agent without changing the engine, so that agent choice remains flexible.
|
|
102
|
+
43. As a user, I want deterministic mode available for every assisted workflow, so that proposals can be replayed and compared.
|
|
103
|
+
44. As a user, I want assisted proposals linked to the source snapshot, report, agent ID, agent version, and evidence, so that the proposal is auditable.
|
|
104
|
+
45. As a human reviewer, I want the agent to propose classifications, documentation improvements, stale-doc explanations, and follow-up checks, so that discovery is faster.
|
|
105
|
+
46. As a human reviewer, I want the agent prevented from approving its own proposal, so that human authority remains explicit.
|
|
106
|
+
47. As an operator, I want agent token budgets, timeouts, redaction, and failure behavior configurable, so that assisted runs are cost-bounded.
|
|
107
|
+
48. As an operator, I want agent failures to degrade to a clearly labeled deterministic result, so that failure is not hidden as success.
|
|
108
|
+
49. As a human reviewer, I want unsupported agent claims rejected or labeled as unverified, so that natural-language confidence cannot override evidence.
|
|
109
|
+
50. As a product owner, I want assisted and deterministic results reported as separate evidence classes, so that case-study metrics remain honest.
|
|
110
|
+
|
|
111
|
+
### Documentation quality and reconciliation
|
|
112
|
+
|
|
113
|
+
51. As a repository owner, I want all in-scope documentation classified, so that presence is not confused with usefulness.
|
|
114
|
+
52. As a repository owner, I want every package and application to have a documentation status, so that missing and stale documentation are visible.
|
|
115
|
+
53. As a human reviewer, I want documentation claims linked to observed code relations, so that contradictions can be located.
|
|
116
|
+
54. As a human reviewer, I want orphaned, stale, conflicting, and unsupported claims grouped at a useful scope, so that the report remains actionable.
|
|
117
|
+
55. As a human reviewer, I want suggestions to include the source evidence and proposed change type, so that fixes can be reviewed mechanically.
|
|
118
|
+
56. As a human reviewer, I want documentation quality rules configurable by project, so that local standards are respected.
|
|
119
|
+
57. As a project owner, I want documentation/code comparison at file, module, or package scope, so that monorepos can choose useful signal granularity.
|
|
120
|
+
58. As a human reviewer, I want high finding volume explained by category and evidence density, so that a documentation debt baseline is interpretable.
|
|
121
|
+
59. As an agent, I want a compact, stable corpus and focused search results, so that I can find ownership and architecture knowledge without loading the entire repository.
|
|
122
|
+
60. As a human reviewer, I want all decisions and structural changes reflected in documentation and ADR/RFC records, so that knowledge does not decay after implementation.
|
|
123
|
+
|
|
124
|
+
### Complete product validation
|
|
125
|
+
|
|
126
|
+
61. As a maintainer, I want the real package artifact installed in a real consumer repository, so that source-only validation cannot pass incorrectly.
|
|
127
|
+
62. As a CLI user, I want discovery, indexing, reconciliation, report, query, and gate commands exercised as real commands, so that the public workflow is validated.
|
|
128
|
+
63. As an MCP user, I want every supported read-only MCP tool exercised against the packaged artifact, so that the agent surface is validated end to end.
|
|
129
|
+
64. As a human user, I want the report validated in a real browser across responsive viewports and themes, so that code correctness is not mistaken for usable UI behavior.
|
|
130
|
+
65. As a human user, I want interaction, loading, accessibility, contrast, overflow, errors, and lazy-loading behavior measured, so that visual quality is a real gate.
|
|
131
|
+
66. As a project owner, I want endpoint and database checks required when the configured project uses them, so that applicable runtime behavior is not skipped.
|
|
132
|
+
67. As an operator, I want non-applicable surfaces declared with reasons, so that the harness does not force irrelevant checks or hide relevant ones.
|
|
133
|
+
68. As a maintainer, I want a complete verification contract tied to the exact source revision and configuration, so that “complete” is reproducible.
|
|
134
|
+
69. As a release owner, I want issue, pull request, and ticket transitions recorded only after authorization, so that external tracking is auditable.
|
|
135
|
+
70. As a product owner, I want residual risks and unsupported areas in the final report, so that enterprise decisions are based on limitations as well as successes.
|
|
136
|
+
|
|
137
|
+
## Implementation Decisions
|
|
138
|
+
|
|
139
|
+
### Common knowledge engine
|
|
140
|
+
|
|
141
|
+
- Keep a canonical versioned knowledge model for entities, relations, documents, evidence, coverage, diagnostics, proposals, workflow stages, and metrics.
|
|
142
|
+
- Separate observed facts from declarations, inferences, heuristics, unresolved states, and not-analyzed states.
|
|
143
|
+
- Preserve stable content-addressed identifiers and deterministic ordering.
|
|
144
|
+
- Add confidence and provenance fields without making confidence a substitute for evidence.
|
|
145
|
+
- Keep raw relations and semantic comparison relations separate so aggregation does not destroy traceability.
|
|
146
|
+
|
|
147
|
+
### Analyzer and coverage architecture
|
|
148
|
+
|
|
149
|
+
- Define a language-neutral analyzer capability contract with discovery, relation extraction, document extraction, coverage reporting, diagnostics, and version metadata.
|
|
150
|
+
- Keep JS/TS as the initial analyzer and Markdown as the initial documentation format.
|
|
151
|
+
- Add explicit support for configured runtime wiring and dynamic loading adapters.
|
|
152
|
+
- Treat generated code as a declared boundary and support source-map or generated-manifest adapters where available.
|
|
153
|
+
- Report coverage by supported construct and by unresolved construct, including counts and evidence.
|
|
154
|
+
- Add known-positive, known-negative, ambiguous, and unsupported fixtures for every supported analyzer capability.
|
|
155
|
+
|
|
156
|
+
### Quality and reconciliation
|
|
157
|
+
|
|
158
|
+
- Add a precision/recall evaluation runner whose fixtures contain expected entities, relations, documentation claims, findings, and unsupported boundaries.
|
|
159
|
+
- Enforce 100% recall and at least 95% precision for the supported fixture matrix before an enterprise profile can pass.
|
|
160
|
+
- Keep the agentskit-os result as a real-world baseline, not as a semantic truth set.
|
|
161
|
+
- Add finding density, category distribution, evidence completeness, and change-over-time metrics.
|
|
162
|
+
- Make package/module/file reconciliation scope configurable while retaining raw file-level evidence.
|
|
163
|
+
|
|
164
|
+
### Workflow and safety
|
|
165
|
+
|
|
166
|
+
- Retain the internal state machine and make transitions, checkpoints, input hashes, output hashes, pipeline version, analyzer versions, and configuration hash mandatory.
|
|
167
|
+
- Make resume, idempotency, cancellation, concurrency protection, bounded resources, and migration behavior explicit contract outcomes.
|
|
168
|
+
- Fail closed when evidence or a required validation surface is unavailable.
|
|
169
|
+
- Keep fix proposals mechanical and human-gated; applying a proposal invalidates prior completion and requires a fresh verification run.
|
|
170
|
+
|
|
171
|
+
### Configuration and profiles
|
|
172
|
+
|
|
173
|
+
- Provide `default`, `strict`, `poc`, `custom`, and `enterprise` profiles.
|
|
174
|
+
- The `enterprise` profile forbids silent exemptions and requires all applicable surfaces; a not-applicable declaration must contain a reason.
|
|
175
|
+
- Keep configuration strict and versioned, with capability-specific settings and plugin compatibility checks.
|
|
176
|
+
- Preserve the low-friction default profile, while making every enterprise relaxation explicit and reportable.
|
|
177
|
+
|
|
178
|
+
### AgentsKit Registry integration
|
|
179
|
+
|
|
180
|
+
- Use `ecosystem-doc-bridge-corpus-scanner` as the default Registry agent.
|
|
181
|
+
- Allow another Registry agent to be selected by configuration only when its identity, version, capabilities, and contract are recorded.
|
|
182
|
+
- Separate deterministic evidence from assisted proposals.
|
|
183
|
+
- Enforce redaction, token/time budgets, provenance, bounded output, human approval, and no self-approval.
|
|
184
|
+
- Keep agent failure visible and never convert an unavailable agent into an ungrounded success.
|
|
185
|
+
|
|
186
|
+
### Report and product surfaces
|
|
187
|
+
|
|
188
|
+
- Keep the report read-only and offline-capable.
|
|
189
|
+
- Preserve progressive loading: package topology and compact evidence indexes first, module/file evidence on demand.
|
|
190
|
+
- Add explicit report sections for coverage, confidence, precision/recall, unresolved boundaries, metrics, and residual risk.
|
|
191
|
+
- Ensure architecture, drift, risks, evidence, filters, breadcrumbs, zoom/pan, double-click navigation, accessibility, responsiveness, and contrast are validated in a real browser.
|
|
192
|
+
- Keep CLI and MCP outputs machine-readable, versioned, bounded, and linked to the same canonical artifacts.
|
|
193
|
+
|
|
194
|
+
### Audit and study data
|
|
195
|
+
|
|
196
|
+
- Persist a per-cycle record containing package version, source revision, configuration hash, snapshot/report hashes, workflow run, verification run, metrics, decisions, approvals, exemptions, and residual risks.
|
|
197
|
+
- Store anonymization-safe benchmark data without repository paths, package names, document contents, credentials, or prompts unless explicitly approved for a private audit.
|
|
198
|
+
- Never replace an approved baseline automatically.
|
|
199
|
+
- Record the exact human approval or authorization action for any completion or external tracking transition.
|
|
200
|
+
|
|
201
|
+
## Testing Decisions
|
|
202
|
+
|
|
203
|
+
Tests must validate externally observable behavior and the real artifact. Unit tests and compilation are supporting evidence only; they cannot satisfy the enterprise completion gate by themselves.
|
|
204
|
+
|
|
205
|
+
### Analyzer and coverage tests
|
|
206
|
+
|
|
207
|
+
- Test JS/TS extraction for static imports, exports, literal dynamic loading, configured runtime wiring, unresolved wiring, test-runtime opt-in, generated-code boundaries, stable IDs, evidence, and resource limits.
|
|
208
|
+
- Test each plugin capability contract with positive, negative, ambiguous, and unsupported fixtures.
|
|
209
|
+
- Test that unsupported behavior is surfaced and never silently omitted.
|
|
210
|
+
|
|
211
|
+
### Quality and reconciliation tests
|
|
212
|
+
|
|
213
|
+
- Run the known-case fixture matrix and calculate precision, recall, false positives, false negatives, evidence ratio, and finding density.
|
|
214
|
+
- Test package/module/file scopes and verify that raw evidence remains available after aggregation.
|
|
215
|
+
- Test stale, conflicting, undocumented, confirmed, unresolved, heuristic, and not-analyzed classifications.
|
|
216
|
+
- Test that an empty or narrowed policy is represented as an explicit configuration decision.
|
|
217
|
+
|
|
218
|
+
### Workflow and safety tests
|
|
219
|
+
|
|
220
|
+
- Test deterministic replay, idempotent rerun, stage reuse, interrupted-stage resume, corrupted-artifact rejection, cancellation, concurrency isolation, resource limits, and schema migration.
|
|
221
|
+
- Test that an applied fix proposal requires a new source revision or fresh verification and cannot inherit completion from an earlier run.
|
|
222
|
+
- Test secret redaction, path boundaries, symlinks, untrusted input, and bounded output.
|
|
223
|
+
|
|
224
|
+
### Configuration and plugin tests
|
|
225
|
+
|
|
226
|
+
- Test defaults, strict validation, profile inheritance/override rules, invalid configuration failure, capability compatibility, and explicit exemption reasons.
|
|
227
|
+
- Test adding a minimal synthetic analyzer without modifying the common engine contract.
|
|
228
|
+
|
|
229
|
+
### Registry-agent tests
|
|
230
|
+
|
|
231
|
+
- Test the default Registry agent identity and a configured alternate agent.
|
|
232
|
+
- Test deterministic replay, provenance binding, bounded output, redaction, timeout, budget exhaustion, unavailable-agent behavior, and human approval gating.
|
|
233
|
+
- Test that unsupported claims cannot become confirmed facts.
|
|
234
|
+
|
|
235
|
+
### CLI and MCP tests
|
|
236
|
+
|
|
237
|
+
- Exercise packaged CLI commands as real processes with machine-readable and human-readable output.
|
|
238
|
+
- Build and validate the MCP artifact, initialize a real stdio session, exercise every supported read-only tool, verify framing, errors, schemas, and bounded responses.
|
|
239
|
+
- Verify exit codes and artifact references for success, incomplete analysis, blocked validation, and failed validation.
|
|
240
|
+
|
|
241
|
+
### Report and UI tests
|
|
242
|
+
|
|
243
|
+
- Use real browser validation across configured desktop, tablet, and mobile viewports and light/dark themes.
|
|
244
|
+
- Verify architecture/drift/risks/evidence lenses, package → module → file navigation, breadcrumbs, selection, filters, lazy chunks, zoom/pan, keyboard controls, loading behavior, and error recovery.
|
|
245
|
+
- Verify responsive layout, keyboard accessibility, accessible names, focus visibility, contrast, text overflow, no horizontal page overflow, console errors, failed requests, and interaction latency.
|
|
246
|
+
- Require explicit human visual approval for the exact report hash after automated checks pass.
|
|
247
|
+
|
|
248
|
+
### Consumer validation
|
|
249
|
+
|
|
250
|
+
- Install the exact packed artifact in agentskit-os.
|
|
251
|
+
- Run the full Doc Bridge workflow against the real monorepo.
|
|
252
|
+
- Validate its discovered architecture, documentation inventory, reconciliation findings, Registry proposals, report, agent search efficiency, and all applicable product surfaces.
|
|
253
|
+
- Keep endpoint/database checks conditional on actual project behavior and record non-applicability explicitly.
|
|
254
|
+
|
|
255
|
+
## Acceptance Criteria
|
|
256
|
+
|
|
257
|
+
The enterprise hardening initiative is complete only when all of the following hold for the same source revision and contract:
|
|
258
|
+
|
|
259
|
+
- The enterprise verification profile returns `COMPLETE`.
|
|
260
|
+
- No required surface is silently skipped or exempted.
|
|
261
|
+
- Supported fixture precision is at least 95% and recall is 100%.
|
|
262
|
+
- Every finding has evidence and provenance; unsupported analysis is explicit and quantified.
|
|
263
|
+
- Workflow interruption, resume, idempotency, concurrency, cancellation, and migration checks pass.
|
|
264
|
+
- The default and enterprise configuration profiles are documented and validated.
|
|
265
|
+
- The default Registry agent and an alternate configured Registry agent pass their bounded-assistance contract.
|
|
266
|
+
- Real CLI and MCP package checks pass.
|
|
267
|
+
- Real-browser UI checks pass with zero automated failures and explicit human visual approval.
|
|
268
|
+
- The agentskit-os dogfood run produces a reproducible, anonymization-safe metric record.
|
|
269
|
+
- Documentation, configuration references, release notes, and structural ADR/RFC records are updated.
|
|
270
|
+
- Any external issue, PR, or ticket transition is recorded only after explicit authorization and includes the exact verification run ID.
|
|
271
|
+
- Residual limitations are visible in the final report; enterprise readiness is not claimed while any required gate is pending.
|
|
272
|
+
|
|
273
|
+
## Out of Scope
|
|
274
|
+
|
|
275
|
+
- A hosted multi-tenant service, persistent remote storage, or centralized telemetry backend.
|
|
276
|
+
- Automatic code or documentation mutation without a human approval step.
|
|
277
|
+
- Perfect resolution of arbitrary reflection, runtime metaprogramming, or generated code without project-provided metadata.
|
|
278
|
+
- Full implementation of every future language analyzer in this initiative.
|
|
279
|
+
- Replacing specialized security scanners, compilers, test runners, API contract tools, or database migration tools.
|
|
280
|
+
- Making the current agentskit-os documentation debt disappear as a prerequisite for improving Doc Bridge; that debt remains a measured consumer outcome.
|
|
281
|
+
|
|
282
|
+
## Further Notes
|
|
283
|
+
|
|
284
|
+
- The current agentskit-os dogfood run is evidence for the product but is not a truth set for semantic precision. A separate fixture corpus is mandatory.
|
|
285
|
+
- The current baseline and cycle history must remain immutable unless a human explicitly authorizes a baseline replacement.
|
|
286
|
+
- The implementation should proceed in vertical slices: common contracts and metrics first, analyzer/coverage next, workflow/profile hardening next, Registry and documentation quality next, and complete CLI/MCP/report validation last.
|
|
287
|
+
- Each slice must run the local harness before consuming CI resources. A failed or unavailable required validation blocks completion and must be recorded with the reason.
|
|
288
|
+
- The PRD is intentionally language-neutral at the contract boundary while keeping the first production analyzer scope to JS/TS and Markdown.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# ADR 0001: Enterprise verification contract
|
|
2
|
+
|
|
3
|
+
- Status: Accepted
|
|
4
|
+
- Date: 2026-08-28
|
|
5
|
+
- Decision owners: Doc Bridge maintainers
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
Doc Bridge needs one completion contract for local development, proof-of-concept work, and enterprise validation. The contract must be machine-readable, resumable, auditable, and fail closed when a surface, metric, approval, or applicability decision is missing. It must also remain low-friction for a new repository.
|
|
10
|
+
|
|
11
|
+
## Decision
|
|
12
|
+
|
|
13
|
+
Keep the verification engine as a versioned, dependency-free module in the existing CLI harness. Add named profiles, explicit surface applicability, a validated state-transition graph, and immutable baseline metadata to the existing JSON run record. Use atomic JSON files under `.codex/verification` for recovery and audit; do not introduce a database or a separate service for this contract. Enforce the policy globally, but keep the verification contract project-local because acceptance criteria and runtime surfaces are repository-specific.
|
|
14
|
+
|
|
15
|
+
The profiles are:
|
|
16
|
+
|
|
17
|
+
- `default`: low-friction local validation with conservative defaults and no hidden surface decisions.
|
|
18
|
+
- `strict`: the existing fail-closed behavior for declared checks.
|
|
19
|
+
- `poc`: allowed only with explicit, reasoned exemptions.
|
|
20
|
+
- `custom`: allowed only with explicit, reasoned exemptions and project-owned policy.
|
|
21
|
+
- `enterprise`: every surface is declared, every applicable surface has a required real check, measurement and tracking are required, and every exemption has a reason.
|
|
22
|
+
|
|
23
|
+
Applicability is declared independently for logic, CLI, MCP, UI, documentation, endpoint, and database. A surface marked not applicable must include a reason. Endpoint and database are conditional: a target that uses either must declare and execute its real check; a target that does not use it must document why it is not applicable.
|
|
24
|
+
|
|
25
|
+
The legal lifecycle is `CLARIFYING` → `PLANNED` → `VERIFYING` → a required approval/authorization state → `COMPLETE`, with `BLOCKED` and `FAILED` terminal outcomes for the current run. Runs are never promoted by a failed or stale artifact. Re-running an unchanged pending or completed input is idempotent; a changed source revision, contract, or harness version creates a new run.
|
|
26
|
+
|
|
27
|
+
## Alternatives considered
|
|
28
|
+
|
|
29
|
+
- A database-backed workflow: rejected because it adds operational cost and a new failure surface to a local CLI problem; atomic JSON already provides the required recovery and audit trail.
|
|
30
|
+
- A single `enterprise` boolean: rejected because named profiles make policy visible, composable, and testable.
|
|
31
|
+
- Inferring endpoint/database applicability from package metadata: rejected as unsafe; metadata cannot prove runtime behavior, so the decision remains explicit and is verified by real checks.
|
|
32
|
+
|
|
33
|
+
## Consequences
|
|
34
|
+
|
|
35
|
+
The JSON contract gains strict validation and some existing configs will need explicit profile or surface reasons. Repositories without a contract must stop in `CLARIFYING` until a human defines one. In return, agents and humans receive the same policy, state, evidence, and run ID, and completion cannot be claimed from compilation or unit tests alone.
|
|
@@ -45,14 +45,30 @@ The first implementation analyzes JavaScript/TypeScript and Markdown. Other lang
|
|
|
45
45
|
|
|
46
46
|
## Relation coverage policy
|
|
47
47
|
|
|
48
|
-
|
|
48
|
+
Agent documents under the configured `corpus.agent.root` may use the conventional
|
|
49
|
+
`packages/<name>.md` or `apps/<name>.md` path to declare package coverage without a
|
|
50
|
+
`humanDoc` field. This is intentionally separate from the human bridge: `humanDoc`
|
|
51
|
+
still reports whether an agent has a resolvable human-facing guide.
|
|
52
|
+
|
|
53
|
+
Missing declarations are configurable because not every implementation import is useful documentation. In the root configuration, `reconciliation.scope` selects the semantic comparison level while `reconciliation.requiredRelationKinds` selects the observed relation kinds that must be declared in Markdown. Raw file relations remain available in the snapshot and report for evidence and exploration:
|
|
49
54
|
|
|
50
55
|
```json
|
|
51
56
|
{
|
|
52
57
|
"reconciliation": {
|
|
53
|
-
"
|
|
58
|
+
"scope": "package",
|
|
59
|
+
"requiredRelationKinds": ["imports", "re-exports", "depends-on"]
|
|
54
60
|
}
|
|
55
61
|
}
|
|
56
62
|
```
|
|
57
63
|
|
|
58
64
|
Omit the option to preserve the original all-relation behavior. Use an empty list for low-friction adoption when package/app coverage and explicitly declared claims matter more than documenting every module, test, or external-library import. Existing declarations are still checked for stale, conflicting, and unresolved references.
|
|
65
|
+
|
|
66
|
+
Package health is not inferred from coverage presence alone. A covered package
|
|
67
|
+
is `fresh` only when its relevant relations are verified; undocumented or
|
|
68
|
+
not-analyzed relations make it `unverified`, while conflicting declarations
|
|
69
|
+
make it `stale`. This prevents a high-level coverage document from hiding a
|
|
70
|
+
large architecture/documentation gap.
|
|
71
|
+
|
|
72
|
+
The reconciliation summary also includes deterministic `diagnosticsByCode` and
|
|
73
|
+
`diagnosticsByStatus` rollups. Use them for triage and dashboards, but keep the
|
|
74
|
+
canonical diagnostics and their evidence as the source of truth.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Analyzer plugin contract v1
|
|
3
|
+
description: Language-neutral extension point for Doc Bridge analyzers.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Analyzer plugin contract v1
|
|
7
|
+
|
|
8
|
+
An analyzer plugin declares an `id`, version, supported languages/frameworks, capabilities, knowledge schema version, compatible pipeline major, unsupported constructs, and resource limits. Registration validates this manifest before scanning and keeps plugin order deterministic by ID.
|
|
9
|
+
|
|
10
|
+
The plugin receives bounded file metadata and may return canonical entities, relations, coverage, and diagnostics. The common registry validates that output, stamps coverage with the plugin identity/version, and converts malformed plugin output into `not-analyzed` coverage. A plugin failure never becomes complete analysis and does not affect unrelated registered plugins.
|
|
11
|
+
|
|
12
|
+
Configuration selects plugins with `analysis.plugins`. `enabled`, `order`, `options`, and any override `reason` are explicit. The common workflow, report, CLI, MCP, and verification contracts consume the canonical output and do not need to change when a new language analyzer is added.
|
|
13
|
+
|
|
14
|
+
```ts
|
|
15
|
+
const registry = createAnalyzerRegistry({ pipelineVersion: '1.0.0' })
|
|
16
|
+
registry.register({
|
|
17
|
+
manifest: {
|
|
18
|
+
id: 'example-analyzer', version: '1.0.0', languages: ['example'], frameworks: [],
|
|
19
|
+
capabilities: ['entities', 'relations'], knowledgeSchemaVersion: 1,
|
|
20
|
+
compatibility: { pipelineMajor: 1 }, unsupportedConstructs: [], resourceLimits: { maxFiles: 10_000 },
|
|
21
|
+
},
|
|
22
|
+
analyze: () => ({ entities: [], relations: [], coverage: [], diagnostics: [] }),
|
|
23
|
+
})
|
|
24
|
+
```
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Benchmark format v1
|
|
3
|
+
description: Reproducible, anonymization-safe semantic and agent-efficiency measurements.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Benchmark format v1
|
|
7
|
+
|
|
8
|
+
`ak-docs benchmark <fixture.json> <observation.json>` compares an observed run with a versioned truth fixture. The command emits JSON by default and a compact human-readable summary with `--text`.
|
|
9
|
+
|
|
10
|
+
The fixture contains only stable identifiers and explicitly supported cases:
|
|
11
|
+
|
|
12
|
+
```json
|
|
13
|
+
{
|
|
14
|
+
"schemaVersion": 1,
|
|
15
|
+
"supported": {
|
|
16
|
+
"entities": ["package:fixture"],
|
|
17
|
+
"relations": ["package:fixture->module:src/index.ts"],
|
|
18
|
+
"findings": ["undocumented-relation"]
|
|
19
|
+
},
|
|
20
|
+
"excluded": {
|
|
21
|
+
"entities": ["generated:fixture"],
|
|
22
|
+
"relations": [],
|
|
23
|
+
"findings": ["ambiguous:dynamic-loading"]
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
The observation contains the same three sets, plus optional evidence identifiers and finding-category counts. The result reports true positives, false positives, false negatives, precision, recall, duplicate observations, evidence ratio, finding density, excluded cases, thresholds, and regressions. Excluded cases are removed from denominators only because they are explicitly listed and their counts remain visible.
|
|
29
|
+
|
|
30
|
+
Benchmark output is aggregate by default: it does not include repository contents, prompts, credentials, or the member lists used to calculate the result. Baselines are not changed by `benchmark` or by a verification run. Use the verification harness's explicit audited baseline command when a new baseline is intentionally approved.
|
package/docs/spec/config-v1.md
CHANGED
|
@@ -7,6 +7,10 @@ description: Configure documentation corpora, ownership routing, conformance, an
|
|
|
7
7
|
|
|
8
8
|
`doc-bridge.config.ts` (or `.js`, `.mjs`, `.json`, or `package.json` → `docBridge`) is the alpha integration point for any project. Layer 0 fields are sufficient to run `index`, `query`, and MCP without an LLM.
|
|
9
9
|
|
|
10
|
+
Reconciliation summaries also expose deterministic `diagnosticsByCode` and
|
|
11
|
+
`diagnosticsByStatus` maps. They are additive rollups for agents and dashboards;
|
|
12
|
+
the canonical `diagnostics` array remains the source of evidence.
|
|
13
|
+
|
|
10
14
|
| npm package | `@agentskit/doc-bridge` |
|
|
11
15
|
| CLI binary | `ak-docs` |
|
|
12
16
|
| Config file | `doc-bridge.config.ts` |
|
|
@@ -71,6 +75,18 @@ export default {
|
|
|
71
75
|
|
|
72
76
|
/** Optional deterministic documentation conformance profiles */
|
|
73
77
|
conformance?: ConformanceConfig
|
|
78
|
+
|
|
79
|
+
/** Optional language analyzers and runtime-wiring coverage */
|
|
80
|
+
analysis?: AnalysisConfig
|
|
81
|
+
|
|
82
|
+
/** Optional reconciliation scope and orphan-document policy */
|
|
83
|
+
reconciliation?: ReconciliationConfig
|
|
84
|
+
|
|
85
|
+
/** Optional resumable workflow state */
|
|
86
|
+
workflow?: WorkflowConfig
|
|
87
|
+
|
|
88
|
+
/** Optional report publication privacy; private is the default */
|
|
89
|
+
report?: { privacy?: 'private' | 'anonymized' }
|
|
74
90
|
} satisfies DocBridgeConfigV1
|
|
75
91
|
```
|
|
76
92
|
|
|
@@ -386,6 +402,83 @@ report status, commands, and the recorded stable-publication HITL decision.
|
|
|
386
402
|
|
|
387
403
|
---
|
|
388
404
|
|
|
405
|
+
## `reconciliation` (optional)
|
|
406
|
+
|
|
407
|
+
```ts
|
|
408
|
+
type ReconciliationConfig = {
|
|
409
|
+
/** Semantic comparison level; discovery still preserves raw file relations. */
|
|
410
|
+
scope?: 'file' | 'module' | 'package'
|
|
411
|
+
/** Observed relation kinds that require documentation declarations. */
|
|
412
|
+
requiredRelationKinds?: string[]
|
|
413
|
+
/** Emit info findings for documentation with no observed package/module join. */
|
|
414
|
+
includeOrphanedDocuments?: boolean
|
|
415
|
+
}
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
Use `scope: 'package'` for monorepos where file imports should be compared as package-level architecture evidence. Omit `requiredRelationKinds` to require all observed kinds; an empty array intentionally disables undocumented-relation findings and must be treated as an explicit exemption.
|
|
419
|
+
|
|
420
|
+
The reconciliation documentation summary reports package health separately from
|
|
421
|
+
relation findings: `fresh` means the package has coverage documentation and no
|
|
422
|
+
known discrepancy; `stale` means a declared relation conflicts with observed
|
|
423
|
+
architecture; `missing` means no coverage document was found; and `unverified`
|
|
424
|
+
means coverage exists but at least one relevant relation or analyzer boundary
|
|
425
|
+
could not be verified. Package-level aggregation preserves the relation
|
|
426
|
+
endpoints used for this classification, so an undocumented relation cannot be
|
|
427
|
+
reported alongside a falsely `fresh` package.
|
|
428
|
+
|
|
429
|
+
## `report` (optional)
|
|
430
|
+
|
|
431
|
+
```ts
|
|
432
|
+
report?: {
|
|
433
|
+
/** Replace project identity, names, paths, snippets, and finding text in HTML output. */
|
|
434
|
+
privacy?: 'private' | 'anonymized'
|
|
435
|
+
}
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
`private` is the default and keeps local evidence useful for debugging. `anonymized` is intended for reports shared outside the repository: it preserves counts, relation kinds, topology, and coverage status while removing project-specific identity and evidence content. The generated HTML and every lazy chunk use the same mode.
|
|
439
|
+
|
|
440
|
+
## `analysis` (optional)
|
|
441
|
+
|
|
442
|
+
```ts
|
|
443
|
+
type AnalysisConfig = {
|
|
444
|
+
/** Ordered language/framework analyzer plugins. */
|
|
445
|
+
plugins?: Array<{
|
|
446
|
+
id: string
|
|
447
|
+
enabled?: boolean
|
|
448
|
+
order?: number
|
|
449
|
+
options?: Record<string, unknown>
|
|
450
|
+
reason?: string
|
|
451
|
+
}>
|
|
452
|
+
jsTs?: {
|
|
453
|
+
/** Property-access methods considered runtime wiring entry points. */
|
|
454
|
+
runtimeWiringMethods?: string[]
|
|
455
|
+
/** Additional adapter methods that represent runtime wiring. */
|
|
456
|
+
runtimeWiringAdapters?: Array<{ id: string; methods: string[] }>
|
|
457
|
+
/** Include test/spec modules in runtime-wiring coverage. Default: false. */
|
|
458
|
+
includeTestRuntimeWiring?: boolean
|
|
459
|
+
}
|
|
460
|
+
}
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
The JS/TS analyzer defaults to `register`, `use`, `mount`, and `attach`.
|
|
464
|
+
Generic APIs such as `bind` and `listen` are intentionally opt-in to avoid
|
|
465
|
+
classifying ordinary function binding and server startup as architecture.
|
|
466
|
+
When a configured method receives an identifier bound to a
|
|
467
|
+
static import, Doc Bridge records a `runtime-wiring` relation with
|
|
468
|
+
`metadata.detection: 'runtime-wiring-static'`. Reflective, computed, or
|
|
469
|
+
otherwise unbound targets remain explicit `not-analyzed` coverage entries.
|
|
470
|
+
Test/spec modules are excluded from this signal by default because their
|
|
471
|
+
registrations usually construct fixtures rather than production architecture;
|
|
472
|
+
set `includeTestRuntimeWiring: true` when test wiring is part of the contract.
|
|
473
|
+
This keeps runtime behavior useful without claiming that arbitrary dependency
|
|
474
|
+
injection or reflection was resolved. Analyzer plugins must implement the
|
|
475
|
+
versioned contract in `docs/spec/analyzer-plugin-v1.md`; malformed or
|
|
476
|
+
unsupported output remains explicit `not-analyzed` evidence.
|
|
477
|
+
|
|
478
|
+
`workflow.stateDir` optionally relocates the content-addressed workflow
|
|
479
|
+
artifacts. Runs are resumable and idempotent; pipeline and analyzer versions
|
|
480
|
+
are part of the run identity.
|
|
481
|
+
|
|
389
482
|
## `surfaces` (optional)
|
|
390
483
|
|
|
391
484
|
```ts
|
|
@@ -466,6 +559,18 @@ type IntelligenceConfig = {
|
|
|
466
559
|
/** Reference runtime; custom path for non-AgentsKit engines later */
|
|
467
560
|
runtime?: 'agentskit' | 'custom'
|
|
468
561
|
runtimeModule?: string
|
|
562
|
+
|
|
563
|
+
registry?: {
|
|
564
|
+
enabled?: boolean
|
|
565
|
+
agentId?: string
|
|
566
|
+
agentRoot?: string
|
|
567
|
+
runnerModule?: string
|
|
568
|
+
deterministic?: boolean
|
|
569
|
+
timeoutMs?: number
|
|
570
|
+
maxTokens?: number
|
|
571
|
+
maxResponseBytes?: number
|
|
572
|
+
maxConcurrency?: number
|
|
573
|
+
}
|
|
469
574
|
}
|
|
470
575
|
|
|
471
576
|
type MemoryAdapterId =
|
|
@@ -475,6 +580,12 @@ type MemoryAdapterId =
|
|
|
475
580
|
| 'bootstrap-delta' // git diff on AGENTS.md
|
|
476
581
|
```
|
|
477
582
|
|
|
583
|
+
Registry agents are advisory and must come from the AgentsKit Registry. The
|
|
584
|
+
adapter redacts secrets, validates the proposal contract, bounds execution and
|
|
585
|
+
response size, and requires human approval before any change is applied. The
|
|
586
|
+
default agent is `ecosystem-doc-bridge-corpus-scanner`; compatible Registry
|
|
587
|
+
agents may be selected explicitly.
|
|
588
|
+
|
|
478
589
|
---
|
|
479
590
|
|
|
480
591
|
## `federation` (optional — ecosystem)
|