@agentskit/doc-bridge 1.7.44 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (223) hide show
  1. package/CHANGELOG.md +471 -0
  2. package/CONTRIBUTING.md +29 -4
  3. package/README.md +87 -40
  4. package/SECURITY.md +7 -0
  5. package/action.yml +1 -1
  6. package/bin/ak-docs.js +2 -2
  7. package/bin/ak-verify.js +13 -7
  8. package/dist/cli/program.d.ts +3 -1
  9. package/dist/cli/program.js +15888 -6061
  10. package/dist/cli/program.js.map +1 -1
  11. package/dist/config/index.d.ts +1 -1
  12. package/dist/config/index.js +91 -9
  13. package/dist/config/index.js.map +1 -1
  14. package/dist/index-Beor6Yhi.d.ts +792 -0
  15. package/dist/index.d.ts +9979 -3257
  16. package/dist/index.js +15954 -5774
  17. package/dist/index.js.map +1 -1
  18. package/docs/MARKETPLACE.md +1 -1
  19. package/docs/PRD-documentation-efficiency-study.md +406 -0
  20. package/docs/PRD-knowledge-retrieval-and-enrichment.md +466 -0
  21. package/docs/RELEASE.md +22 -8
  22. package/docs/adr/0002-documentation-audit-boundary.md +22 -0
  23. package/docs/adr/0003-study-protocol-and-historical-evidence.md +40 -0
  24. package/docs/adr/0004-controlled-study-runner.md +25 -0
  25. package/docs/adr/0005-documentation-quality-and-criticality.md +20 -0
  26. package/docs/adr/0006-registry-semantic-grounding.md +20 -0
  27. package/docs/adr/0007-longitudinal-study-metrics.md +21 -0
  28. package/docs/adr/0008-study-verification-boundary.md +21 -0
  29. package/docs/adr/0009-study-provider-cli-adapter.md +25 -0
  30. package/docs/agent-corpus/INDEX.md +14 -3
  31. package/docs/agent-corpus/OVERVIEW.md +25 -0
  32. package/docs/agent-corpus/chat.md +7 -3
  33. package/docs/agent-corpus/cli.md +18 -2
  34. package/docs/agent-corpus/conformance.md +14 -2
  35. package/docs/agent-corpus/doc-bridge.md +48 -1
  36. package/docs/agent-corpus/doctor.md +10 -2
  37. package/docs/agent-corpus/gates.md +6 -2
  38. package/docs/agent-corpus/mcp.md +15 -2
  39. package/docs/agent-corpus/memory.md +6 -2
  40. package/docs/agent-corpus/query.md +35 -2
  41. package/docs/bench/README.md +122 -0
  42. package/docs/bench/retrieval-baseline-v1.json +28 -0
  43. package/docs/bench/retrieval-suite-v1.json +1033 -0
  44. package/docs/chat-and-rag.md +3 -2
  45. package/docs/for-agents.md +9 -1
  46. package/docs/getting-started.md +4 -11
  47. package/docs/guides/gate-ci.md +11 -1
  48. package/docs/guides/install-and-run.md +9 -65
  49. package/docs/index.md +22 -1
  50. package/docs/knowledge-engine-runbook.md +51 -4
  51. package/docs/landing/assets/context-payload-reduction.svg +21 -0
  52. package/docs/landing/assets/controlled-ab-comparison.svg +30 -0
  53. package/docs/landing/index.html +119 -5
  54. package/docs/loop-workflow.md +117 -0
  55. package/docs/mcp.md +6 -1
  56. package/docs/parity/public-claims-v1.json +145 -0
  57. package/docs/playbook/doc-bridge-pattern.md +1 -1
  58. package/docs/query.md +90 -2
  59. package/docs/recipes/index-pipeline.md +1 -1
  60. package/docs/schemas/agent-handoff-v1.md +15 -0
  61. package/docs/schemas/doc-bridge-index-v1.md +65 -0
  62. package/docs/spec/benchmark-v1.md +39 -1
  63. package/docs/spec/cli.md +30 -10
  64. package/docs/spec/config-v1.md +192 -8
  65. package/docs/spec/documentation-audit-v1.md +61 -0
  66. package/docs/spec/enrichment-overlay-v1.md +241 -0
  67. package/docs/spec/graph-signals-v1.md +92 -0
  68. package/docs/spec/incremental-scan-v1.md +102 -0
  69. package/docs/spec/markdown-analyzer-v1.md +73 -0
  70. package/docs/spec/mcp-knowledge-tools-v1.md +147 -0
  71. package/docs/spec/measured-enrichment-v1.md +229 -0
  72. package/docs/spec/public-parity-v1.md +119 -0
  73. package/docs/spec/registry-agents.md +6 -0
  74. package/docs/spec/render-v1.md +122 -0
  75. package/docs/spec/retrieval-index-v1.md +164 -0
  76. package/docs/spec/study-metrics-v1.md +58 -0
  77. package/docs/spec/study-protocol-v1.md +46 -0
  78. package/docs/spec/study-provider-cli-v1.md +116 -0
  79. package/docs/spec/study-runner-v1.md +35 -0
  80. package/docs/spec/study-task-suite-v1.md +41 -0
  81. package/docs/spec/study-verification-v1.md +40 -0
  82. package/docs/study/README.md +84 -0
  83. package/docs/study/ab-adjudicated-cost-analysis-v1.md +29 -0
  84. package/docs/study/ab-adjudicated-cost-plan-v1.json +33 -0
  85. package/docs/study/ab-adjudicated-cost-plan-v2-v1.json +33 -0
  86. package/docs/study/ab-adjudicated-cost-result-v1.json +80 -0
  87. package/docs/study/ab-baseline-analysis-v1.md +21 -0
  88. package/docs/study/ab-baseline-plan-v1.json +33 -0
  89. package/docs/study/ab-baseline-recovery-plan-v1.json +33 -0
  90. package/docs/study/ab-baseline-result-v1.json +79 -0
  91. package/docs/study/documentation-audit-round-2026-08-31.json +183 -0
  92. package/docs/study/historical-evidence-v1.json +252 -0
  93. package/docs/study/observation-ledger-v1.json +30632 -0
  94. package/docs/study/phase3-task-coverage-v1.json +34 -0
  95. package/docs/study/phase4-public-pilot-ledger-v1.json +1344 -0
  96. package/docs/study/phase4-public-pilot-result-v1.json +52 -0
  97. package/docs/study/phase4-public-pilot-run-plan-v1.json +26 -0
  98. package/docs/study/phase4-public-pilot-task-suite-v1.json +71 -0
  99. package/docs/study/pilot-round-2026-08-31.json +46 -0
  100. package/docs/study/protocol-v1.json +90 -0
  101. package/docs/study/publication-gate-v1.md +45 -0
  102. package/docs/study/quality-scorecard-cycle-plan.md +545 -0
  103. package/docs/study/quality-scorecard-v1.json +38 -0
  104. package/docs/study/round-1-adjudicated-smoke-v1.json +30642 -0
  105. package/docs/study/round-1-instrumentation-plan-v1.md +39 -0
  106. package/docs/study/round-2-expanded-adjudication-v1.json +91 -0
  107. package/docs/study/round-2-expanded-validation-v1.md +58 -0
  108. package/docs/study/round-3-evidence-contract-v1.json +75 -0
  109. package/docs/study/round-3-evidence-contract-v1.md +57 -0
  110. package/docs/study/round-4-confirmation-v1.json +75 -0
  111. package/docs/study/round-4-confirmation-v1.md +55 -0
  112. package/docs/study/run-plan-v1.json +33 -0
  113. package/docs/study/semantic-adjudication-cycle-8.md +20 -0
  114. package/docs/study/task-suite-v1.json +96 -0
  115. package/docs/study/token-efficiency-plan-v1.md +337 -0
  116. package/docs/study/token-efficiency-protocol-v2.json +62 -0
  117. package/docs/study/verification-binding-v1.json +27 -0
  118. package/docs/validation-cycle-plan.md +33 -0
  119. package/docs/verification-harness.md +15 -6
  120. package/ecosystem-claims.json +2 -2
  121. package/ecosystem-upstream.json +2 -2
  122. package/ecosystem.json +4 -4
  123. package/mcpb/manifest.json +9 -1
  124. package/package.json +89 -72
  125. package/scripts/check-ecosystem-upstream.mjs +36 -7
  126. package/scripts/report-visual-check.mjs +20 -3
  127. package/skills/doc-bridge-handoff/fixtures/synthetic-repo/docs/for-agents/packages/payments.md +7 -0
  128. package/skills/doc-bridge-handoff/scripts/resolve-handoff.mjs +1 -1
  129. package/src/agents/registry-adapter.ts +192 -24
  130. package/src/audit/documentation.ts +513 -0
  131. package/src/bench/baseline.ts +198 -0
  132. package/src/bench/overlay-delta.ts +139 -0
  133. package/src/bench/retrieval.ts +319 -0
  134. package/src/budget/compile.ts +91 -0
  135. package/src/budget/sections.ts +70 -0
  136. package/src/cli/demo.ts +2 -2
  137. package/src/cli/program.ts +699 -79
  138. package/src/cli/usage.ts +71 -0
  139. package/src/config/defaults.ts +1 -0
  140. package/src/config/index.ts +4 -0
  141. package/src/config/load-config.ts +7 -1
  142. package/src/config/schema.ts +121 -4
  143. package/src/conformance/documentation-standard-v1.ts +22 -14
  144. package/src/discovery/areas.ts +182 -0
  145. package/src/discovery/documentation.ts +255 -23
  146. package/src/discovery/identity.ts +24 -0
  147. package/src/discovery/incremental.ts +314 -0
  148. package/src/discovery/inputs.ts +110 -0
  149. package/src/discovery/markdown.ts +481 -0
  150. package/src/discovery/repository.ts +557 -125
  151. package/src/doctor/run-doctor.ts +246 -27
  152. package/src/enrich/approvals.ts +190 -0
  153. package/src/enrich/cache.ts +93 -0
  154. package/src/enrich/context-pack.ts +272 -0
  155. package/src/enrich/overlay.ts +255 -0
  156. package/src/enrich/review.ts +106 -0
  157. package/src/enrich/stage.ts +374 -0
  158. package/src/enrich/stats.ts +100 -0
  159. package/src/enrich/validate.ts +410 -0
  160. package/src/federation/llms.ts +74 -24
  161. package/src/findings/report.ts +103 -0
  162. package/src/fixes/proposals.ts +4 -3
  163. package/src/graph/build.ts +356 -0
  164. package/src/graph/memory.ts +208 -0
  165. package/src/index-builder/build-handoffs.ts +22 -11
  166. package/src/index-builder/build-index.ts +132 -3
  167. package/src/index-builder/human-adapters/fumadocs.ts +1 -1
  168. package/src/index-builder/llms-txt.ts +48 -8
  169. package/src/index-builder/project-corpus.ts +111 -0
  170. package/src/index-builder/watch-index.ts +1 -1
  171. package/src/index.ts +630 -2
  172. package/src/lib/bounded-text.ts +15 -10
  173. package/src/lib/fuzzy-match.ts +235 -0
  174. package/src/mcp/knowledge.ts +554 -0
  175. package/src/mcp/server.ts +113 -18
  176. package/src/metrics/benchmark.ts +21 -0
  177. package/src/parity/check.ts +309 -0
  178. package/src/parity/claims.ts +259 -0
  179. package/src/parity/resolve.ts +160 -0
  180. package/src/query/handoff.ts +326 -0
  181. package/src/query/load-index.ts +53 -1
  182. package/src/query/query.ts +92 -59
  183. package/src/query/search.ts +289 -92
  184. package/src/query/text.ts +155 -0
  185. package/src/reconciliation/reconcile.ts +148 -15
  186. package/src/render/data.ts +356 -0
  187. package/src/render/engine.ts +398 -0
  188. package/src/render/generated.ts +77 -0
  189. package/src/render/render.ts +209 -0
  190. package/src/render/template-source.ts +52 -0
  191. package/src/render/templates.ts +289 -0
  192. package/src/report/html.ts +23 -17
  193. package/src/retrieval/bm25.ts +161 -0
  194. package/src/retrieval/project.ts +495 -0
  195. package/src/retrieval/rank.ts +383 -0
  196. package/src/retrieval/weights.ts +39 -0
  197. package/src/retriever/doc-bridge-retriever.ts +100 -15
  198. package/src/rules/engine.ts +45 -12
  199. package/src/safety/repository.ts +1 -1
  200. package/src/schemas/agent-handoff.ts +56 -0
  201. package/src/schemas/budget.ts +37 -0
  202. package/src/schemas/doc-bridge-index.ts +53 -2
  203. package/src/schemas/enrichment.ts +369 -0
  204. package/src/schemas/json-schemas.ts +39 -2
  205. package/src/schemas/knowledge.ts +19 -3
  206. package/src/schemas/retrieval-index.ts +152 -0
  207. package/src/shims/graphology.d.ts +91 -0
  208. package/src/study/adjudication.ts +196 -0
  209. package/src/study/execution.ts +350 -0
  210. package/src/study/expectations.ts +219 -0
  211. package/src/study/metrics.ts +467 -0
  212. package/src/study/protocol.ts +271 -0
  213. package/src/study/provider-cli.ts +115 -0
  214. package/src/study/provider-telemetry.ts +47 -0
  215. package/src/study/quality-scorecard.ts +164 -0
  216. package/src/study/runner.ts +461 -0
  217. package/src/study/task-suite.ts +321 -0
  218. package/src/study/verification.ts +134 -0
  219. package/src/validate.ts +8 -5
  220. package/src/version.ts +1 -1
  221. package/src/workflow/engine.ts +36 -11
  222. package/dist/index-C2PCQSrB.d.ts +0 -2251
  223. package/scripts/verification-harness.mjs +0 -483
@@ -0,0 +1,39 @@
1
+ ---
2
+ title: Round 1 instrumentation plan v1
3
+ description: Acceptance contract for configurable study cost attribution and independent adjudication.
4
+ ---
5
+
6
+ # Round 1 instrumentation plan v1
7
+
8
+ ## Objective
9
+
10
+ Make controlled study results auditable without bundling a local model runtime: record configured USD attribution when usage is available and support a separate adjudicator process that cannot approve its own provider output.
11
+
12
+ ## Acceptance criteria
13
+
14
+ | ID | Criterion | Evidence |
15
+ | --- | --- | --- |
16
+ | R1-C1 | Provider and adjudicator rates are schema-validated, content-addressed, and use an explicit USD formula. | Focused provider CLI test and current configuration hash. |
17
+ | R1-C2 | The runner records `agentCostUsd` only for labeled provider usage with configured rates; missing data remains missing. | Runner test and metrics output. |
18
+ | R1-C3 | A separate bounded CLI can adjudicate a ledger using only anonymized structured candidate data. | Independent adjudicator test and CLI smoke run. |
19
+ | R1-C4 | Adjudicator timeout, invalid output, unavailable command, and output-limit failures remain `pending`. | Focused failure-path test. |
20
+ | R1-C5 | Token-equivalent and USD measurements remain distinct and auditable by source, run ID, configuration hash, and content hashes. | Ledger schema, metrics report, and verification harness evidence. |
21
+ | R1-C6 | The smoke run against a private consumer repository produces no publication-bound private path, repository content, prompt, or raw response. | Deterministic privacy gate and anonymized output inspection. |
22
+
23
+ ## Deliberate boundary
24
+
25
+ Round 1 adjudicates the bounded structured candidate record, not raw model prose or private repository contents. This proves the independent process, privacy boundary, cost accounting, and failure behavior. Semantic review of documentation quality and code/documentation contradictions remains a later round because it requires richer evidence and a separately approved evaluation protocol.
26
+
27
+ ## Measurement fields
28
+
29
+ - `agentCostUsd`: configured-rate USD estimate for the evaluated provider.
30
+ - `adjudicatorCostUsd`: configured-rate USD estimate for the independent adjudicator.
31
+ - `providerTokenCostUnits` and `adjudicatorTokenCostUnits`: provider-reported input plus output tokens.
32
+ - `adjudicatorLatencyMs`: elapsed adjudicator process time.
33
+ - `adjudication.status`, `actor`, `method`, `outcome`, `confidence`, and `reasonCodes`: independent decision provenance.
34
+
35
+ Rates are experimental configuration inputs, not vendor billing claims. Every result must retain the config hash and run identifier used to produce it.
36
+
37
+ ## Recorded smoke evidence
38
+
39
+ The private-consumer smoke processed one observation with `--limit 1` and kept the remaining ledger observations unchanged. The anonymized output ledger hash is `3f2c80e6768ffd33a14f38ca8fb9eeb482f4722c9f028ec29adc7d09d54aaef6`; the configuration hash is `7ea4b4496d23c15f93527120927ec46105d8fc46c66114e0c710426f5f417ea8`. The independent adjudicator recorded 150 token-cost units, USD 0.00034 at the configured study rates, and 49 ms latency. The privacy gate scanned the output and found zero forbidden matches.
@@ -0,0 +1,91 @@
1
+ {
2
+ "type": "expanded-adjudication-result",
3
+ "schemaVersion": 1,
4
+ "round": "round-2-expanded-adjudication-2026-08-31",
5
+ "sourceLedgerHash": "cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5",
6
+ "outputLedgerHash": "a89fa316b88eea707347c1b2caae64c6895ce19688a57a7909e15abb51058f20",
7
+ "configurationHash": "d816148be351fd8dacdf82f204d6694b41e3f3af90ce5f8cae11ced8c99ef37b",
8
+ "sampleSize": 12,
9
+ "independentAdjudicationCount": 12,
10
+ "pendingCount": 0,
11
+ "coverage": 1,
12
+ "providerNetwork": true,
13
+ "adjudicatorModelId": "gpt-5.6-luna",
14
+ "scenarioCoverage": [
15
+ "deterministic-doc-bridge",
16
+ "repository-only"
17
+ ],
18
+ "categoryCoverage": [
19
+ "architecture",
20
+ "discovery",
21
+ "documentation",
22
+ "implementation"
23
+ ],
24
+ "repositoryCoverage": [
25
+ "consumer-01",
26
+ "consumer-03",
27
+ "consumer-04",
28
+ "consumer-05",
29
+ "consumer-06"
30
+ ],
31
+ "inputTokens": 226577,
32
+ "outputTokens": 3297,
33
+ "tokenCostUnits": 229874,
34
+ "configuredCostUsd": 0.302758,
35
+ "latencyMs": {
36
+ "min": 8078,
37
+ "max": 11503,
38
+ "average": 9841.5,
39
+ "p95": 11503
40
+ },
41
+ "outcomeCounts": {
42
+ "incomplete": 2,
43
+ "partial": 5,
44
+ "blocked": 5
45
+ },
46
+ "reasonCodeCounts": {
47
+ "acceptance-check-blocked": 1,
48
+ "acceptance-check-failed": 1,
49
+ "acceptance-check-not-established": 1,
50
+ "acceptance-check-not-executed": 1,
51
+ "acceptance-check-not-passed": 2,
52
+ "acceptance-check-unavailable": 1,
53
+ "architecture-artifact-not-inspected": 1,
54
+ "budget-exceeded-does-not-establish-blocked-target": 1,
55
+ "candidate-outcome-misclassified": 1,
56
+ "candidate-record-omits-discovery-map": 1,
57
+ "correct-target-and-evidence": 1,
58
+ "discovery-evidence-not-validated": 3,
59
+ "documentation-only-scope": 1,
60
+ "entrypoint-evidence-present": 1,
61
+ "entrypoints-and-owners-not-stated": 1,
62
+ "evidence-content-not-provided": 1,
63
+ "evidence-identifiers-alone-do-not-prove-correctness": 1,
64
+ "execution-budget-exceeded": 1,
65
+ "issue-category-not-provided": 1,
66
+ "knowledge-gap-evidence-present": 3,
67
+ "minimal-documentation-only-proposal": 1,
68
+ "precise-documentation-issue-not-provided": 1,
69
+ "required-acceptance-check-blocked": 1,
70
+ "required-acceptance-check-failed": 2,
71
+ "required-evidence-insufficient": 1,
72
+ "success-not-supported": 1,
73
+ "target-document-identified": 3,
74
+ "target-document-not-identified": 1,
75
+ "verification-command-blocked": 1,
76
+ "verification-command-unavailable": 1,
77
+ "verification-plan-present": 3
78
+ },
79
+ "privacy": {
80
+ "forbiddenMatchCount": 0,
81
+ "rawRepositoryPathsIncluded": false,
82
+ "rawRepositoryContentIncluded": false
83
+ },
84
+ "limitations": [
85
+ "The sample is anonymized and bounded; it is not a full-repository evaluation.",
86
+ "The configured USD rates are experimental study rates, not a vendor invoice.",
87
+ "Independent adjudication evaluates the bounded candidate record and cannot recover evidence omitted from it."
88
+ ],
89
+ "contentHashAlgo": "sha256-normalized-v1",
90
+ "contentHash": "df4a869c9da7354c7fdd6a6ed993f67f9fb16ccb8c2953b8c937328e248701ce"
91
+ }
@@ -0,0 +1,58 @@
1
+ ---
2
+ title: Round 2 expanded validation v1
3
+ description: Auditable independent adjudication results for an anonymized study sample.
4
+ ---
5
+
6
+ # Round 2 expanded validation v1
7
+
8
+ ## Objective
9
+
10
+ Measure whether a real, configurable Codex CLI can independently evaluate bounded study observations while preserving provenance, privacy, token accounting, cost attribution, and latency measurements.
11
+
12
+ ## Contract and sample
13
+
14
+ The run selected 12 observations from the anonymized `phase-9-ab-adjudicated-cost-03` run. The sample covers all four task categories (`architecture`, `discovery`, `documentation`, and `implementation`), both scenarios (`repository-only` and `deterministic-doc-bridge`), and five anonymized consumer identifiers. The adjudicator was the real Codex CLI running `gpt-5.6-luna`, separated from the provider execution being evaluated.
15
+
16
+ The adapter sent only bounded task metadata and candidate measurements. It did not send repository paths, raw repository content, prompts, credentials, or raw provider responses.
17
+
18
+ ## Results
19
+
20
+ | Metric | Result |
21
+ | --- | ---: |
22
+ | Sample size | 12 |
23
+ | Independent adjudications | 12 |
24
+ | Adjudication coverage | 100% |
25
+ | Pending adjudications | 0 |
26
+ | Input tokens | 226,577 |
27
+ | Output tokens | 3,297 |
28
+ | Token cost units | 229,874 |
29
+ | Configured study cost | USD 0.302758 |
30
+ | Mean latency | 9,841.5 ms |
31
+ | P95 latency | 11,503 ms |
32
+
33
+ The independent outcomes were 2 `incomplete`, 5 `partial`, and 5 `blocked`; no observation was promoted to `success`. The dominant signals were missing or unvalidated discovery evidence, unavailable or unexecuted acceptance checks, and insufficiently specific documentation evidence. These are actionable study findings, not a claim that the underlying repositories are defective in every respect.
34
+
35
+ ## Acceptance evidence
36
+
37
+ | Criterion | Result | Evidence |
38
+ | --- | --- | --- |
39
+ | Real configurable adjudicator executed | Validated | Output ledger hash `a89fa316b88eea707347c1b2caae64c6895ce19688a57a7909e15abb51058f20`; 12 automated decisions. |
40
+ | Coverage and category/scenario representation measured | Validated | `round-2-expanded-adjudication-v1.json`. |
41
+ | Token, latency, and configured cost measured | Validated | Per-observation adjudicator measurements and aggregate summary. |
42
+ | Privacy boundary preserved | Validated | `forbiddenMatchCount: 0`; raw paths and content excluded. |
43
+ | Result reproducibility bound to inputs | Validated | Source ledger hash, output ledger hash, configuration hash, and content hash are recorded. |
44
+ | Full-repository semantic quality established | Not analyzed | This bounded sample evaluates candidate records; it does not replace a full documentation/code comparison. |
45
+
46
+ ## Provenance
47
+
48
+ - Source ledger: `cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5`.
49
+ - Adjudicator configuration: `d816148be351fd8dacdf82f204d6694b41e3f3af90ce5f8cae11ced8c99ef37b`.
50
+ - Summary artifact: `df4a869c9da7354c7fdd6a6ed993f67f9fb16ccb8c2953b8c937328e248701ce`.
51
+
52
+ ## Limitations
53
+
54
+ - The sample is anonymized and bounded; it is not a full-repository evaluation.
55
+ - The configured USD rates are experimental study rates, not a vendor invoice.
56
+ - The adjudicator can evaluate only evidence present in the bounded candidate record and cannot recover omitted evidence.
57
+
58
+ The initial batch attempt exposed an invalid strict JSON schema and produced only `pending` results. The schema was corrected to require nullable usage fields, a one-observation CLI smoke passed, and the controlled 12-observation run was repeated successfully. The failed attempt is retained only as an operational limitation, not as quality evidence.
@@ -0,0 +1,75 @@
1
+ {
2
+ "type": "expanded-adjudication-result",
3
+ "schemaVersion": 1,
4
+ "round": "round-3-evidence-contract-2026-08-31",
5
+ "sourceLedgerHash": "cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5",
6
+ "outputLedgerHash": "3a5191e9bb839da91c7bf8bcfe58ba77e718a381c27e8867243fc7c859167eee",
7
+ "configurationHash": "1735f10937845e1db35873c27fd38700e8c5a93bbd412884df5bd0b550628ce5",
8
+ "sampleSize": 12,
9
+ "independentAdjudicationCount": 12,
10
+ "pendingCount": 0,
11
+ "coverage": 1,
12
+ "providerNetwork": true,
13
+ "adjudicatorModelId": "gpt-5.6-luna",
14
+ "scenarioCoverage": ["deterministic-doc-bridge", "repository-only"],
15
+ "categoryCoverage": ["architecture", "discovery", "documentation", "implementation"],
16
+ "repositoryCoverage": ["consumer-01", "consumer-03", "consumer-04", "consumer-05", "consumer-06"],
17
+ "inputTokens": 227535,
18
+ "outputTokens": 3029,
19
+ "tokenCostUnits": 230564,
20
+ "configuredCostUsd": 0.330994,
21
+ "latencyMs": {"min": 7360, "max": 11643, "average": 9498.08, "p95": 11643},
22
+ "outcomeCounts": {"success": 1, "incomplete": 2, "blocked": 5, "partial": 4},
23
+ "reasonCodeCounts": {
24
+ "acceptance-budget-exceeded": 1,
25
+ "acceptance-check-failed": 5,
26
+ "acceptance-check-not-passed": 1,
27
+ "acceptance-check-passed": 1,
28
+ "acceptance-check-unexecuted": 4,
29
+ "ak-docs-command-unavailable": 1,
30
+ "architecture-artifact-unavailable": 1,
31
+ "candidate-issue-not-provided": 1,
32
+ "command-not-found": 1,
33
+ "discovery-check-not-passed": 1,
34
+ "evidence-present-but-validation-failed": 1,
35
+ "evidence-quality-high": 1,
36
+ "missing-documentation-evidence": 1,
37
+ "missing-review-limitation": 1,
38
+ "no-basis-for-success": 1,
39
+ "not-blocked": 1,
40
+ "patch-evidence-missing": 1,
41
+ "proposal-bounded-and-safe": 1,
42
+ "required-evidence-missing": 4,
43
+ "required-evidence-present": 3,
44
+ "required-patch-evidence-missing": 1,
45
+ "required-verification-plan-evidence-missing": 1,
46
+ "target-and-gap-identified": 1,
47
+ "target-document-not-safely-identified": 2,
48
+ "verification-blocked-by-environment": 1,
49
+ "verification-check-unexecuted": 1,
50
+ "verification-plan-evidence-missing": 1,
51
+ "verification-plan-present-but-runner-evidence-missing": 1
52
+ },
53
+ "comparisonToRound2": {
54
+ "coverageDelta": 0,
55
+ "inputTokensDelta": 958,
56
+ "outputTokensDelta": -268,
57
+ "tokenCostUnitsDelta": 690,
58
+ "configuredCostUsdDelta": 0.028236,
59
+ "latencyAverageDeltaMs": -343.42,
60
+ "latencyP95DeltaMs": 140,
61
+ "successCountDelta": 1,
62
+ "partialCountDelta": -1,
63
+ "incompleteCountDelta": 0,
64
+ "blockedCountDelta": 0
65
+ },
66
+ "privacy": {"forbiddenMatchCount": 0, "rawRepositoryPathsIncluded": false, "rawRepositoryContentIncluded": false},
67
+ "limitations": [
68
+ "The same anonymized bounded sample was re-adjudicated; this isolates contract effects but does not establish causality.",
69
+ "The configured USD rates are experimental study rates, not a vendor invoice.",
70
+ "A changed model response can affect outcomes; round-3 results are directional until replicated.",
71
+ "Independent adjudication evaluates the bounded candidate record and cannot recover evidence omitted from it."
72
+ ],
73
+ "contentHashAlgo": "sha256-normalized-v1",
74
+ "contentHash": "a75b0bdd23eeb0ba65d25e1ff0c9836922190f78a25f317cc38172266c586e32"
75
+ }
@@ -0,0 +1,57 @@
1
+ ---
2
+ title: Round 3 evidence contract validation v1
3
+ description: Before-and-after measurement of structured evidence coverage in independent study adjudication.
4
+ ---
5
+
6
+ # Round 3 evidence contract validation v1
7
+
8
+ ## Objective
9
+
10
+ Improve the study evidence contract so an independent adjudicator can distinguish missing evidence from unexecuted acceptance checks without receiving repository content or paths.
11
+
12
+ ## Change
13
+
14
+ The adjudicator input now includes a bounded `evidenceCoverage` object with presence for each required evidence identifier, a reference count, and an `acceptanceExecution` object with execution status, observed passed/total counts, and a measurement-presence flag. The provider contract also asks agents to execute declared acceptance checks when available and to emit typed coverage measurements. No raw repository data, credentials, prompts, or provider responses are included.
15
+
16
+ ## Results
17
+
18
+ The exact 12-observation anonymized sample from round 2 was re-adjudicated with the real Codex CLI (`gpt-5.6-luna`).
19
+
20
+ | Metric | Round 2 | Round 3 | Change |
21
+ | --- | ---: | ---: | ---: |
22
+ | Independent coverage | 100% | 100% | 0 pp |
23
+ | Success outcomes | 0 | 1 | +1 |
24
+ | Partial outcomes | 5 | 4 | -1 |
25
+ | Incomplete outcomes | 2 | 2 | 0 |
26
+ | Blocked outcomes | 5 | 5 | 0 |
27
+ | Adjudicator input tokens | 226,577 | 227,535 | +958 |
28
+ | Adjudicator output tokens | 3,297 | 3,029 | -268 |
29
+ | Token cost units | 229,874 | 230,564 | +690 |
30
+ | Configured study cost | USD 0.302758 | USD 0.330994 | +USD 0.028236 |
31
+ | Mean latency | 9,841.5 ms | 9,498.08 ms | -343.42 ms |
32
+ | P95 latency | 11,503 ms | 11,643 ms | +140 ms |
33
+
34
+ The new contract produced clearer reason codes, including `required-evidence-present`, `required-evidence-missing`, and `acceptance-check-unexecuted`. One observation was classified as `success`, but the sample remains inconclusive: five observations stayed blocked and the same sample was re-adjudicated, so model response variance is a confounder.
35
+
36
+ ## Acceptance evidence
37
+
38
+ | Criterion | Result | Evidence |
39
+ | --- | --- | --- |
40
+ | Evidence coverage is explicit and bounded | Validated | Round-3 adapter input contract and 12 automated adjudications. |
41
+ | Real Codex CLI executed with the new contract | Validated | Output ledger hash `3a5191e9bb839da91c7bf8bcfe58ba77e718a381c27e8867243fc7c859167eee`. |
42
+ | Before/after metrics are auditable | Validated | `round-3-evidence-contract-v1.json` and round-2 summary. |
43
+ | Privacy boundary preserved | Validated | Zero forbidden matches; raw paths and content excluded. |
44
+ | Full semantic documentation quality proven | Not analyzed | The bounded sample measures evidence sufficiency, not full-repository truth. |
45
+
46
+ ## Provenance
47
+
48
+ - Source ledger: `cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5`.
49
+ - Round-3 configuration: `1735f10937845e1db35873c27fd38700e8c5a93bbd412884df5bd0b550628ce5`.
50
+ - Summary artifact: `a75b0bdd23eeb0ba65d25e1ff0c9836922190f78a25f317cc38172266c586e32`.
51
+
52
+ ## Limitations
53
+
54
+ - This is a same-sample re-adjudication, not a causal A/B experiment.
55
+ - The configured USD rates are experimental study rates, not a vendor invoice.
56
+ - The adjudicator cannot recover evidence omitted from the candidate record.
57
+ - The next confirmatory round should use a fresh representative sample and test whether the improved contract generalizes.
@@ -0,0 +1,75 @@
1
+ {
2
+ "type": "expanded-adjudication-result",
3
+ "schemaVersion": 1,
4
+ "round": "round-4-confirmation-2026-08-31",
5
+ "sourceLedgerHash": "cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5",
6
+ "initialOutputLedgerHash": "1a394924244ba13fb1a0aefa73f81a245b4be3a37d3d9b0af270c498d7268622",
7
+ "recoveryOutputLedgerHash": "1a2464518ce659d8fdda176a5ab30ef4313fb571c65be20e2416a940bcbfa101",
8
+ "configurationHash": "80590115a04157a385a6535cdc84d2040955599301ff9ae3b2f5194bc64ca482",
9
+ "sampleSelection": {"runId": "phase-9-ab-adjudicated-cost-03", "offset": 12, "limit": 12},
10
+ "recoverySelection": {"offset": 17, "limit": 1},
11
+ "sampleSize": 12,
12
+ "independentAdjudicationCount": 12,
13
+ "pendingCount": 0,
14
+ "coverage": 1,
15
+ "providerNetwork": true,
16
+ "adjudicatorModelId": "gpt-5.6-luna",
17
+ "scenarioCoverage": ["deterministic-doc-bridge", "repository-only"],
18
+ "categoryCoverage": ["architecture", "discovery", "documentation", "implementation"],
19
+ "modelCoverage": ["low-cost-model", "reference-model"],
20
+ "repositoryCoverage": ["consumer-01", "consumer-02", "consumer-04", "consumer-05", "consumer-06"],
21
+ "inputTokens": 227799,
22
+ "outputTokens": 2416,
23
+ "tokenCostUnits": 230215,
24
+ "configuredCostUsd": 0.301934,
25
+ "latencyMs": {"min": 6840, "max": 12474, "average": 9672.67, "p95": 12474},
26
+ "outcomeCounts": {"incomplete": 3, "blocked": 8, "partial": 1},
27
+ "reasonCodeCounts": {
28
+ "acceptance-check-failed": 8,
29
+ "acceptance-check-result-unverified": 1,
30
+ "acceptance-check-timed-out": 1,
31
+ "acceptance-check-unexecuted": 2,
32
+ "architecture-artifact-unavailable": 1,
33
+ "architecture-evidence-present": 1,
34
+ "architecture-map-not-generated": 1,
35
+ "artifact-generation-blocked": 1,
36
+ "artifact-inspection-blocked-by-eprem": 1,
37
+ "attention-point-evidence-backed": 1,
38
+ "candidate-claims-success-without-evidence": 1,
39
+ "directional-relations-present": 1,
40
+ "discovery-not-validated": 1,
41
+ "discovery-unvalidated": 1,
42
+ "issue-not-precisely-classified": 1,
43
+ "issue-plausibly-classified-as-redundant": 1,
44
+ "required-documentation-evidence-not-present": 1,
45
+ "required-evidence-marked-missing": 1,
46
+ "required-evidence-missing": 7,
47
+ "required-review-limitation-not-present": 1,
48
+ "source-corpus-available": 1,
49
+ "validation-unavailable": 1
50
+ },
51
+ "comparisonToRound3": {
52
+ "coverageDelta": 0,
53
+ "inputTokensDelta": 264,
54
+ "outputTokensDelta": -613,
55
+ "tokenCostUnitsDelta": -349,
56
+ "configuredCostUsdDelta": -0.02906,
57
+ "latencyAverageDeltaMs": 174.59,
58
+ "latencyP95DeltaMs": 831,
59
+ "successCountDelta": -1,
60
+ "partialCountDelta": -3,
61
+ "incompleteCountDelta": 1,
62
+ "blockedCountDelta": 3
63
+ },
64
+ "comparisonToRound2": {"coverageDelta": 0, "successCountDelta": 0},
65
+ "privacy": {"forbiddenMatchCount": 0, "rawRepositoryPathsIncluded": false, "rawRepositoryContentIncluded": false},
66
+ "limitations": [
67
+ "The sample is fresh relative to rounds 2 and 3 but remains anonymized and bounded; it is not a full-repository evaluation.",
68
+ "One observation required an explicit timeout recovery; the initial timeout remains recorded as operational evidence.",
69
+ "The configured USD rates are experimental study rates, not a vendor invoice.",
70
+ "Independent adjudication evaluates the bounded candidate record and cannot recover evidence omitted from it.",
71
+ "The final sample did not demonstrate semantic improvement: no observation reached success."
72
+ ],
73
+ "contentHashAlgo": "sha256-normalized-v1",
74
+ "contentHash": "98a56b9c351e08c7b04860ee4da52eabab7654072fdd8f59d419aac0d167b27e"
75
+ }
@@ -0,0 +1,55 @@
1
+ ---
2
+ title: Round 4 final confirmation v1
3
+ description: Final fresh-sample confirmation of the evidence contract and study limitations.
4
+ ---
5
+
6
+ # Round 4 final confirmation v1
7
+
8
+ ## Objective
9
+
10
+ Test whether the round-3 evidence contract generalizes to a fresh representative anonymized sample, with the real configurable Codex CLI adjudicator and auditable operational measurements.
11
+
12
+ ## Sample and recovery
13
+
14
+ The run selected 12 records at offset 12 from `phase-9-ab-adjudicated-cost-03`, avoiding the 12 records used in rounds 2 and 3. The sample covered all four task categories, both controlled scenarios, both configured models, and five anonymized consumers. One architecture observation timed out and was retried in isolation at offset 17; the initial timeout remains recorded and is not counted as a successful decision.
15
+
16
+ ## Final result
17
+
18
+ | Metric | Result |
19
+ | --- | ---: |
20
+ | Sample size | 12 |
21
+ | Independent adjudications | 12 |
22
+ | Final coverage | 100% |
23
+ | Input tokens | 227,799 |
24
+ | Output tokens | 2,416 |
25
+ | Token cost units | 230,215 |
26
+ | Configured study cost | USD 0.301934 |
27
+ | Mean latency | 9,672.67 ms |
28
+ | P95 latency | 12,474 ms |
29
+ | `success` outcomes | 0 |
30
+ | `partial` outcomes | 1 |
31
+ | `incomplete` outcomes | 3 |
32
+ | `blocked` outcomes | 8 |
33
+
34
+ The final result is `inconclusive` for semantic improvement. The round-3 contract improved the structure of the adjudicator's explanations, but this fresh sample produced no successful outcome. The dominant issues were failed or unexecuted acceptance checks, missing required evidence, unavailable architecture artifacts, and unavailable validation commands. The evidence contract is therefore useful for diagnosis and measurement, but not sufficient by itself to establish documentation quality or delivery correctness.
35
+
36
+ ## Acceptance evidence
37
+
38
+ | Criterion | Result | Evidence |
39
+ | --- | --- | --- |
40
+ | Fresh sample was selected deterministically | Validated | Selection `offset: 12`, `limit: 12`; recovery `offset: 17`, `limit: 1`. |
41
+ | Real Codex adjudicator executed | Validated | Initial and recovery ledger hashes recorded in `round-4-confirmation-v1.json`. |
42
+ | Coverage, tokens, latency, cost, and outcomes measured | Validated | Summary artifact with per-run aggregate values and round-3 deltas. |
43
+ | Timeout recovery is explicit and auditable | Validated | Initial output and recovery output hashes are both recorded. |
44
+ | Privacy and reproducibility boundaries hold | Validated | Zero forbidden matches; source/config/output hashes recorded; no raw paths or content published. |
45
+ | Semantic documentation quality is proven | Not analyzed | The bounded candidate-record study cannot establish full-repository truth. |
46
+
47
+ ## Study conclusion
48
+
49
+ Across four rounds, Doc Bridge now has a reproducible measurement path for bounded evidence coverage, provider/adjudicator tokens, configured cost, latency, privacy, and failure recovery. The study does not support a claim that the evidence-contract change improves semantic correctness. A stronger claim requires a larger fresh sample, executable repository acceptance checks, richer validated evidence artifacts, and an independently approved semantic review protocol.
50
+
51
+ ## Provenance
52
+
53
+ - Source ledger: `cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5`.
54
+ - Round-4 configuration: `80590115a04157a385a6535cdc84d2040955599301ff9ae3b2f5194bc64ca482`.
55
+ - Summary artifact: `98a56b9c351e08c7b04860ee4da52eabab7654072fdd8f59d419aac0d167b27e`.
@@ -0,0 +1,33 @@
1
+ {
2
+ "type": "controlled-study-run-plan",
3
+ "schemaVersion": 1,
4
+ "planVersion": "v1",
5
+ "protocolVersion": "v1",
6
+ "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
7
+ "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
8
+ "sourceRevisionHash": "e74d5608daa1b6c7b3c44628f6976a0b0ca522f9aa61ba9b24b7545bf295002e",
9
+ "configurationHash": "f94d74f17dfcd00f3bcd8b972caf48371158d29244afb2e5358613736429650a",
10
+ "docBridgeVersion": "1.7.45",
11
+ "models": [
12
+ { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
13
+ { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
14
+ ],
15
+ "scenarios": [
16
+ { "id": "repository-only", "network": false },
17
+ { "id": "deterministic-doc-bridge", "network": false },
18
+ { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
19
+ ],
20
+ "taskIds": [
21
+ "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
22
+ "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
23
+ "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
24
+ "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
25
+ "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
26
+ "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
27
+ ],
28
+ "sampling": { "strategy": "balanced-task-strata", "sampleSize": 24 },
29
+ "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
30
+ "runId": "phase-7-canonical-metrics-01",
31
+ "contentHashAlgo": "sha256-normalized-v1",
32
+ "contentHash": "29e95cc2d8fa7f3e6a789884943f9d2cf20896ddb5405196b5ae869e87577d28"
33
+ }
@@ -0,0 +1,20 @@
1
+ # Cycle 8 — Bounded semantic adjudication
2
+
3
+ Run evidence: `.codex/verification-0.28-round28/semantic-adjudication/`.
4
+
5
+ Hypothesis: published metadata, CI configuration, and the deterministic README gate can distinguish a real contradiction from a local unreleased version, substantiate a CI claim, and expose remaining deterministic gaps without automatic documentation edits.
6
+
7
+ Budget: local read-only inspection, one npm metadata lookup, one deterministic README gate, no provider calls, no automatic documentation edits, and no external tracking mutation.
8
+
9
+ - [x] **1. Freeze the adjudication scope and keep it read-only.**
10
+ - [x] **2. Load the prior two-model semantic review as candidate evidence.**
11
+ - [x] **3. Resolve the published npm version from package metadata.**
12
+ - [x] **4. Compare the published version with the README claim.**
13
+ - [x] **5. Classify the local `package.json` version separately from the published version.**
14
+ - [x] **6. Verify the README gate is configured on the pull-request workflow.**
15
+ - [x] **7. Re-run the deterministic README gate and preserve its failure output.**
16
+ - [x] **8. Reproduce the onboarding overlap candidate.**
17
+ - [x] **9. Record adjudications without silently editing documentation.**
18
+ - [x] **10. Preserve the open source-hash gap and define the next corrective action.**
19
+
20
+ Decision: bounded adjudication completed. The published version and README agree (`1.7.45`); the local `package.json` version (`1.8.0`) is not a contradiction by itself. The CI claim is substantiated by `.github/workflows/ci.yml`. The README gate exposes three stale generated source hashes, which is the next deterministic correction. The onboarding overlap remains a low-severity documentation-ownership candidate. No automatic documentation edit is authorized.