@agentskit/doc-bridge 1.7.45 → 1.9.0

package/docs/spec/study-runner-v1.md

@@ -0,0 +1,35 @@
+---
+title: Controlled study runner v1
+description: Isolated, bounded, provenance-preserving execution for the controlled study baseline.
+---
+
+# Controlled study runner v1
+
+`ControlledStudyRunPlanV1` binds a task suite to one source revision, protocol hash, configuration hash, Doc Bridge version, two pinned model configurations, the three study scenarios, and a deterministic sampling policy. A plan is content-addressed and cannot be treated as a baseline merely because it parses.
+
+The default `balanced-task-strata` strategy preserves the original study sample. The `pairwise-task-strata` strategy selects exactly two declared scenarios for every selected task/model combination. It is intended for A/B comparisons such as `repository-only` versus `deterministic-doc-bridge`; the task, model, repository, and replicate remain constant across both arms.
+
+`runControlledCommand` starts a fresh child process for each attempt with `shell: false`, a unique session identifier, bounded runtime, bounded output, and an explicit retry limit. It records only status, exit metadata, timing, byte counts, hashes, labeled token counts, context-token attribution, first-evidence latency when observed, and tool-call counts. Raw stdout, stderr, prompts, repository content, and credentials are never persisted in an observation.
+
+The standard provider process contract is documented in [Study provider CLI contract v1](./study-provider-cli-v1.md). Provider selection is intentionally external to the core runner, so a hosted model CLI or an AgentsKit Registry agent can be used without bundling a local model runtime.
+
+The runner recognizes `completed`, `failed`, `timed-out`, `unavailable`, `invalid-output`, and `budget-exceeded` outcomes. An unavailable or invalid Registry agent is evidence of a failed/blocked scenario, never a successful task result. Provider token counts are labeled `provider`; calculated counts must be labeled `estimate`.
+
+When a provider reports labeled input and output usage and its configuration contains `pricing`, the runner also records `agentCostUsd` using the configured USD rates. Missing pricing or missing usage remains missing data. Independent adjudication is a separate `study adjudicate` command and records its own actor, method, token usage, and optional `adjudicatorCostUsd`; provider output is never used as its own approval.
+
+The current Codex plan uses a calibrated ceiling of `400000` tokens per task. This ceiling was raised only after a real pilot measured `331325` input tokens for a repository-only discovery task; the pilot remains a budget finding and is not counted as a successful observation. The ceiling is a safety bound, not a target, and the study still reports the measured token distribution. The current run selects 24 tasks, one per task, balanced across the six model×scenario strata (four tasks per stratum), and permits one attempt per task to avoid duplicate spend after the pilot showed the cost of retries.
+
+`ControlledStudyObservationV1` keeps model, scenario, task, variant, replicate, source, plan, and run provenance together with an automated deterministic or human adjudication state. The runner does not approve semantic results. `upsertControlledStudyObservation` is idempotent for the same run/task and rejects conflicting replacements; ledger hashes make changes detectable. Provider token usage also produces the explicit `providerTokenCostUnits` measurement: one unit is one provider-reported input or output token, not a currency claim. `contextTokens` is paired with `contextTokenMethod`; the default runner value is a byte-based estimate (`ceil(contextBytes / 4)`) and is never presented as provider usage. `firstEvidenceLatencyMs` is optional because a provider must not invent it.
+
+An observation may also record `round`, `taskOutcome`, `evidenceQuality`, `safetyOutcome`, `clarificationRequests`, `reworkCount`, and bounded numeric `measurements`. The measurement envelope is extensible; the v1 metrics calculator reserves names for search hit rate, acceptance checks, errors, documentation findings/examples/freshness/quality, and analysis or Registry-agent cost. Unknown measurements are preserved for future metric versions but are not interpreted automatically.
+
+Validate and inspect the committed plan and ledger with:
+
+```bash
+ak-docs study plan docs/study/run-plan-v1.json --text
+ak-docs study ledger docs/study/observation-ledger-v1.json --text
+```
+
+Use `ak-docs study run ... --dry-run` with a provider CLI configuration and one local repository root per declared population identifier to validate the planned sample before execution. A real run requires those explicit operational inputs and persists the ledger after each observation for recovery. For a pairwise run, set `sampling.strategy` to `pairwise-task-strata`, list two `sampling.scenarioIds`, and set `sampling.sampleSize` to a multiple of the model/scenario stratum count. The canonical pairwise matrix for 24 tasks and two models is 96 observations; a reduced pilot must use a separately hashed run plan and run ID and must be reported as pilot evidence.
+
+The committed ledger contains pilot, reduced-sample, and failed structured-provider observations. Failed observations remain immutable evidence; a recovered provider run must use a new run identifier and must not overwrite them. Use a new round identifier when the recovery is a separate experimental round; same-round recovery is valid only when the report explicitly selects the run IDs.

package/docs/spec/study-task-suite-v1.md

@@ -0,0 +1,41 @@
+---
+title: Study task suite v1
+description: A fixed, anonymization-safe task and adjudication contract for controlled study baselines.
+---
+
+# Study task suite v1
+
+`docs/study/task-suite-v1.json` is the canonical Phase 1 controlled task contract. It contains exactly four bounded tasks (discovery, architecture, documentation, and implementation) for each population identifier. The checked-in canonical artifact uses six stable anonymized consumer identifiers and therefore contains 24 tasks; reduced public pilots may declare a smaller population without changing the canonical artifact.
+
+The suite is a definition, not evidence that a task was executed. Execution records belong to a later observation ledger and must retain the suite hash, protocol hash, source revision, model, scenario, replicate, and verification run ID.
+
+## Required task fields
+
+Every task declares:
+
+- objective and initial context;
+- allowed tools and forbidden actions;
+- expected outcome, evidence requirements, and acceptance commands;
+- task-specific token/runtime budget;
+- every logical, endpoint, database, CLI, MCP, UI, and documentation surface, with a reason for each non-applicable surface;
+- separate `success`, `partial`, `incorrect`, `incomplete`, and `blocked` rubric criteria;
+- two distinguishable equivalent variants in its own variant group.
+
+The schema rejects unknown repositories, missing categories, budget overruns, duplicate identifiers, unsafe public text, and tasks without a complete rubric. It also rejects suites whose planned execution count exceeds `maxRuns`.
+
+## Execution plan
+
+The canonical plan is 24 tasks × 3 scenarios × 2 model slots × 2 replicates = 288 planned executions. `selectTaskExecutions` produces a deterministic ordering from the declared seed and assigns executions of each variant to every task. Scenario order, model identity, and replicate remain explicit in each selected execution. A pilot must publish its population, task count, replicate count, and sampling plan; it must not be presented as six-repository evidence.
+
+Run the contract through the packaged CLI:
+
+```bash
+ak-docs study tasks docs/study/task-suite-v1.json --text
+ak-docs study select docs/study/task-suite-v1.json --json
+```
+
+## Adjudication
+
+Agent output is never the adjudication result. During a real run, the deterministic evaluator receives the execution status, the bounded acceptance-check measurements (`acceptanceChecksPassed`, `acceptanceChecksTotal`, and `acceptanceChecksExecuted`), and the number of evidence items, then records an `automated` adjudication with method `deterministic-rubric-v1`. A success requires every declared check to be observed as executed and passed; missing execution telemetry is blocked rather than inferred. The evaluator returns one of the five rubric statuses without reading the provider's self-reported `taskOutcome`. Human adjudication remains required for semantic correctness, documentation quality, and any claim that cannot be proven by the declared artifact checks.
+
+No baseline is approved by creating or selecting the suite. Baseline execution, comparison, and immutable approval are separate phases.

package/docs/spec/study-verification-v1.md

@@ -0,0 +1,40 @@
+---
+title: Study verification binding v1
+description: Provenance, privacy, budget, recovery, and audit rules for controlled study evidence.
+---
+
+# Study verification binding v1
+
+`ControlledStudyVerifiedResultV1` is the publication-safe boundary for a controlled study result. It wraps a result with the exact verification run, source revision, protocol/configuration hashes, immutable baseline hash, artifact hashes, privacy scan outcome, and consumed budget.
+
+The binding is content-addressed with `sha256-normalized-v1`. It is not a baseline and cannot replace one. Baseline replacement is a separate, explicit, human-authorized study operation; the `@agentskit/harness@0.9.0` verification CLI does not replace baselines during a normal run. The replacement must record the new artifact hash, actor, intent, and timestamp in the study evidence ledger.
+
+Validate a binding through the real CLI:
+
+```bash
+ak-docs study verification docs/study/verification-binding-v1.json --text
+```
+
+## Required evidence
+
+Every controlled result must bind:
+
+- `verificationRunId` and `sourceRevisionHash`;
+- `protocolHash`, `configurationHash`, and `baselineHash`;
+- hashes for every included artifact, never raw repository content;
+- anonymized privacy status, artifact count, zero forbidden matches, and publication-review state;
+- maximum and consumed token/runtime budget.
+
+The binding rejects budget exhaustion and tampered content. A result without this envelope may be useful for local exploration, but is not a verified study result.
+
+## Privacy boundary
+
+`scanStudyPublicationArtifact` recursively checks machine-readable artifacts and logs for private paths, URLs, credentials, prompts, raw agent output, snippets, and repository-content fields. `scripts/study-privacy-gate.mjs` applies the same scan to a file or directory. It reports only field paths, never the matched secret value. Publication requires human review even when the deterministic scan has zero matches.
+
+Study artifacts use anonymized consumer identifiers and bounded hashes. Do not place repository names, internal paths, document contents, prompts, credentials, or raw responses in a ledger, report, log, or publication export.
+
+## Recovery and idempotency
+
+Observation ledgers are updated with `upsertControlledStudyObservation`; the same run/task is reused when its content hash matches and rejected when it conflicts. Workflow stage artifacts are atomic and hash-verified. A failed, cancelled, stale, corrupted, or interrupted workflow can resume from the last valid completed stage; invalid artifacts are not silently reused. Changed source, configuration, protocol, task definition, model, tool version, or evidence produces a new run.
+
+The harness records `CLARIFYING`, `PLANNED`, `VERIFYING`, `AWAITING_HUMAN_APPROVAL`, `AWAITING_AUTHORIZATION`, `BLOCKED`, and `COMPLETE` distinctly. Missing checks, stale evidence, privacy matches, budget exhaustion, or regressions cannot be reported as complete.

package/docs/study/README.md

@@ -0,0 +1,84 @@
+---
+title: Documentation efficiency study
+description: Anonymized measurements for Doc Bridge context efficiency, evidence quality, and agent workflows.
+---
+
+# Documentation efficiency study
+
+The next improvement series is governed by the [Token efficiency improvement
+plan](./token-efficiency-plan-v1.md). It separates context-payload reduction,
+provider-token reduction, and tokens-to-correct-action so that a smaller
+payload cannot be presented as a correctness or token-saving result by itself.
+
+This directory contains the anonymized, versioned evidence behind the Doc Bridge performance narrative. It is designed to let contributors inspect the numbers without exposing repository contents, private paths, prompts, credentials, or raw agent responses.
+
+## Executive summary
+
+An earlier anonymized dogfooding cycle estimated up to **99% context-payload reduction** between the scanned repository corpus and the P95 payload returned to an agent. This is a historical context-payload measurement, not a guaranteed token reduction or correctness result, and it must not be read as a provider-token result.
+
+The published controlled A/B study with 96 executions showed a directional operational signal of 18.46% fewer paired provider token-equivalent units across 46 token-complete pairs and 39.75 seconds lower P95 latency than repository-only context. Operational completion was 87.5% versus 75.0%. The sample is not a causal or enterprise-readiness claim.
+
+The independent bounded adjudicator recorded **0 adjudicator-success outcomes in both arms**. Because this adjudicator is mechanical and does not independently judge semantic correctness, this round does not demonstrate improved task correctness; it remains `inconclusive` and should be used only as an auditable historical directional measurement. A newer local pilot is not included in the public narrative until its semantic evaluation and publication review are complete.
+
+Acceptance telemetry is now checked separately from task correctness: every pilot observation records passed, total, and executed acceptance-check counts, while the deterministic adjudicator blocks a success when any declared check was not observed as executed. This prevents a missing execution measurement from being mistaken for a passing task.
+
+Semantic correctness and natural-language contradiction detection are explicit
+limitations of this study, not missing evidence hidden by the publication
+artifacts. They remain open validation work for a future adjudicated corpus.
+
+![Estimated context payload reduction](../landing/assets/context-payload-reduction.svg)
+
+![Controlled A/B comparison](../landing/assets/controlled-ab-comparison.svg)
+
+## Evidence classification
+
+| Label | Meaning |
+| --- | --- |
+| `controlled` | Fixed task, model, scenario, and measurement contract. |
+| `estimated` | Derived from bounded measurements where direct provider data was unavailable. |
+| `observational` | Real-world dogfooding evidence without a controlled counterfactual. |
+| `inconclusive` | Useful signal, but insufficient evidence for a causal claim. |
+
+## Published artifacts
+
+| Artifact | Purpose |
+| --- | --- |
+| [Protocol](./protocol-v1.json) | Metric definitions, scenarios, privacy boundary, and stopping rules. |
+| [Token efficiency protocol v2](./token-efficiency-protocol-v2.json) | Correct-action token metric, evidence gates, and non-regression targets. |
+| [Task suite](./task-suite-v1.json) | Fixed anonymized discovery, architecture, documentation, and implementation tasks. |
+| [Phase 4 public pilot suite](./phase4-public-pilot-task-suite-v1.json) | One public fixture, four tasks, and explicit pilot-only scope for validating the real runner without duplicating a repository. |
+| [Phase 4 public pilot plan](./phase4-public-pilot-run-plan-v1.json) | Hashed 16-observation pairwise pilot using the two configured Codex model slots. |
+| [Phase 4 public pilot ledger](./phase4-public-pilot-ledger-v1.json) | Bounded structured observations with hashes, statuses, provider usage, evidence IDs, and automated adjudication. |
+| [Phase 4 public pilot result](./phase4-public-pilot-result-v1.json) | Anonymized execution counts, paired token/latency measurements, failed preparation runs, and limitations. |
+| [Publication gate v1](./publication-gate-v1.md) | Evidence and claim-boundary checks required before publishing study results. |
+| [A/B baseline result](./ab-baseline-result-v1.json) | Controlled repository-only versus deterministic Doc Bridge measurements. |
+| [A/B baseline analysis](./ab-baseline-analysis-v1.md) | Human-readable interpretation of the controlled baseline. |
+| [Adjudicated A/B result](./ab-adjudicated-cost-result-v1.json) | Cost-attributed execution and independent bounded adjudication. |
+| [Round 2](./round-2-expanded-validation-v1.md) | Expanded adjudication and operational measurements. |
+| [Round 3](./round-3-evidence-contract-v1.md) | Evidence-coverage contract validation. |
+| [Round 4](./round-4-confirmation-v1.md) | Fresh-sample confirmation and final limitations. |
+| [Historical evidence](./historical-evidence-v1.json) | Earlier anonymized dogfooding and validation-cycle measurements. |
+
+All publication-bound artifacts pass the deterministic privacy gate. The values are intentionally anonymized and should be interpreted together with their provenance and limitations.
+
+The Phase 4 public pilot is intentionally separate from the six-population canonical suite. It is an execution-path and measurement smoke study, not evidence of enterprise-wide documentation quality or generalization. Its local provider and repository configuration stay outside the publication artifacts; only hashes, bounded observations, and the run identifier belong in published evidence.
+
+## Reproduce the charts
+
+From the repository root:
+
+```bash
+node scripts/generate-study-charts.mjs
+node scripts/generate-study-charts.mjs --check
+```
+
+The generator uses only the checked-in JSON artifacts and Node.js APIs. It does not call an LLM or access a consumer repository.
+
+## What the study does not prove
+
+- It does not prove that Doc Bridge makes every agent answer correctly.
+- It does not prove a universal 99% token reduction.
+- It does not establish causality across all repositories, languages, or models.
+- It does not replace semantic human review of documentation/code contradictions.
+
+The study is a transparent measurement baseline and a contribution surface for better analyzers, adapters, acceptance checks, and future replicates.

package/docs/study/ab-adjudicated-cost-analysis-v1.md

@@ -0,0 +1,29 @@
+# Controlled A/B study — adjudicated cost round
+
+This round compares `repository-only` with `deterministic-doc-bridge` using 24 anonymized task definitions, two configured CLI models, one replicate, and 48 observations per arm. The run completed all 96 planned observations and records execution failures rather than dropping them.
+
+## Result
+
+The deterministic Doc Bridge arm completed 42/48 executions (87.5%) versus 36/48 (75.0%) for repository-only. Its latency p95 was 84,440 ms versus 124,193 ms. Across 46 paired observations with token data, the deterministic arm used 53,325 fewer provider token-equivalent units on average (−18.459%). Provider token-equivalent units are input plus output tokens; they are not currency.
+
+Evidence citation was unchanged at 93.75%. The high-quality evidence rate was 20.833% for deterministic Doc Bridge versus 22.917% for repository-only. The independent bounded adjudicator recorded zero successful outcomes in both arms. That means this run does not demonstrate improved semantic correctness, even though it shows directional operational improvement.
+
+The result is `inconclusive`. It is useful as an auditable directional measurement, not as a causal, enterprise-readiness, or USD-cost claim.
+
+## Provenance
+
+- Plan: `ab-adjudicated-cost-plan-v2-v1.json`, hash `f61bf70063fcceb9dfe823ac9d0707f0f004a075143cae6e9518523c64ba1b4f`
+- Run: `phase-9-ab-adjudicated-cost-03`
+- Ledger hash: `cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5`
+- Metrics report hash: `d2849c3515da91385af91e9a80b70fa25b5f2ad6f248255229aba6ce73b3fd22`
+- Structured result: [ab-adjudicated-cost-result-v1.json](./ab-adjudicated-cost-result-v1.json)
+
+## What changed from the previous round
+
+- Provider token totals are now derived and labeled as token-equivalent units when both input and output counts exist.
+- Provider-reported outcomes are no longer treated as the final outcome; the runner records a deterministic bounded adjudication.
+- Child process groups are terminated on timeout or output limits, preventing nested CLI processes from surviving a failed observation.
+
+## Remaining measurement gap
+
+The adjudicator is mechanical and bounded. It does not independently judge whether the answer is semantically correct, whether documentation claims are true, or whether a repository change is safe. A future round needs a blinded human or separately configured review agent with an explicit rubric before correctness or quality claims can be strengthened.

package/docs/study/ab-adjudicated-cost-plan-v1.json

@@ -0,0 +1,33 @@
+{
+  "type": "controlled-study-run-plan",
+  "schemaVersion": 1,
+  "planVersion": "v1-ab-adjudicated-cost",
+  "protocolVersion": "v1",
+  "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
+  "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
+  "sourceRevisionHash": "0dfd7a83cfdfaecd78055642f74ce6552bfd83cf14f7bc6ed308ec978eda0ef6",
+  "configurationHash": "0a7dede28ccf1e672044c7b429a67d64d34e68c133dd21ffc07d763ff50642fe",
+  "docBridgeVersion": "1.7.45",
+  "models": [
+    { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
+    { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
+  ],
+  "scenarios": [
+    { "id": "repository-only", "network": false },
+    { "id": "deterministic-doc-bridge", "network": false },
+    { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
+  ],
+  "taskIds": [
+    "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
+    "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
+    "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
+    "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
+    "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
+    "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
+  ],
+  "sampling": { "strategy": "pairwise-task-strata", "sampleSize": 96, "scenarioIds": ["repository-only", "deterministic-doc-bridge"] },
+  "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
+  "runId": "phase-9-ab-adjudicated-cost-01",
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "7d03ca649d4d4d23cc9c7cb7237aba6e51697d04d2d8ac1912101ccb439b08b8"
+}

package/docs/study/ab-adjudicated-cost-plan-v2-v1.json

@@ -0,0 +1,33 @@
+{
+  "type": "controlled-study-run-plan",
+  "schemaVersion": 1,
+  "planVersion": "v1-ab-adjudicated-cost",
+  "protocolVersion": "v1",
+  "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
+  "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
+  "sourceRevisionHash": "c4a2decdccfd7262f64c78b6d8bb1c1101094d0afed519750b8ebfbfde1c1019",
+  "configurationHash": "0a7dede28ccf1e672044c7b429a67d64d34e68c133dd21ffc07d763ff50642fe",
+  "docBridgeVersion": "1.7.45",
+  "models": [
+    { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
+    { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
+  ],
+  "scenarios": [
+    { "id": "repository-only", "network": false },
+    { "id": "deterministic-doc-bridge", "network": false },
+    { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
+  ],
+  "taskIds": [
+    "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
+    "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
+    "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
+    "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
+    "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
+    "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
+  ],
+  "sampling": { "strategy": "pairwise-task-strata", "sampleSize": 96, "scenarioIds": ["repository-only", "deterministic-doc-bridge"] },
+  "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
+  "runId": "phase-9-ab-adjudicated-cost-03",
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "f61bf70063fcceb9dfe823ac9d0707f0f004a075143cae6e9518523c64ba1b4f"
+}

package/docs/study/ab-adjudicated-cost-result-v1.json

@@ -0,0 +1,80 @@
+{
+  "type": "controlled-study-ab-result",
+  "schemaVersion": 1,
+  "resultVersion": "v1",
+  "round": "ab-adjudicated-cost-2026-08-31",
+  "runId": "phase-9-ab-adjudicated-cost-03",
+  "planHash": "f61bf70063fcceb9dfe823ac9d0707f0f004a075143cae6e9518523c64ba1b4f",
+  "ledgerHash": "cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5",
+  "metricsReportHash": "d2849c3515da91385af91e9a80b70fa25b5f2ad6f248255229aba6ce73b3fd22",
+  "sample": {
+    "taskCount": 24,
+    "modelCount": 2,
+    "scenarioCount": 2,
+    "observationsPerArm": 48,
+    "pairedComparisons": 48
+  },
+  "arms": [
+    {
+      "scenarioId": "repository-only",
+      "observationCount": 48,
+      "executionStatus": { "completed": 36, "budget-exceeded": 11, "timed-out": 1 },
+      "completedRate": 0.75,
+      "taskOutcome": { "success": 3, "partial": 34, "blocked": 10, "missing": 1 },
+      "adjudicationOutcome": { "success": 0, "partial": 33, "blocked": 13, "incomplete": 2 },
+      "adjudicatedSuccessRate": 0,
+      "evidenceCitationRate": 0.9375,
+      "evidenceQualityRate": 0.229167,
+      "providerTokens": 13416254,
+      "providerTokenObservations": 47,
+      "providerTokenCostUnits": 13416254,
+      "latencyMeanMs": 68064.75,
+      "latencyP95Ms": 124193,
+      "totalCostUsd": null
+    },
+    {
+      "scenarioId": "deterministic-doc-bridge",
+      "observationCount": 48,
+      "executionStatus": { "completed": 42, "budget-exceeded": 5, "timed-out": 1 },
+      "completedRate": 0.875,
+      "taskOutcome": { "success": 1, "partial": 29, "blocked": 17, "missing": 1 },
+      "adjudicationOutcome": { "success": 0, "partial": 40, "blocked": 6, "incomplete": 2 },
+      "adjudicatedSuccessRate": 0,
+      "evidenceCitationRate": 0.9375,
+      "evidenceQualityRate": 0.208333,
+      "providerTokens": 11048857,
+      "providerTokenObservations": 47,
+      "providerTokenCostUnits": 11048857,
+      "latencyMeanMs": 58335.292,
+      "latencyP95Ms": 84440,
+      "totalCostUsd": null
+    }
+  ],
+  "pairedDeltas": {
+    "comparisonDirection": "deterministic-doc-bridge minus repository-only",
+    "providerTokenCostUnitsAverage": -53325.065,
+    "providerTokenCostUnitsRelative": -0.18459,
+    "providerTokenPairCount": 46,
+    "latencyAverageMs": -9729.458,
+    "latencyRelative": -0.142944,
+    "latencyP95Ms": -39753,
+    "completedRate": 0.125,
+    "evidenceCitationRate": 0,
+    "evidenceQualityRate": -0.020834,
+    "adjudicatedSuccessRate": 0
+  },
+  "interpretation": {
+    "classification": "inconclusive",
+    "summary": "The deterministic Doc Bridge arm completed more executions and used fewer provider token-equivalent units with lower latency p95 in this controlled sample. Evidence citation was unchanged and high-quality evidence was slightly lower. The independent adjudicator found no successful outcome in either arm, so the study does not establish improved task correctness.",
+    "decision": "Do not use this run as a causal, enterprise-readiness, or currency-cost claim. Use it as an auditable directional measurement and as input to the next study design."
+  },
+  "limitations": [
+    "The cost metric is provider token-equivalent units, not USD; the CLI provider emitted no currency cost.",
+    "Two observations lacked provider token counts, so paired token comparison uses 46 of 48 pairs.",
+    "Adjudication is deterministic and bounded by execution status, acceptance metrics, and evidence count; it is not independent semantic review by a human or second model.",
+    "The sample has 24 task definitions, two models, one replicate, and two scenarios; it does not establish causality or generalize to other repositories.",
+    "The study task acceptance instrumentation produced no adjudicated successes; semantic correctness remains not analyzed."
+  ],
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "9589b4e36283c36face82acdacbd756b3b96e001c59f2fb0574f9b5be4ef034c"
+}

package/docs/study/ab-baseline-analysis-v1.md

@@ -0,0 +1,21 @@
+# A/B baseline analysis — 2026-08-31
+
+This is the first paired baseline for `repository-only` versus `deterministic-doc-bridge`. It uses 24 anonymized task definitions, two pinned Codex CLI models, and 48 observations per arm. The result is directional and intentionally not an enterprise or causal claim.
+
+## Result
+
+The deterministic Doc Bridge arm completed 40/48 executions (83.3%) versus 38/48 (79.2%) for repository-only. Its completion p95 interval is 70.4–91.3%, while repository-only is 65.7–88.3%. Task success was 5/48 (10.4%) versus 4/47 (8.5%) where an outcome was recorded; this difference is inconclusive at this sample size.
+
+On paired observations, deterministic Doc Bridge used 5,762 fewer provider tokens on average (−2.29%, 47 pairs with token data). Its latency p95 was 3,150 ms lower (94,339 ms versus 97,489 ms). Evidence quality was higher (27.1% versus 23.4%), while evidence citation rate was slightly lower (91.7% versus 93.8%).
+
+The result is `inconclusive`: the signal is encouraging but small, one repository-only observation lacks provider token data, one repository-only observation timed out, and provider cost was not emitted. The next round must add cost attribution and independent outcome adjudication before using the study as a market claim.
+
+## Provenance
+
+- Plan: `ab-baseline-recovery-plan-v1.json`, hash `42d96e1152014318eadd2e0790259efc0ea96b138a1dcc4ac60c182ec357215f`
+- Run: `phase-8-ab-baseline-recovery-01`
+- Ledger hash: `9bf49dcee8071a4b5bfc7c515d2ee3d53c5f02344ffcf8bdeb7a412ad3f4e678`
+- Metrics report hash: `a04b7b1a3081d1011bcd99548d9929ebb96de75fd217823f498112a586533428`
+- Structured result: [ab-baseline-result-v1.json](./ab-baseline-result-v1.json)
+
+The failed first attempt remains in the immutable ledger as a failed run. The recovery result excludes it; no observation was overwritten.

package/docs/study/ab-baseline-plan-v1.json

@@ -0,0 +1,33 @@
+{
+  "type": "controlled-study-run-plan",
+  "schemaVersion": 1,
+  "planVersion": "v1-ab",
+  "protocolVersion": "v1",
+  "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
+  "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
+  "sourceRevisionHash": "1dc24c4e810bfdc49d0964c21e05b425252d99528dd488010653ebd3fd41dd49",
+  "configurationHash": "c56fb5612317decb45f65a5047481663e3a9df9e23c22efde8a78e1f8b5dc544",
+  "docBridgeVersion": "1.7.45",
+  "models": [
+    { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
+    { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
+  ],
+  "scenarios": [
+    { "id": "repository-only", "network": false },
+    { "id": "deterministic-doc-bridge", "network": false },
+    { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
+  ],
+  "taskIds": [
+    "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
+    "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
+    "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
+    "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
+    "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
+    "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
+  ],
+  "sampling": { "strategy": "pairwise-task-strata", "sampleSize": 96, "scenarioIds": ["repository-only", "deterministic-doc-bridge"] },
+  "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
+  "runId": "phase-8-ab-baseline-01",
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "ab735034299c9c44f464d31f2e95e637dc436416d1eb156b5e75f743b4c4ecc5"
+}

package/docs/study/ab-baseline-recovery-plan-v1.json

@@ -0,0 +1,33 @@
+{
+  "type": "controlled-study-run-plan",
+  "schemaVersion": 1,
+  "planVersion": "v1-ab-recovery",
+  "protocolVersion": "v1",
+  "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
+  "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
+  "sourceRevisionHash": "1dc24c4e810bfdc49d0964c21e05b425252d99528dd488010653ebd3fd41dd49",
+  "configurationHash": "b1e6373e423f20f24e991d9c9d5f1170a1230057c11d2462311c8244ec6df3b1",
+  "docBridgeVersion": "1.7.45",
+  "models": [
+    { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
+    { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
+  ],
+  "scenarios": [
+    { "id": "repository-only", "network": false },
+    { "id": "deterministic-doc-bridge", "network": false },
+    { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
+  ],
+  "taskIds": [
+    "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
+    "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
+    "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
+    "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
+    "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
+    "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
+  ],
+  "sampling": { "strategy": "pairwise-task-strata", "sampleSize": 96, "scenarioIds": ["repository-only", "deterministic-doc-bridge"] },
+  "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
+  "runId": "phase-8-ab-baseline-recovery-01",
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "42d96e1152014318eadd2e0790259efc0ea96b138a1dcc4ac60c182ec357215f"
+}

package/docs/study/ab-baseline-result-v1.json

@@ -0,0 +1,79 @@
+{
+  "type": "controlled-study-ab-result",
+  "schemaVersion": 1,
+  "resultVersion": "v1",
+  "round": "ab-baseline-2026-08-31",
+  "runId": "phase-8-ab-baseline-recovery-01",
+  "planHash": "42d96e1152014318eadd2e0790259efc0ea96b138a1dcc4ac60c182ec357215f",
+  "ledgerHash": "9bf49dcee8071a4b5bfc7c515d2ee3d53c5f02344ffcf8bdeb7a412ad3f4e678",
+  "metricsReportHash": "a04b7b1a3081d1011bcd99548d9929ebb96de75fd217823f498112a586533428",
+  "sample": {
+    "taskCount": 24,
+    "modelCount": 2,
+    "scenarioCount": 2,
+    "observationsPerArm": 48,
+    "pairedComparisons": 48
+  },
+  "arms": [
+    {
+      "scenarioId": "repository-only",
+      "observationCount": 48,
+      "executionStatus": { "completed": 38, "budget-exceeded": 9, "timed-out": 1 },
+      "completedRate": 0.7916666667,
+      "completedConfidence95": { "low": 0.6574, "high": 0.8827 },
+      "taskOutcome": { "success": 4, "partial": 29, "blocked": 12, "incomplete": 2, "missing": 1 },
+      "successRate": 0.085106383,
+      "successOutcomeCoverage": 47,
+      "evidenceCitationRate": 0.9375,
+      "evidenceQualityRate": 0.2340425532,
+      "providerTokens": 11841294,
+      "providerTokenObservations": 47,
+      "latencyP95Ms": 97489,
+      "contextBytesP95": 2085,
+      "responseBytesP95": 1084,
+      "totalCostUsd": null
+    },
+    {
+      "scenarioId": "deterministic-doc-bridge",
+      "observationCount": 48,
+      "executionStatus": { "completed": 40, "budget-exceeded": 8, "timed-out": 0 },
+      "completedRate": 0.8333333333,
+      "completedConfidence95": { "low": 0.7042, "high": 0.913 },
+      "taskOutcome": { "success": 5, "partial": 22, "blocked": 19, "incomplete": 2 },
+      "successRate": 0.1041666667,
+      "successOutcomeCoverage": 48,
+      "evidenceCitationRate": 0.9166666667,
+      "evidenceQualityRate": 0.2708333333,
+      "providerTokens": 12142335,
+      "providerTokenObservations": 48,
+      "latencyP95Ms": 94339,
+      "contextBytesP95": 2094,
+      "responseBytesP95": 1120,
+      "totalCostUsd": null
+    }
+  ],
+  "pairedDeltas": {
+    "providerTokensAverage": -5761.7021277,
+    "providerTokensRelative": -0.022868,
+    "latencyAverageMs": 609.5625,
+    "latencyP95Ms": -3150,
+    "contextP95Bytes": 9,
+    "responseP95Bytes": 22,
+    "completedRate": 0.0416666666,
+    "successRate": 0.0190602837,
+    "evidenceCitationRate": -0.0208333333,
+    "evidenceQualityRate": 0.0359297801
+  },
+  "interpretation": {
+    "classification": "inconclusive",
+    "summary": "The deterministic Doc Bridge arm shows higher completion and evidence-quality rates and lower latency p95 in this sample, but the paired token reduction is small and citation rate is slightly lower. The sample is too small and outcome coverage is incomplete for an enterprise or causal claim."
+  },
+  "limitations": [
+    "Provider cost was not emitted by the CLI adapter; cost comparison is not analyzed.",
+    "Repository-only has one observation without provider token counts and one timed-out execution; token comparison uses 47 paired token observations.",
+    "Task outcomes were not adjudicated by an independent reviewer; success rates are directional study measurements.",
+    "This is one controlled run with 24 tasks and two models; it does not establish causality or generalize to other repositories."
+  ],
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "94ac03bcb9f1d600ded42084503ca27d6a3a8e730a338b05fdd3c365d5e5ed7a"
+}