@agentskit/doc-bridge 1.7.45 → 1.10.0

package/docs/spec/study-metrics-v1.md

@@ -0,0 +1,58 @@
+---
+title: Longitudinal study metrics v1
+description: Deterministic, content-addressed comparison of controlled study rounds.
+---
+
+# Longitudinal study metrics v1
+
+`ak-docs study metrics <observation-ledger.json>` calculates an anonymization-safe report from `ControlledStudyObservationV1` records. It groups observations by task, repository, category, difficulty, model, scenario, replicate, and aggregate, then compares the selected baseline and current rounds. When a round contains retries or recovery runs, pass `--baseline-run-id` and `--current-run-id` to bind each side to an exact run without rewriting ledger history.
+
+When both sides intentionally share one round (for example, two arms of a pairwise run), the same-round selection keeps only the explicitly named run IDs. A same-round report exposes the scenario groups for arm-level analysis; it does not manufacture a round comparison.
+
+```bash
+ak-docs study metrics docs/study/observation-ledger-v1.json --text
+ak-docs study metrics docs/study/observation-ledger-v1.json --baseline-round baseline --current-round cycle-1 --json
+ak-docs study metrics docs/study/observation-ledger-v1.json --baseline-round baseline --baseline-run-id baseline-run-01 --current-round cycle-1 --current-run-id cycle-1-run-01 --json
+```
+
+## Metric contract
+
+The report keeps metric families separate. Counts are sums, rates are weighted ratios or means as stated below, and latency/answer-cost distributions use p95.
+
+| Family | Metrics | Collection |
+| --- | --- | --- |
+| Discovery/context | `providerTokens`, `estimatedTokens`, `tokensToCorrectAnswerP95`, `timeToCorrectAnswerP95Ms`, `contextBytesP95`, `responseBytesP95`, `searchHitRate`, `evidenceCitationRate`, `averageToolCalls`, `clarificationRate` | provider or explicitly labeled observation measurements |
+| Task/delivery | `successRate`, `completedRate`, `acceptanceCheckRate`, `errorRate`, `reworkRate`, `safetyRate` | adjudication and observation measurements |
+| Documentation | `documentationFindingCount`, `documentationExampleRate`, `documentationFreshnessRate`, `documentationCorrectnessRate`, `documentationCompletenessRate`, `documentationClarityRate`, `documentationMaintainabilityRate` | audit or human/adjudicator measurements |
+| Operations/cost | `latencyP95Ms`, `analysisCostUsd`, `agentCostUsd`, `adjudicatorCostUsd`, `totalCostUsd`, `providerTokenCostUnits`, `adjudicatorTokenCostUnits` | runner and explicitly labeled cost measurements |
+
+Provider token totals and estimated token totals are never combined. An estimate is valid only when the observation says `tokenMethod: estimate`. Known values are retained in partial totals; if any observation is missing a value, the metric is marked with a `-partial` entry in `missingMetrics` and must not be used as a complete-round comparison. A value that was not collected at all is `null`, listed in `missingMetrics`, and excluded from its denominator.
+
+Observations may carry additional bounded numeric values in `measurements`. The reserved keys above have the following meanings:
+
+- `searchHitRate`, `errorRate`, and documentation `*Rate` values are ratios from `0` to `1`.
+- `acceptanceChecksPassed` and `acceptanceChecksTotal` form a weighted acceptance ratio.
+- `documentationFindingCount`, `analysisCostUsd`, `agentCostUsd`, `adjudicatorCostUsd`, `providerTokenCostUnits`, and `adjudicatorTokenCostUnits` are additive values. Token-unit fields are one unit per provider-reported input or output token and are not currency values. USD fields are valid only when the run records configured rates and provider-reported usage.
+- `safetyOutcome` is preferred for the safety rate; `safe` is `1`, `unsafe` is `0`, and `not-applicable` is excluded.
+
+## Comparison rules
+
+Each comparison contains baseline and current values, absolute change, relative change when the baseline is non-zero, sample sizes, limitations, and one of `improved`, `unchanged`, `regressed`, `inconclusive`, or `not-analyzed`.
+
+- Fewer tokens, lower latency, less context, fewer clarifications, less rework, and lower cost are directionally better.
+- Higher success, evidence, safety, acceptance, search-hit, and documentation-quality rates are directionally better.
+- A lower-efficiency value cannot make a round `improved` when success, evidence citation, evidence quality, acceptance, safety, or rework regresses.
+- A comparison with fewer than two observations on either side is `inconclusive`, not a pass.
+- Missing baseline/current groups are `not-analyzed`; the calculator does not invent values or backfill historical data.
+
+Wilson 95% intervals are emitted for completion and task success rates. They communicate uncertainty; they do not establish causality. Aggregate results must be read with the subgroup results because model, scenario, repository, task difficulty, and documentation changes can confound a round.
+
+## Provenance and immutability
+
+The report is normalized and content-hashed with `sha256-normalized-v1`. JSON and text output expose the same group/comparison metric values and report hash. Calculating metrics never writes or replaces a baseline. Baseline approval and replacement remain explicit, audited harness operations.
+
+The report contains identifiers, versions, hashes, counts, timings, and classifications only. It must not contain repository contents, private paths, prompts, credentials, or raw agent responses.
+
+The CLI exits non-zero for a comparison classified as `regressed`. Structural verification that only checks report generation may pass `--allow-regressions`; regressions remain present in the report and should still fail a quality gate that is intended to enforce improvement.
+
+The first anonymized deterministic documentation snapshot is [Audit round 2026-08-31](../study/documentation-audit-round-2026-08-31.json). It is an observational inventory, not the controlled baseline: all six consumers still require semantic review, and its missing metrics must remain missing rather than being inferred.

package/docs/spec/study-protocol-v1.md

@@ -0,0 +1,46 @@
+---
+title: Study protocol v1
+description: Versioned and anonymization-safe contracts for measuring documentation quality and agent efficiency.
+---
+
+# Study protocol v1
+
+Phase 0 defines two content-addressed JSON artifacts:
+
+- `StudyProtocolV1` defines the population, evidence classes, task categories, model slots, scenarios, metrics, executable outcome checks, budget, privacy policy, and stopping rules.
+- `HistoricalEvidenceRegistryV1` preserves historical observations, available numeric measurements, explicit missing measurements, limitations, and validation state.
+
+The contracts are validated by the real Doc Bridge CLI:
+
+```bash
+ak-docs study protocol docs/study/protocol-v1.json --text
+ak-docs study history docs/study/historical-evidence-v1.json --protocol docs/study/protocol-v1.json --text
+```
+
+Both artifacts contain a SHA-256 normalized content hash. A hash mismatch, unknown cross-reference, duplicate identifier, unsafe public field, or incompatible schema fails closed.
+
+## Evidence classes
+
+`historical` records preserve observations from prior cycles. They may show evolution but cannot support causal claims because the task suite, models, and metrics were not fixed.
+
+`controlled` records belong to the approved benchmark protocol and can be compared only when the source revision, protocol, model, scenario, and validation contract match.
+
+## Outcome coverage
+
+Every protocol outcome must map to one or more executable checks. An outcome outside the current phase must instead declare an explicit `notApplicableReason`; an empty mapping is never silently accepted.
+
+## Privacy contract
+
+Public study artifacts use anonymized repository identifiers. They do not contain repository contents, paths, prompts, credentials, private identifiers, or raw agent responses. Publication always requires human review. The schema deliberately uses bounded identifiers and safe source references instead of arbitrary URLs or file paths.
+
+## Missing data
+
+Unavailable historical metrics are represented in `missingMetrics` with one of `missing`, `not-analyzed`, `blocked`, or `not-applicable` plus a reason. The importer never derives a missing value from another metric.
+
+## Protocol changes
+
+Changing task definitions, metric formulas, model identity, scenario behavior, privacy rules, or validation requirements changes the protocol hash. Existing baseline and historical evidence remains immutable; a changed protocol starts a new controlled evidence series.
+
+## Human authority
+
+The protocol and historical registry can be discovered and summarized by agents, but agents cannot approve protocol changes, replace a baseline, adjudicate their own output, or publish a report. Those actions remain human-gated through the verification harness.

package/docs/spec/study-provider-cli-v1.md

@@ -0,0 +1,116 @@
+---
+title: Study provider CLI contract v1
+description: Generic, bounded command contract for model and agent execution in controlled studies.
+---
+
+# Study provider CLI contract v1
+
+The provider adapter lets a study use a hosted model CLI, a vendor CLI, or an AgentsKit Registry agent without adding that provider to Doc Bridge. It does not install a model runtime and does not assume Ollama.
+
+## Configuration
+
+Create a content-addressed configuration with `createStudyProviderCliConfig` or validate one with:
+
+```bash
+ak-docs study providers path/to/provider-cli.json --text
+```
+
+Each provider entry contains:
+
+| Field | Meaning |
+| --- | --- |
+| `modelId` | Must match one pinned model in the run plan. |
+| `scenarioIds` | One or more of `repository-only`, `deterministic-doc-bridge`, `registry-assisted`. |
+| `command` | Executable name or path. It is launched without a shell. |
+| `args` | Ordered arguments passed unchanged to the executable. |
+| `envAllowlist` | Names copied from the parent environment. Credentials must be named explicitly. |
+| `providerNetwork` | Audit declaration that the provider may send input to an external service. |
+| `maxInputBytes` / `maxOutputBytes` | Provider-specific bounds; the lower bound wins against the run-plan budget. |
+| `pricing` | Optional configured USD rates used only to derive auditable `agentCostUsd`; omitted rates produce no currency claim. |
+
+The optional top-level `adjudicator` uses the same bounded command fields and adds `id` and `modelId`. It is a separate process from the provider under evaluation. Its output is used to record an independent rubric decision; the provider cannot approve its own result.
+
+```json
+{
+  "adjudicator": {
+    "id": "independent-reviewer",
+    "modelId": "reference-model",
+    "command": "/absolute/path/to/adjudicator",
+    "args": [],
+    "envAllowlist": [],
+    "providerNetwork": true,
+    "maxInputBytes": 1000000,
+    "maxOutputBytes": 50000,
+    "pricing": {
+      "currency": "USD",
+      "inputPerMillionUsd": 1,
+      "cachedInputPerMillionUsd": 0.5,
+      "outputPerMillionUsd": 3
+    }
+  }
+}
+```
+
+The cost formula is `(uncached input × input rate + cached input × cached rate + output × output rate) / 1,000,000`. Rates are study configuration, not a vendor billing assertion; the report must preserve the configuration hash and run ID. Cost is emitted only when the process reports both input and output token counts. `providerTokenCostUnits` remains the provider-reported input plus output total and is never presented as currency.
+
+The parser rejects stale content hashes, duplicate model/scenario mappings, unsafe environment names, and invalid command arguments. A command must exist and be executable before any task starts.
+
+## Provider process contract
+
+The child process receives one JSON request on stdin and must emit exactly one JSON object on stdout. Human-readable logs must go to stderr. The request contains the task, scenario, expected outcome, evidence requirements, acceptance checks, allowed tools, and forbidden actions. It does not contain the configured repository path.
+
+The generic runner does not enforce scenario tool semantics. A provider mapping must enforce its own scenario contract: a `repository-only` command must not query Doc Bridge, while a `deterministic-doc-bridge` command must execute the configured deterministic query or handoff and pass its bounded result to the model. A scenario label or `allowedTools` field alone is not evidence that Doc Bridge was used.
+
+The response may contain the bounded metrics below. Unknown fields are ignored by the generic runner, while unknown numeric measurements are preserved by the observation schema for future metric versions. The bundled Codex adapter additionally passes a JSON Schema to the provider CLI and requires the semantic fields (`taskOutcome`, `evidenceQuality`, `safetyOutcome`, evidence IDs, clarification/rework counts, an observed latency or explicit `null`, and measurements) so missing output is visible as a provider failure rather than silently becoming an empty result. When observed, measurements must use canonical names: `searchHitRate`, `acceptanceChecksPassed`, `acceptanceChecksTotal`, `errorRate`, `documentationFindingCount`, the documentation `*Rate` fields, `analysisCostUsd`, and `agentCostUsd`.
+
+```json
+{
+  "inputTokens": 1200,
+  "outputTokens": 340,
+  "tokenMethod": "provider",
+  "toolCalls": 4,
+  "firstEvidenceLatencyMs": 820,
+  "taskOutcome": "success",
+  "evidenceQuality": "high",
+  "safetyOutcome": "safe",
+  "evidenceIds": ["artifact-hash:abc123"],
+  "clarificationRequests": 0,
+  "reworkCount": 0,
+  "measurements": {
+    "searchHitRate": 1,
+    "timeToCorrectAnswerP95Ms": 820
+  }
+}
+```
+
+Raw prompts, responses, repository contents, paths, and credentials are not written to the observation ledger. The ledger stores status, hashes, timing, labeled token counts, context-token attribution, first-evidence latency when observed, tool counts, metric fields, and automated or pending human adjudication. The runner derives `providerTokenCostUnits` from provider-reported input plus output tokens; this is a transparent token-equivalent cost metric and must not be presented as currency. Context-token estimates are stored separately with `contextTokenMethod: "estimate"`; they are never combined with provider usage.
+
+The bundled Codex adapter also records privacy-safe aggregate context telemetry: `observedToolEventCount`, `observedToolInputBytes`, `observedToolOutputBytes`, `observedProviderInputBytes`, `observedAgentMessageBytes`, `observedContextBytes`, `observedProviderDurationMs`, and (when a tool event is observed) `timeToFirstToolEventMs`. These are byte counts and durations only; commands, paths, prompts, responses, and repository content are not emitted by the adapter. The duration is wall-clock time for the isolated provider process, while `timeToFirstToolEventMs` measures the first observed tool event, not semantic answer quality. The deterministic handoff is serialized as compact JSON before it is passed to the model, and the adapter explicitly tells the model to use its `startHere` and `readBeforeEditing` paths before exploring the repository. This preserves canonical content while removing formatting-only bytes and makes the intended low-context path measurable. Gzip and a cross-process cache are intentionally not used for model context: the model must receive readable content, and the current adapter starts an isolated process per observation, so either mechanism would add complexity without reducing billed model tokens. The public `--agent` CLI path also emits compact JSON, so machine consumers do not pay for presentation-only whitespace.
+
+## Independent adjudication
+
+Adjudicate a persisted ledger in a separate process:
+
+```bash
+ak-docs study adjudicate docs/study/observation-ledger-v1.json docs/study/task-suite-v1.json \
+  --adjudicator path/to/provider-cli.json --output .tmp/adjudicated-ledger.json \
+  --run-id cycle-1 --offset 24 --limit 24 --json
+```
+
+The adjudicator receives only anonymized task metadata and the bounded structured candidate record: outcomes, evidence IDs, per-requirement evidence presence, acceptance execution state, numeric measurements, execution status, duration, and response size. It does not receive repository paths, raw stdout, prompts, repository contents, or credentials. Automated adjudication records the adjudicator configuration hash in addition to its actor and method. A timeout, unavailable command, oversized output, or invalid JSON leaves the adjudication `pending`; no deterministic fallback silently converts that failure into a success. `--offset` and `--limit` support deterministic low-cost samples while preserving all unselected observations in the output ledger; offset counts only observations matching `--run-id`.
+
+## Running a study
+
+First validate the complete plan without invoking a provider:
+
+```bash
+ak-docs study run docs/study/run-plan-v1.json docs/study/task-suite-v1.json \
+  --providers path/to/provider-cli.json \
+  --repositories path/to/repositories.json \
+  --ledger path/to/observation-ledger.json \
+  --dry-run --text
+```
+
+The canonical run plan dry run validates the configured 24-task balanced sample, repository roots, executable commands, and input limits. It writes no ledger and makes no provider call. To execute, omit `--dry-run`. The ledger is persisted after each observation, so an interrupted run can resume and skip completed task executions idempotently. Reduced pilots may use fewer population identifiers and tasks, but require their own hashed suite, run plan, repository configuration, run ID, and explicit pilot label. A full 288-execution matrix requires a separately hashed run plan with `sampling.sampleSize` set to 288.
+
+Provider configuration and repository-root configuration are local operational inputs. Do not commit credentials, private paths, raw provider output, or consumer repository content to a publication artifact.

package/docs/spec/study-runner-v1.md

@@ -0,0 +1,35 @@
+---
+title: Controlled study runner v1
+description: Isolated, bounded, provenance-preserving execution for the controlled study baseline.
+---
+
+# Controlled study runner v1
+
+`ControlledStudyRunPlanV1` binds a task suite to one source revision, protocol hash, configuration hash, Doc Bridge version, two pinned model configurations, the three study scenarios, and a deterministic sampling policy. A plan is content-addressed and cannot be treated as a baseline merely because it parses.
+
+The default `balanced-task-strata` strategy preserves the original study sample. The `pairwise-task-strata` strategy selects exactly two declared scenarios for every selected task/model combination. It is intended for A/B comparisons such as `repository-only` versus `deterministic-doc-bridge`; the task, model, repository, and replicate remain constant across both arms.
+
+`runControlledCommand` starts a fresh child process for each attempt with `shell: false`, a unique session identifier, bounded runtime, bounded output, and an explicit retry limit. It records only status, exit metadata, timing, byte counts, hashes, labeled token counts, context-token attribution, first-evidence latency when observed, and tool-call counts. Raw stdout, stderr, prompts, repository content, and credentials are never persisted in an observation.
+
+The standard provider process contract is documented in [Study provider CLI contract v1](./study-provider-cli-v1.md). Provider selection is intentionally external to the core runner, so a hosted model CLI or an AgentsKit Registry agent can be used without bundling a local model runtime.
+
+The runner recognizes `completed`, `failed`, `timed-out`, `unavailable`, `invalid-output`, and `budget-exceeded` outcomes. An unavailable or invalid Registry agent is evidence of a failed/blocked scenario, never a successful task result. Provider token counts are labeled `provider`; calculated counts must be labeled `estimate`.
+
+When a provider reports labeled input and output usage and its configuration contains `pricing`, the runner also records `agentCostUsd` using the configured USD rates. Missing pricing or missing usage remains missing data. Independent adjudication is a separate `study adjudicate` command and records its own actor, method, token usage, and optional `adjudicatorCostUsd`; provider output is never used as its own approval.
+
+The current Codex plan uses a calibrated ceiling of `400000` tokens per task. This ceiling was raised only after a real pilot measured `331325` input tokens for a repository-only discovery task; the pilot remains a budget finding and is not counted as a successful observation. The ceiling is a safety bound, not a target, and the study still reports the measured token distribution. The current run selects 24 tasks, one per task, balanced across the six model×scenario strata (four tasks per stratum), and permits one attempt per task to avoid duplicate spend after the pilot showed the cost of retries.
+
+`ControlledStudyObservationV1` keeps model, scenario, task, variant, replicate, source, plan, and run provenance together with an automated deterministic or human adjudication state. The runner does not approve semantic results. `upsertControlledStudyObservation` is idempotent for the same run/task and rejects conflicting replacements; ledger hashes make changes detectable. Provider token usage also produces the explicit `providerTokenCostUnits` measurement: one unit is one provider-reported input or output token, not a currency claim. `contextTokens` is paired with `contextTokenMethod`; the default runner value is a byte-based estimate (`ceil(contextBytes / 4)`) and is never presented as provider usage. `firstEvidenceLatencyMs` is optional because a provider must not invent it.
+
+An observation may also record `round`, `taskOutcome`, `evidenceQuality`, `safetyOutcome`, `clarificationRequests`, `reworkCount`, and bounded numeric `measurements`. The measurement envelope is extensible; the v1 metrics calculator reserves names for search hit rate, acceptance checks, errors, documentation findings/examples/freshness/quality, and analysis or Registry-agent cost. Unknown measurements are preserved for future metric versions but are not interpreted automatically.
+
+Validate and inspect the committed plan and ledger with:
+
+```bash
+ak-docs study plan docs/study/run-plan-v1.json --text
+ak-docs study ledger docs/study/observation-ledger-v1.json --text
+```
+
+Use `ak-docs study run ... --dry-run` with a provider CLI configuration and one local repository root per declared population identifier to validate the planned sample before execution. A real run requires those explicit operational inputs and persists the ledger after each observation for recovery. For a pairwise run, set `sampling.strategy` to `pairwise-task-strata`, list two `sampling.scenarioIds`, and set `sampling.sampleSize` to a multiple of the model/scenario stratum count. The canonical pairwise matrix for 24 tasks and two models is 96 observations; a reduced pilot must use a separately hashed run plan and run ID and must be reported as pilot evidence.
+
+The committed ledger contains pilot, reduced-sample, and failed structured-provider observations. Failed observations remain immutable evidence; a recovered provider run must use a new run identifier and must not overwrite them. Use a new round identifier when the recovery is a separate experimental round; same-round recovery is valid only when the report explicitly selects the run IDs.

package/docs/spec/study-task-suite-v1.md

@@ -0,0 +1,41 @@
+---
+title: Study task suite v1
+description: A fixed, anonymization-safe task and adjudication contract for controlled study baselines.
+---
+
+# Study task suite v1
+
+`docs/study/task-suite-v1.json` is the canonical Phase 1 controlled task contract. It contains exactly four bounded tasks (discovery, architecture, documentation, and implementation) for each population identifier. The checked-in canonical artifact uses six stable anonymized consumer identifiers and therefore contains 24 tasks; reduced public pilots may declare a smaller population without changing the canonical artifact.
+
+The suite is a definition, not evidence that a task was executed. Execution records belong to a later observation ledger and must retain the suite hash, protocol hash, source revision, model, scenario, replicate, and verification run ID.
+
+## Required task fields
+
+Every task declares:
+
+- objective and initial context;
+- allowed tools and forbidden actions;
+- expected outcome, evidence requirements, and acceptance commands;
+- task-specific token/runtime budget;
+- every logical, endpoint, database, CLI, MCP, UI, and documentation surface, with a reason for each non-applicable surface;
+- separate `success`, `partial`, `incorrect`, `incomplete`, and `blocked` rubric criteria;
+- two distinguishable equivalent variants in its own variant group.
+
+The schema rejects unknown repositories, missing categories, budget overruns, duplicate identifiers, unsafe public text, and tasks without a complete rubric. It also rejects suites whose planned execution count exceeds `maxRuns`.
+
+## Execution plan
+
+The canonical plan is 24 tasks × 3 scenarios × 2 model slots × 2 replicates = 288 planned executions. `selectTaskExecutions` produces a deterministic ordering from the declared seed and assigns executions of each variant to every task. Scenario order, model identity, and replicate remain explicit in each selected execution. A pilot must publish its population, task count, replicate count, and sampling plan; it must not be presented as six-repository evidence.
+
+Run the contract through the packaged CLI:
+
+```bash
+ak-docs study tasks docs/study/task-suite-v1.json --text
+ak-docs study select docs/study/task-suite-v1.json --json
+```
+
+## Adjudication
+
+Agent output is never the adjudication result. During a real run, the deterministic evaluator receives the execution status, the bounded acceptance-check measurements (`acceptanceChecksPassed`, `acceptanceChecksTotal`, and `acceptanceChecksExecuted`), and the number of evidence items, then records an `automated` adjudication with method `deterministic-rubric-v1`. A success requires every declared check to be observed as executed and passed; missing execution telemetry is blocked rather than inferred. The evaluator returns one of the five rubric statuses without reading the provider's self-reported `taskOutcome`. Human adjudication remains required for semantic correctness, documentation quality, and any claim that cannot be proven by the declared artifact checks.
+
+No baseline is approved by creating or selecting the suite. Baseline execution, comparison, and immutable approval are separate phases.

package/docs/spec/study-verification-v1.md

@@ -0,0 +1,40 @@
+---
+title: Study verification binding v1
+description: Provenance, privacy, budget, recovery, and audit rules for controlled study evidence.
+---
+
+# Study verification binding v1
+
+`ControlledStudyVerifiedResultV1` is the publication-safe boundary for a controlled study result. It wraps a result with the exact verification run, source revision, protocol/configuration hashes, immutable baseline hash, artifact hashes, privacy scan outcome, and consumed budget.
+
+The binding is content-addressed with `sha256-normalized-v1`. It is not a baseline and cannot replace one. Baseline replacement is a separate, explicit, human-authorized study operation; the `@agentskit/harness@0.9.0` verification CLI does not replace baselines during a normal run. The replacement must record the new artifact hash, actor, intent, and timestamp in the study evidence ledger.
+
+Validate a binding through the real CLI:
+
+```bash
+ak-docs study verification docs/study/verification-binding-v1.json --text
+```
+
+## Required evidence
+
+Every controlled result must bind:
+
+- `verificationRunId` and `sourceRevisionHash`;
+- `protocolHash`, `configurationHash`, and `baselineHash`;
+- hashes for every included artifact, never raw repository content;
+- anonymized privacy status, artifact count, zero forbidden matches, and publication-review state;
+- maximum and consumed token/runtime budget.
+
+The binding rejects budget exhaustion and tampered content. A result without this envelope may be useful for local exploration, but is not a verified study result.
+
+## Privacy boundary
+
+`scanStudyPublicationArtifact` recursively checks machine-readable artifacts and logs for private paths, URLs, credentials, prompts, raw agent output, snippets, and repository-content fields. `scripts/study-privacy-gate.mjs` applies the same scan to a file or directory. It reports only field paths, never the matched secret value. Publication requires human review even when the deterministic scan has zero matches.
+
+Study artifacts use anonymized consumer identifiers and bounded hashes. Do not place repository names, internal paths, document contents, prompts, credentials, or raw responses in a ledger, report, log, or publication export.
+
+## Recovery and idempotency
+
+Observation ledgers are updated with `upsertControlledStudyObservation`; the same run/task is reused when its content hash matches and rejected when it conflicts. Workflow stage artifacts are atomic and hash-verified. A failed, cancelled, stale, corrupted, or interrupted workflow can resume from the last valid completed stage; invalid artifacts are not silently reused. Changed source, configuration, protocol, task definition, model, tool version, or evidence produces a new run.
+
+The harness records `CLARIFYING`, `PLANNED`, `VERIFYING`, `AWAITING_HUMAN_APPROVAL`, `AWAITING_AUTHORIZATION`, `BLOCKED`, and `COMPLETE` distinctly. Missing checks, stale evidence, privacy matches, budget exhaustion, or regressions cannot be reported as complete.

package/docs/study/README.md

@@ -0,0 +1,84 @@
+---
+title: Documentation efficiency study
+description: Anonymized measurements for Doc Bridge context efficiency, evidence quality, and agent workflows.
+---
+
+# Documentation efficiency study
+
+The next improvement series is governed by the [Token efficiency improvement
+plan](./token-efficiency-plan-v1.md). It separates context-payload reduction,
+provider-token reduction, and tokens-to-correct-action so that a smaller
+payload cannot be presented as a correctness or token-saving result by itself.
+
+This directory contains the anonymized, versioned evidence behind the Doc Bridge performance narrative. It is designed to let contributors inspect the numbers without exposing repository contents, private paths, prompts, credentials, or raw agent responses.
+
+## Executive summary
+
+An earlier anonymized dogfooding cycle estimated up to **99% context-payload reduction** between the scanned repository corpus and the P95 payload returned to an agent. This is a historical context-payload measurement, not a guaranteed token reduction or correctness result, and it must not be read as a provider-token result.
+
+The published controlled A/B study with 96 executions showed a directional operational signal of 18.46% fewer paired provider token-equivalent units across 46 token-complete pairs and 39.75 seconds lower P95 latency than repository-only context. Operational completion was 87.5% versus 75.0%. The sample is not a causal or enterprise-readiness claim.
+
+The independent bounded adjudicator recorded **0 adjudicator-success outcomes in both arms**. Because this adjudicator is mechanical and does not independently judge semantic correctness, this round does not demonstrate improved task correctness; it remains `inconclusive` and should be used only as an auditable historical directional measurement. A newer local pilot is not included in the public narrative until its semantic evaluation and publication review are complete.
+
+Acceptance telemetry is now checked separately from task correctness: every pilot observation records passed, total, and executed acceptance-check counts, while the deterministic adjudicator blocks a success when any declared check was not observed as executed. This prevents a missing execution measurement from being mistaken for a passing task.
+
+Semantic correctness and natural-language contradiction detection are explicit
+limitations of this study, not missing evidence hidden by the publication
+artifacts. They remain open validation work for a future adjudicated corpus.
+
+![Estimated context payload reduction](../landing/assets/context-payload-reduction.svg)
+
+![Controlled A/B comparison](../landing/assets/controlled-ab-comparison.svg)
+
+## Evidence classification
+
+| Label | Meaning |
+| --- | --- |
+| `controlled` | Fixed task, model, scenario, and measurement contract. |
+| `estimated` | Derived from bounded measurements where direct provider data was unavailable. |
+| `observational` | Real-world dogfooding evidence without a controlled counterfactual. |
+| `inconclusive` | Useful signal, but insufficient evidence for a causal claim. |
+
+## Published artifacts
+
+| Artifact | Purpose |
+| --- | --- |
+| [Protocol](./protocol-v1.json) | Metric definitions, scenarios, privacy boundary, and stopping rules. |
+| [Token efficiency protocol v2](./token-efficiency-protocol-v2.json) | Correct-action token metric, evidence gates, and non-regression targets. |
+| [Task suite](./task-suite-v1.json) | Fixed anonymized discovery, architecture, documentation, and implementation tasks. |
+| [Phase 4 public pilot suite](./phase4-public-pilot-task-suite-v1.json) | One public fixture, four tasks, and explicit pilot-only scope for validating the real runner without duplicating a repository. |
+| [Phase 4 public pilot plan](./phase4-public-pilot-run-plan-v1.json) | Hashed 16-observation pairwise pilot using the two configured Codex model slots. |
+| [Phase 4 public pilot ledger](./phase4-public-pilot-ledger-v1.json) | Bounded structured observations with hashes, statuses, provider usage, evidence IDs, and automated adjudication. |
+| [Phase 4 public pilot result](./phase4-public-pilot-result-v1.json) | Anonymized execution counts, paired token/latency measurements, failed preparation runs, and limitations. |
+| [Publication gate v1](./publication-gate-v1.md) | Evidence and claim-boundary checks required before publishing study results. |
+| [A/B baseline result](./ab-baseline-result-v1.json) | Controlled repository-only versus deterministic Doc Bridge measurements. |
+| [A/B baseline analysis](./ab-baseline-analysis-v1.md) | Human-readable interpretation of the controlled baseline. |
+| [Adjudicated A/B result](./ab-adjudicated-cost-result-v1.json) | Cost-attributed execution and independent bounded adjudication. |
+| [Round 2](./round-2-expanded-validation-v1.md) | Expanded adjudication and operational measurements. |
+| [Round 3](./round-3-evidence-contract-v1.md) | Evidence-coverage contract validation. |
+| [Round 4](./round-4-confirmation-v1.md) | Fresh-sample confirmation and final limitations. |
+| [Historical evidence](./historical-evidence-v1.json) | Earlier anonymized dogfooding and validation-cycle measurements. |
+
+All publication-bound artifacts pass the deterministic privacy gate. The values are intentionally anonymized and should be interpreted together with their provenance and limitations.
+
+The Phase 4 public pilot is intentionally separate from the six-population canonical suite. It is an execution-path and measurement smoke study, not evidence of enterprise-wide documentation quality or generalization. Its local provider and repository configuration stay outside the publication artifacts; only hashes, bounded observations, and the run identifier belong in published evidence.
+
+## Reproduce the charts
+
+From the repository root:
+
+```bash
+node scripts/generate-study-charts.mjs
+node scripts/generate-study-charts.mjs --check
+```
+
+The generator uses only the checked-in JSON artifacts and Node.js APIs. It does not call an LLM or access a consumer repository.
+
+## What the study does not prove
+
+- It does not prove that Doc Bridge makes every agent answer correctly.
+- It does not prove a universal 99% token reduction.
+- It does not establish causality across all repositories, languages, or models.
+- It does not replace semantic human review of documentation/code contradictions.
+
+The study is a transparent measurement baseline and a contribution surface for better analyzers, adapters, acceptance checks, and future replicates.

package/docs/study/ab-adjudicated-cost-analysis-v1.md

@@ -0,0 +1,29 @@
+# Controlled A/B study — adjudicated cost round
+
+This round compares `repository-only` with `deterministic-doc-bridge` using 24 anonymized task definitions, two configured CLI models, one replicate, and 48 observations per arm. The run completed all 96 planned observations and records execution failures rather than dropping them.
+
+## Result
+
+The deterministic Doc Bridge arm completed 42/48 executions (87.5%) versus 36/48 (75.0%) for repository-only. Its latency p95 was 84,440 ms versus 124,193 ms. Across 46 paired observations with token data, the deterministic arm used 53,325 fewer provider token-equivalent units on average (−18.459%). Provider token-equivalent units are input plus output tokens; they are not currency.
+
+Evidence citation was unchanged at 93.75%. The high-quality evidence rate was 20.833% for deterministic Doc Bridge versus 22.917% for repository-only. The independent bounded adjudicator recorded zero successful outcomes in both arms. That means this run does not demonstrate improved semantic correctness, even though it shows directional operational improvement.
+
+The result is `inconclusive`. It is useful as an auditable directional measurement, not as a causal, enterprise-readiness, or USD-cost claim.
+
+## Provenance
+
+- Plan: `ab-adjudicated-cost-plan-v2-v1.json`, hash `f61bf70063fcceb9dfe823ac9d0707f0f004a075143cae6e9518523c64ba1b4f`
+- Run: `phase-9-ab-adjudicated-cost-03`
+- Ledger hash: `cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5`
+- Metrics report hash: `d2849c3515da91385af91e9a80b70fa25b5f2ad6f248255229aba6ce73b3fd22`
+- Structured result: [ab-adjudicated-cost-result-v1.json](./ab-adjudicated-cost-result-v1.json)
+
+## What changed from the previous round
+
+- Provider token totals are now derived and labeled as token-equivalent units when both input and output counts exist.
+- Provider-reported outcomes are no longer treated as the final outcome; the runner records a deterministic bounded adjudication.
+- Child process groups are terminated on timeout or output limits, preventing nested CLI processes from surviving a failed observation.
+
+## Remaining measurement gap
+
+The adjudicator is mechanical and bounded. It does not independently judge whether the answer is semantically correct, whether documentation claims are true, or whether a repository change is safe. A future round needs a blinded human or separately configured review agent with an explicit rubric before correctness or quality claims can be strengthened.

package/docs/study/ab-adjudicated-cost-plan-v1.json

@@ -0,0 +1,33 @@
+{
+  "type": "controlled-study-run-plan",
+  "schemaVersion": 1,
+  "planVersion": "v1-ab-adjudicated-cost",
+  "protocolVersion": "v1",
+  "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
+  "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
+  "sourceRevisionHash": "0dfd7a83cfdfaecd78055642f74ce6552bfd83cf14f7bc6ed308ec978eda0ef6",
+  "configurationHash": "0a7dede28ccf1e672044c7b429a67d64d34e68c133dd21ffc07d763ff50642fe",
+  "docBridgeVersion": "1.7.45",
+  "models": [
+    { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
+    { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
+  ],
+  "scenarios": [
+    { "id": "repository-only", "network": false },
+    { "id": "deterministic-doc-bridge", "network": false },
+    { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
+  ],
+  "taskIds": [
+    "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
+    "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
+    "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
+    "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
+    "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
+    "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
+  ],
+  "sampling": { "strategy": "pairwise-task-strata", "sampleSize": 96, "scenarioIds": ["repository-only", "deterministic-doc-bridge"] },
+  "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
+  "runId": "phase-9-ab-adjudicated-cost-01",
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "7d03ca649d4d4d23cc9c7cb7237aba6e51697d04d2d8ac1912101ccb439b08b8"
+}

package/docs/study/ab-adjudicated-cost-plan-v2-v1.json

@@ -0,0 +1,33 @@
+{
+  "type": "controlled-study-run-plan",
+  "schemaVersion": 1,
+  "planVersion": "v1-ab-adjudicated-cost",
+  "protocolVersion": "v1",
+  "protocolHash": "7a56ae4da91ae56fb8b4b7e506adce525ab6fcb637d635cafb7091c2bd437adf",
+  "taskSuiteHash": "d7810828bce3c6616f00897fd7c92dfa4ecdf12e549923795cd95cc81a2b0029",
+  "sourceRevisionHash": "c4a2decdccfd7262f64c78b6d8bb1c1101094d0afed519750b8ebfbfde1c1019",
+  "configurationHash": "0a7dede28ccf1e672044c7b429a67d64d34e68c133dd21ffc07d763ff50642fe",
+  "docBridgeVersion": "1.7.45",
+  "models": [
+    { "id": "low-cost-model", "role": "low-cost", "provider": "codex", "model": "gpt-5.6-sol", "version": "codex-cli-0.149.0", "parametersHash": "6dc5481139cc4fc6960a38ffd4c7e68605108d0d3eac9ac3c269c92703200273", "contextLimit": 272000, "toolConfigurationHash": "a08b38a4fa7e559811713cabc2e2f62cb05ef266d406dcb38cf46be627417e20", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" },
+    { "id": "reference-model", "role": "reference", "provider": "codex", "model": "gpt-5.6-luna", "version": "codex-cli-0.149.0", "parametersHash": "785593a535c713ae8015defb031047fb9ae55926ec4ac0cb0cafcdaa5cba1e30", "contextLimit": 272000, "toolConfigurationHash": "d2fe8bbf1fa6add2b357c4a199a37224c03bbe4edf29901ff45ae0ef8ee99366", "promptContractHash": "5ba19f0e4ea57c4206d2965a55fc1bbd081348a8ccf89c86ff64a81df93bf98b" }
+  ],
+  "scenarios": [
+    { "id": "repository-only", "network": false },
+    { "id": "deterministic-doc-bridge", "network": false },
+    { "id": "registry-assisted", "agentId": "ecosystem-doc-bridge-corpus-scanner", "agentVersion": "v1.0.0", "network": false }
+  ],
+  "taskIds": [
+    "consumer-01-discovery", "consumer-01-architecture", "consumer-01-documentation", "consumer-01-implementation",
+    "consumer-02-discovery", "consumer-02-architecture", "consumer-02-documentation", "consumer-02-implementation",
+    "consumer-03-discovery", "consumer-03-architecture", "consumer-03-documentation", "consumer-03-implementation",
+    "consumer-04-discovery", "consumer-04-architecture", "consumer-04-documentation", "consumer-04-implementation",
+    "consumer-05-discovery", "consumer-05-architecture", "consumer-05-documentation", "consumer-05-implementation",
+    "consumer-06-discovery", "consumer-06-architecture", "consumer-06-documentation", "consumer-06-implementation"
+  ],
+  "sampling": { "strategy": "pairwise-task-strata", "sampleSize": 96, "scenarioIds": ["repository-only", "deterministic-doc-bridge"] },
+  "budget": { "maxTokens": 400000, "maxRuntimeMs": 180000, "maxOutputBytes": 256000, "maxAttempts": 1 },
+  "runId": "phase-9-ab-adjudicated-cost-03",
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "f61bf70063fcceb9dfe823ac9d0707f0f004a075143cae6e9518523c64ba1b4f"
+}

package/docs/study/ab-adjudicated-cost-result-v1.json

@@ -0,0 +1,80 @@
+{
+  "type": "controlled-study-ab-result",
+  "schemaVersion": 1,
+  "resultVersion": "v1",
+  "round": "ab-adjudicated-cost-2026-08-31",
+  "runId": "phase-9-ab-adjudicated-cost-03",
+  "planHash": "f61bf70063fcceb9dfe823ac9d0707f0f004a075143cae6e9518523c64ba1b4f",
+  "ledgerHash": "cc3eb8a816f961b4a4d8fdefd96365e6b89f3fd7d5aad287c3c5cc79a80ed0f5",
+  "metricsReportHash": "d2849c3515da91385af91e9a80b70fa25b5f2ad6f248255229aba6ce73b3fd22",
+  "sample": {
+    "taskCount": 24,
+    "modelCount": 2,
+    "scenarioCount": 2,
+    "observationsPerArm": 48,
+    "pairedComparisons": 48
+  },
+  "arms": [
+    {
+      "scenarioId": "repository-only",
+      "observationCount": 48,
+      "executionStatus": { "completed": 36, "budget-exceeded": 11, "timed-out": 1 },
+      "completedRate": 0.75,
+      "taskOutcome": { "success": 3, "partial": 34, "blocked": 10, "missing": 1 },
+      "adjudicationOutcome": { "success": 0, "partial": 33, "blocked": 13, "incomplete": 2 },
+      "adjudicatedSuccessRate": 0,
+      "evidenceCitationRate": 0.9375,
+      "evidenceQualityRate": 0.229167,
+      "providerTokens": 13416254,
+      "providerTokenObservations": 47,
+      "providerTokenCostUnits": 13416254,
+      "latencyMeanMs": 68064.75,
+      "latencyP95Ms": 124193,
+      "totalCostUsd": null
+    },
+    {
+      "scenarioId": "deterministic-doc-bridge",
+      "observationCount": 48,
+      "executionStatus": { "completed": 42, "budget-exceeded": 5, "timed-out": 1 },
+      "completedRate": 0.875,
+      "taskOutcome": { "success": 1, "partial": 29, "blocked": 17, "missing": 1 },
+      "adjudicationOutcome": { "success": 0, "partial": 40, "blocked": 6, "incomplete": 2 },
+      "adjudicatedSuccessRate": 0,
+      "evidenceCitationRate": 0.9375,
+      "evidenceQualityRate": 0.208333,
+      "providerTokens": 11048857,
+      "providerTokenObservations": 47,
+      "providerTokenCostUnits": 11048857,
+      "latencyMeanMs": 58335.292,
+      "latencyP95Ms": 84440,
+      "totalCostUsd": null
+    }
+  ],
+  "pairedDeltas": {
+    "comparisonDirection": "deterministic-doc-bridge minus repository-only",
+    "providerTokenCostUnitsAverage": -53325.065,
+    "providerTokenCostUnitsRelative": -0.18459,
+    "providerTokenPairCount": 46,
+    "latencyAverageMs": -9729.458,
+    "latencyRelative": -0.142944,
+    "latencyP95Ms": -39753,
+    "completedRate": 0.125,
+    "evidenceCitationRate": 0,
+    "evidenceQualityRate": -0.020834,
+    "adjudicatedSuccessRate": 0
+  },
+  "interpretation": {
+    "classification": "inconclusive",
+    "summary": "The deterministic Doc Bridge arm completed more executions and used fewer provider token-equivalent units with lower latency p95 in this controlled sample. Evidence citation was unchanged and high-quality evidence was slightly lower. The independent adjudicator found no successful outcome in either arm, so the study does not establish improved task correctness.",
+    "decision": "Do not use this run as a causal, enterprise-readiness, or currency-cost claim. Use it as an auditable directional measurement and as input to the next study design."
+  },
+  "limitations": [
+    "The cost metric is provider token-equivalent units, not USD; the CLI provider emitted no currency cost.",
+    "Two observations lacked provider token counts, so paired token comparison uses 46 of 48 pairs.",
+    "Adjudication is deterministic and bounded by execution status, acceptance metrics, and evidence count; it is not independent semantic review by a human or second model.",
+    "The sample has 24 task definitions, two models, one replicate, and two scenarios; it does not establish causality or generalize to other repositories.",
+    "The study task acceptance instrumentation produced no adjudicated successes; semantic correctness remains not analyzed."
+  ],
+  "contentHashAlgo": "sha256-normalized-v1",
+  "contentHash": "9589b4e36283c36face82acdacbd756b3b96e001c59f2fb0574f9b5be4ef034c"
+}