@dailephd/my-dev-kit-lab 0.5.0 → 0.5.2

package/CHANGELOG.md

@@ -4,6 +4,32 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
+## [0.5.2] - 2026-09-23
+
+Warm-index real-agent campaigns.
+
+- Added three frozen `warm-index-reuse` campaign presets — `codex-full`, `claude-full`, and `codex-timeout-isolation` — each owning exactly one provider (Codex or Claude), the bundled production corpus, and project profiles. `codex-timeout-isolation` narrows to three large/mixed cases (`warm-large-health-label`, `warm-large-ts-leaderboard`, `warm-large-broad-analytics-comparison`) for provider timeout/partial-outcome exercises without the full 12-case corpus. Default per-agent timeout is 240000 ms, overridable with `--timeout-ms`.
+- Added the `experiment run --experiment warm-index-reuse` campaign surface: `--campaign <preset>`, `--include-real-agents` (required), `--case`, `--benchmark-project`, `--timeout-ms`, `--kit-command`, `--out`. Campaign mode rejects `--target`, explicit `--cases`/`--project-profiles`, and the agent-matrix flags owned by `context-strategy-comparison`. Legacy non-campaign warm-index runs are unchanged.
+- Real-agent evaluation consumes the exact `rawBaseline.contextText`/`warmRetrieval.contextText` the experiment already measured; providers never independently run my-dev-kit retrieval. The real-agent prompt explicitly forbids repository inspection, filesystem search, my-dev-kit execution, shell commands, external information, and file modification, and delivers the prompt over stdin. Each provider call runs from a neutral temporary working directory, never the repository, benchmark target, installed package, or agent artifact directory.
+- Added Codex JSONL and Claude JSON response parsing to the existing agent adapters, sharing the existing prompt/parser/classifier/scorer pipeline with the deterministic fake agent.
+- Extended provider outcome classification with `agent-unavailable`, `agent-limit-reached`, and `timeout`, alongside the existing `completed`, `failed`, `invalid-output`, and `skipped`. Warm-index infrastructure status (index build, raw/warm retrieval) and agent/provider outcome status are reported and reasoned about separately: infrastructure can report `completed` while campaign evidence is `partial` or `unavailable`.
+- Extended the warm-index report (schema unchanged: `my-dev-kit-lab-warm-index-report-v1`) with agent identity (`agent.id`, `agent.mode`) and an additive campaign summary (preset, provider, timeout, selected/scheduled/executed/not-run-for-missing-context side counts, closed provider outcome counts, `agentEvidenceStatus`, `tokenEvidenceStatus`). Generic agent metric IDs (`agent-correctness-score`, `agent-total-tokens`, `cumulative-agent-total-tokens`) and their formulas are unchanged; correctness remains deterministic answer-key scoring and missing token evidence remains explicitly unavailable, never zero, never substituted with an estimated context-token count. The existing four warm-index plots are unchanged in formula and gained provider-aware titles/labels; a v1 report from before this change still plots correctly as legacy fake-agent evidence.
+- A real-agent campaign run now automatically produces, beneath its output root: the existing report (`report.json`/`.txt`/`.html`), the same four warm-index SVG charts under `plots/`, a best-effort `report.png` (skipped without a Playwright/browser runtime — non-fatal — and reported as an unsuccessful command only on an unexpected capture failure, which still preserves the report/plots/gallery already written), and a narrow three-item campaign gallery (`gallery/gallery-manifest.json`, `gallery-index.html`: `warm-index-campaign-report`, `warm-index-campaign-plots`, `warm-index-execution`) with relative, bounded paths. Legacy non-campaign warm-index runs remain report-only. The `warm-index-reuse` plugin itself owns no presentation orchestration; the command layer sequences report → plots → screenshot → gallery.
+- Hardened `runMeasuredCommand`'s command-timeout path: a bounded fallback now resolves the measured-command promise even if a killed process's descendant outlives an intermediate shim, and its streams are explicitly detached so the calling process can exit; a provider evaluation's neutral-working-directory cleanup is now best-effort so an OS-level cleanup failure never masks already-classified agent evidence.
+- Extended the packed-package installed-execution acceptance gate (`npm run verify:packed-package`) with deterministic, cross-platform, local-only Codex/Claude provider fixtures proving, through the installed package: completed-with-tokens, token-unavailable, failed, invalid-output, agent-unavailable, agent-limit-reached, and timeout provider outcomes; the `codex-timeout-isolation` preset's continuation past a failed case; captured and skipped screenshot paths; the three-item campaign gallery; provider-argv privacy (no prompt/context in argv, frozen stdin transport flags); and target/installed-package immutability. No real Codex or Claude is ever invoked by this gate.
+
+## [0.5.1] - 2026-09-22
+
+Expanded warm-index benchmark suite.
+
+- Added the dedicated warm-index benchmark corpus `benchmarks/contracts/warm-index-benchmark-cases.json` with 12 tasks: six for `task-workflow-medium-ts` and six for `task-analytics-large-mixed`. Each project has one localized task, four cross-module tasks, and one broad-change negative control.
+- Added optional task-locality metadata (`localized`, `cross-module`, `broad-change`) to evaluation cases. Existing case files without it stay valid, invalid values are rejected, and the dedicated corpus requires it.
+- Every corpus task has an answer key with expected facts, plus expected files and symbols that must exactly match the answer key and must exist in the benchmark project.
+- `npm run verify:benchmarks` now validates the dedicated corpus, requires at least five tasks and every locality category for each warm-index benchmark project, and checks project-profile task statistics against the corpus. The medium and large/mixed profiles now record six tasks each (complexity scores 31 and 47 under the unchanged formula).
+- Regression tests prove the existing one-index-per-project warm-index runtime at six tasks per project: two index builds for the 12 tasks, task ordinals 1–6, amortized and cumulative metrics, per-project reset, strict-prefix availability, and isolation of single-task and single-project failures.
+- The existing warm-index report (schema `my-dev-kit-lab-warm-index-report-v1`) and the existing four plots cover the expanded suite: 2 projects, 12 tasks, and ordinals 1–6 per project.
+- The packed-package acceptance gate now checks that the corpus is in the tarball, readable from a clean install, and selectable through the installed CLI's existing `--cases` option.
+
 ## [0.5.0] - 2026-09-21
 
 Warm-index reuse experiment support.

package/README.md

@@ -6,16 +6,17 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.5.0 (warm-index reuse experiment support). The previous release, v0.4.9, added semantic native-select option selection for declarative browser tutorials through the value-only `select-option` action backed by Playwright `Locator.selectOption({ value })`. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
+The latest release is v0.5.2 (warm-index real-agent campaigns). The previous release, v0.5.1, added the expanded warm-index benchmark suite. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
 
-The current release includes v0.5.0 warm-index reuse. The package and installed CLI are version 0.5.0; v0.4.9 remains the previous release. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for the validated release state.
+The current release is v0.5.2, and the package and installed CLI are version 0.5.2. v0.5.1 (expanded warm-index benchmark suite) is the previous release; v0.6.0 (index freshness and changed-file detection) is next planned. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
 ## Current capabilities
 
 - **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
-- **Measure warm-index reuse (v0.5.0):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
+- **Measure warm-index reuse (v0.5.0–v0.5.1, released):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. The v0.5.1 release adds a dedicated 12-task benchmark corpus (`benchmarks/contracts/warm-index-benchmark-cases.json`: six medium and six large/mixed tasks tagged `localized`, `cross-module`, or `broad-change`) for comparing warm-index behavior as task count grows. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry.
+- **Run selectable real-agent warm-index campaigns (v0.5.2, released):** `--campaign codex-full`, `claude-full`, or `codex-timeout-isolation` (with `--include-real-agents`) runs exactly one provider — Codex or Claude — against the exact raw and warm contexts the experiment already measured, over the frozen production corpus each preset owns. Provider outcomes (`completed`, `failed`, `timeout`, `invalid-output`, `agent-unavailable`, `agent-limit-reached`, `skipped`) are classified and reported separately from warm-index infrastructure status, so a campaign can complete its indexing/retrieval work even when provider evidence is partial or unavailable. A campaign run automatically produces the report, the same four warm-index SVG charts, a best-effort report screenshot (skipped without a browser runtime, never fatal), and a three-item gallery (report, plots, bounded execution evidence). Legacy runs without `--campaign` keep the existing deterministic fake-agent, report-only behavior unchanged. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
 - **Validate Android projects:** run nineteen static checks by default, with Gradle operations, external tools, and network access available only through explicit opt-in flags.
@@ -156,18 +157,31 @@ npm run experiment:run -- `
   --no-screenshot
 ```
 
-Run the warm-index reuse plugin deterministically with the fake my-dev-kit fixture (omit `--kit-command` to use the default `npx @dailephd/my-dev-kit@latest`):
+Run the warm-index reuse plugin over the dedicated expanded benchmark corpus released in v0.5.1, deterministically with the fake my-dev-kit fixture (omit `--kit-command` to use the default `npx @dailephd/my-dev-kit@latest`):
 
 ```bash
 npm run experiment:run -- \
   --experiment warm-index-reuse \
-  --cases tests/fixtures/warm-index-reuse/multi-task-cases.json \
+  --cases benchmarks/contracts/warm-index-benchmark-cases.json \
   --kit-command "node tests/fixtures/fake-my-dev-kit-cli.js" \
   --out lab-output/warm-index-reuse
 npm run generate-experiment-plots -- --experiment lab-output/warm-index-reuse --out lab-output/warm-index-plots
 ```
 
-The bundled `examples/token-savings-cases.json` currently has one task per benchmark project, so multi-task reuse needs a cases file with several tasks per project, such as the test fixture above; an expanded bundled warm-index benchmark suite is planned.
+The default cases file, `examples/token-savings-cases.json`, is a small compatibility corpus with one task per benchmark project and remains the default when `--cases` is omitted. For multi-task reuse, select the dedicated corpus explicitly as above: it runs six tasks for each of the two benchmark projects against one index per project. Add `--benchmark-project task-workflow-medium-ts` or `--case <id>` to narrow the selection.
+
+Run a selectable real-agent warm-index campaign (v0.5.2; requires a locally available `codex` or `claude` CLI and `--include-real-agents`). A campaign preset owns its provider and corpus, so `--target`, explicit `--cases`/`--project-profiles`, and the agent-matrix flags below are rejected in campaign mode:
+
+```bash
+npm run experiment:run -- \
+  --experiment warm-index-reuse \
+  --campaign codex-full \
+  --include-real-agents \
+  --case warm-medium-complete-idempotent \
+  --out lab-output/warm-index-campaign
+```
+
+This automatically writes `report.json`/`.txt`/`.html`, the four warm-index charts under `plots/`, a best-effort `report.png`, and a three-item `gallery/` (report, plots, bounded execution evidence) beneath `--out`. `--timeout-ms` overrides the preset's default per-agent timeout (240000 ms). `codex-timeout-isolation` narrows the corpus to three larger cases for exercising provider timeout/partial-outcome handling without running the full 12-case corpus.
 
 When `--target` is omitted, the experiment runs in self mode against my-dev-kit-lab. When `--target <path>` is provided, the lab remains the tool root and the target project is inspected separately. Generated experiment outputs stay under lab-controlled output directories by default, not inside the target project.
 
@@ -187,6 +201,7 @@ When `--target` is omitted, the experiment runs in self mode against my-dev-kit-
 | Plugin experiment report HTML | `lab-output/experiments/<plugin>/<target>/<run>/report.html` |
 | Plugin experiment report text | `lab-output/experiments/<plugin>/<target>/<run>/report.txt` |
 | Warm-index execution evidence | `<experiment out>/warm-index-execution.json` (plus `indexes/`, `commands/`, and `agents/` beneath the same output root) |
+| Warm-index campaign presentation (v0.5.2) | `<experiment out>/plots/` (four charts), `<experiment out>/report.png` (best-effort), `<experiment out>/gallery/gallery-manifest.json` and `gallery-index.html` — automatic for `--campaign` runs only |
 | Plot data | `lab-output/<plots>/plot-data.json` |
 | SVG charts | `lab-output/<plots>/charts/*.svg` |
 | Gallery manifest | `lab-output/<gallery>/gallery-manifest.json` |
@@ -217,11 +232,11 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 ## Current limitations
 
 - Token savings shown in fake-agent runs are based on estimated character counts, not provider billing telemetry
-- Claude does not expose token totals; token savings comparisons are unavailable for Claude runs
-- Codex may expose token totals but can produce timeouts or invalid-output runs
+- Claude and Codex token totals are used only when the CLI/run exposes them; missing provider token evidence stays explicitly unavailable (never zero, and never substituted with an estimated context-token count)
+- Codex and Claude campaign runs can produce timeouts, invalid-output, agent-unavailable, or agent-limit-reached outcomes; these are structured provider states, not warm-index infrastructure failures
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework has two plugins: `context-strategy-comparison` and the current-release v0.5.0 `warm-index-reuse`. Later plugins such as incremental-change and context-window scaling are future roadmap work
-- `warm-index-reuse` uses the deterministic fake agent only; selectable Codex/Claude warm-index campaigns remain planned for v0.5.2, the expanded benchmark suite remains planned for v0.5.1, and warm-index screenshots/gallery integration are not implemented. Its component-duration sums are not wall-clock latency, and it calculates no token-savings percentage, break-even task, winner, or ranking
+- The generic experiment-plugin framework has two plugins: `context-strategy-comparison` and the v0.5.0-introduced `warm-index-reuse`. Later plugins such as incremental-change and context-window scaling are future roadmap work
+- `warm-index-reuse` without `--campaign` still uses the deterministic fake agent only. With `--campaign`, exactly one real provider (Codex or Claude) runs per campaign preset — there is no multi-agent matrix, retry, provider switching, or scheduler. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence when run without `--campaign`. No warm-index mode calculates a token-savings percentage, break-even task, winner, or ranking, and component-duration sums are not wall-clock latency
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Provider telemetry dashboards, semantic LLM judging, and cloud API billing integration are not yet implemented
 - The six new stage-context strategies have no CLI flags yet, are configured programmatically, and do not yet include plots, screenshots, or gallery integration
@@ -231,6 +246,7 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - v0.4.7 added declarative browser tutorial validation and execution with bounded actions/assertions, persistent Chromium sessions, WebM, screenshots, SRT/VTT, Markdown, and `tutorial-manifest.json`. Chromium is a separate local runtime prerequisite; the package does not download browser binaries automatically. Gallery consumption remains future scope.
 - v0.4.8 added `pointer-click` and `pointer-drag` for normalized fraction positions inside a single located element. Existing `drag` remains element-to-element between source and target locators.
 - v0.4.9 `select-option` identifies an option by HTML value only: label selection, index selection, multi-select arrays, and automatic selector fallback are not supported, and there is no keyboard (`ArrowDown`/`Enter`) fallback. Native popup/menu traversal is not animated. Chromium remains a separately installed local runtime prerequisite; neither package installation nor tutorial execution downloads it automatically.
+- v0.5.2 (released) selectable real-agent warm-index campaigns run exactly one provider per preset (`codex-full`, `claude-full`, `codex-timeout-isolation`); there is no retry, provider switching, or scheduler. Correctness remains deterministic answer-key scoring, not semantic LLM judging. No composite score, winner, ranking, or break-even task is calculated for campaign or legacy warm-index runs.
 
 ---
 

package/benchmarks/contracts/benchmark-project-profiles.json

@@ -581,11 +581,11 @@
         "answer-key correctness scoring"
       ],
       "taskStats": {
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 3,
-        "expectedRelevantSymbolsAverage": 4
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 3.83,
+        "expectedRelevantSymbolsAverage": 4.33
       },
-      "complexityScore": 30,
+      "complexityScore": 31,
       "complexityMetrics": {
         "fileCount": 20,
         "sourceFileCount": 11,
@@ -597,9 +597,9 @@
         "dependencyFileCount": 1,
         "internalImportCount": 17,
         "exportedSymbolEstimate": 21,
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 3,
-        "expectedRelevantSymbolsAverage": 4,
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 3.83,
+        "expectedRelevantSymbolsAverage": 4.33,
         "maxFileLines": 77,
         "averageFileLines": 26.31,
         "packageDependencyCount": 2,
@@ -839,11 +839,11 @@
         "answer-key correctness scoring"
       ],
       "taskStats": {
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 5,
-        "expectedRelevantSymbolsAverage": 5
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 4.5,
+        "expectedRelevantSymbolsAverage": 4.67
       },
-      "complexityScore": 48,
+      "complexityScore": 47,
       "complexityMetrics": {
         "fileCount": 35,
         "sourceFileCount": 22,
@@ -855,9 +855,9 @@
         "dependencyFileCount": 1,
         "internalImportCount": 28,
         "exportedSymbolEstimate": 24,
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 5,
-        "expectedRelevantSymbolsAverage": 5,
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 4.5,
+        "expectedRelevantSymbolsAverage": 4.67,
         "maxFileLines": 49,
         "averageFileLines": 18.55,
         "packageDependencyCount": 2,