@dailephd/my-dev-kit-lab 0.4.9 → 0.5.1

package/CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
+## [0.5.1] - 2026-09-22
+
+Expanded warm-index benchmark suite.
+
+- Added the dedicated warm-index benchmark corpus `benchmarks/contracts/warm-index-benchmark-cases.json` with 12 tasks: six for `task-workflow-medium-ts` and six for `task-analytics-large-mixed`. Each project has one localized task, four cross-module tasks, and one broad-change negative control.
+- Added optional task-locality metadata (`localized`, `cross-module`, `broad-change`) to evaluation cases. Existing case files without it stay valid, invalid values are rejected, and the dedicated corpus requires it.
+- Every corpus task has an answer key with expected facts, plus expected files and symbols that must exactly match the answer key and must exist in the benchmark project.
+- `npm run verify:benchmarks` now validates the dedicated corpus, requires at least five tasks and every locality category for each warm-index benchmark project, and checks project-profile task statistics against the corpus. The medium and large/mixed profiles now record six tasks each (complexity scores 31 and 47 under the unchanged formula).
+- Regression tests prove the existing one-index-per-project warm-index runtime at six tasks per project: two index builds for the 12 tasks, task ordinals 1–6, amortized and cumulative metrics, per-project reset, strict-prefix availability, and isolation of single-task and single-project failures.
+- The existing warm-index report (schema `my-dev-kit-lab-warm-index-report-v1`) and the existing four plots cover the expanded suite: 2 projects, 12 tasks, and ordinals 1–6 per project.
+- The packed-package acceptance gate now checks that the corpus is in the tarball, readable from a clean install, and selectable through the installed CLI's existing `--cases` option.
+- No CLI command or flag, warm-index metric formula, report schema, or plot was added. The default cases file (`examples/token-savings-cases.json`) and the default my-dev-kit command are unchanged. Real-agent warm-index campaigns remain planned for v0.5.2.
+
+## [0.5.0] - 2026-09-21
+
+Warm-index reuse experiment support.
+
+- Added the experimental `warm-index-reuse` experiment plugin, registered alongside `context-strategy-comparison` and reachable through `experiment list`, `experiment describe`, and `experiment run`.
+- Built exactly one my-dev-kit index per benchmark-project group and reused it across every task in that group, with a matched `raw-full-file` baseline for each task; cases that disagree on target or source roots fail structurally instead of being indexed.
+- Separated index construction from retrieval: one measured `index` command per project, and search/lookup/slice/source against the prepared index for each task, without re-indexing.
+- Added deterministic fake-agent evaluation of each task side with context evidence, reusing the existing prompt generator, answer parser, run classifier, and correctness scorer.
+- Reported index build duration, raw context-construction and warm retrieval durations, raw and retrieved context size, estimated context tokens, amortized index build duration (build duration / task ordinal), cumulative component durations, cumulative estimated context tokens, fake-agent correctness, fake-agent total tokens, and cumulative fake-agent total tokens, each with explicit `available`/`unavailable`/`not-applicable` availability and strict-prefix cumulative rules.
+- Added a bounded warm-index section to the plugin `report.json`, `report.txt`, and `report.html`, explaining cold-start versus warm-reuse cost, with a neutral interpretation and explicit limitations. The bounded `warm-index-execution.json` records execution evidence without context text.
+- Added four warm-index plots through the existing `plots generate` command: amortized index cost, raw versus retrieved context size, fake-agent correctness, and cumulative fake-agent token usage.
+- Added the warm-index-only `experiment run --kit-command <command>` option (default `npx @dailephd/my-dev-kit@latest`), grouped `experiment run --help` options by plugin, and corrected `plots generate` help to describe supported inputs.
+- Extended the exact packed-package acceptance gate with installed warm-index execution, reports, and plots while preserving installed-package and target immutability.
+- Limitations: agent evidence comes from the deterministic fake agent only; fake-agent correctness and token totals are simulated harness evidence, not real-model results or provider billing telemetry; estimated context tokens remain a separate context-size estimate; cumulative durations are component sums, not wall-clock latency; no token-savings percentage, break-even task, winner, or ranking is calculated; the bundled cases provide one task per benchmark project; real-agent warm-index campaigns, warm-index screenshots, and gallery integration are not included.
+
 ## [0.4.9] - 2026-09-20
 
 Semantic native-select option selection for declarative browser tutorials.

package/README.md

@@ -6,13 +6,16 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.4.9 (semantic native-select option selection for declarative browser tutorials, adding the value-only `select-option` action backed by Playwright `Locator.selectOption({ value })`). The previous release, v0.4.8, added locator-anchored pointer gestures (`pointer-click` and `pointer-drag`) using normalized locator-relative fraction positions while existing `drag` remains element-to-element. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
+The latest release is v0.5.1 (expanded warm-index benchmark suite). The previous release, v0.5.0, added warm-index reuse experiment support. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
+
+The current release includes v0.5.1 expanded warm-index benchmarks. The package and installed CLI are version 0.5.1; v0.5.0 remains the previous release. v0.5.2 real-agent warm-index campaigns are planned next. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for the validated release state.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
 ## Current capabilities
 
 - **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
+- **Measure warm-index reuse (v0.5.1):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. The v0.5.1 release adds a dedicated 12-task benchmark corpus (`benchmarks/contracts/warm-index-benchmark-cases.json`: six medium and six large/mixed tasks tagged `localized`, `cross-module`, or `broad-change`) for comparing warm-index behavior as task count grows. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
 - **Validate Android projects:** run nineteen static checks by default, with Gradle operations, external tools, and network access available only through explicit opt-in flags.
@@ -153,6 +156,19 @@ npm run experiment:run -- `
   --no-screenshot
 ```
 
+Run the warm-index reuse plugin over the dedicated expanded benchmark corpus released in v0.5.1, deterministically with the fake my-dev-kit fixture (omit `--kit-command` to use the default `npx @dailephd/my-dev-kit@latest`):
+
+```bash
+npm run experiment:run -- \
+  --experiment warm-index-reuse \
+  --cases benchmarks/contracts/warm-index-benchmark-cases.json \
+  --kit-command "node tests/fixtures/fake-my-dev-kit-cli.js" \
+  --out lab-output/warm-index-reuse
+npm run generate-experiment-plots -- --experiment lab-output/warm-index-reuse --out lab-output/warm-index-plots
+```
+
+The default cases file, `examples/token-savings-cases.json`, is a small compatibility corpus with one task per benchmark project and remains the default when `--cases` is omitted. For multi-task reuse, select the dedicated corpus explicitly as above: it runs six tasks for each of the two benchmark projects against one index per project. Add `--benchmark-project task-workflow-medium-ts` or `--case <id>` to narrow the selection.
+
 When `--target` is omitted, the experiment runs in self mode against my-dev-kit-lab. When `--target <path>` is provided, the lab remains the tool root and the target project is inspected separately. Generated experiment outputs stay under lab-controlled output directories by default, not inside the target project.
 
 ---
@@ -170,6 +186,7 @@ When `--target` is omitted, the experiment runs in self mode against my-dev-kit-
 | Plugin experiment report JSON | `lab-output/experiments/<plugin>/<target>/<run>/report.json` |
 | Plugin experiment report HTML | `lab-output/experiments/<plugin>/<target>/<run>/report.html` |
 | Plugin experiment report text | `lab-output/experiments/<plugin>/<target>/<run>/report.txt` |
+| Warm-index execution evidence | `<experiment out>/warm-index-execution.json` (plus `indexes/`, `commands/`, and `agents/` beneath the same output root) |
 | Plot data | `lab-output/<plots>/plot-data.json` |
 | SVG charts | `lab-output/<plots>/charts/*.svg` |
 | Gallery manifest | `lab-output/<gallery>/gallery-manifest.json` |
@@ -203,7 +220,8 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Claude does not expose token totals; token savings comparisons are unavailable for Claude runs
 - Codex may expose token totals but can produce timeouts or invalid-output runs
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework currently ships one plugin, `context-strategy-comparison`; future plugins such as warm-index reuse, incremental-change, and context-window scaling are not implemented yet
+- The generic experiment-plugin framework has two plugins: `context-strategy-comparison` and the v0.5.0-introduced `warm-index-reuse`. Later plugins such as incremental-change and context-window scaling are future roadmap work
+- `warm-index-reuse` uses the deterministic fake agent only; selectable Codex/Claude warm-index campaigns remain planned for v0.5.2, and warm-index screenshots/gallery integration are not implemented. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence, not real-agent evidence. Its component-duration sums are not wall-clock latency, and it calculates no token-savings percentage, break-even task, winner, or ranking
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Provider telemetry dashboards, semantic LLM judging, and cloud API billing integration are not yet implemented
 - The six new stage-context strategies have no CLI flags yet, are configured programmatically, and do not yet include plots, screenshots, or gallery integration

package/benchmarks/contracts/benchmark-project-profiles.json

@@ -581,11 +581,11 @@
         "answer-key correctness scoring"
       ],
       "taskStats": {
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 3,
-        "expectedRelevantSymbolsAverage": 4
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 3.83,
+        "expectedRelevantSymbolsAverage": 4.33
       },
-      "complexityScore": 30,
+      "complexityScore": 31,
       "complexityMetrics": {
         "fileCount": 20,
         "sourceFileCount": 11,
@@ -597,9 +597,9 @@
         "dependencyFileCount": 1,
         "internalImportCount": 17,
         "exportedSymbolEstimate": 21,
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 3,
-        "expectedRelevantSymbolsAverage": 4,
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 3.83,
+        "expectedRelevantSymbolsAverage": 4.33,
         "maxFileLines": 77,
         "averageFileLines": 26.31,
         "packageDependencyCount": 2,
@@ -839,11 +839,11 @@
         "answer-key correctness scoring"
       ],
       "taskStats": {
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 5,
-        "expectedRelevantSymbolsAverage": 5
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 4.5,
+        "expectedRelevantSymbolsAverage": 4.67
       },
-      "complexityScore": 48,
+      "complexityScore": 47,
       "complexityMetrics": {
         "fileCount": 35,
         "sourceFileCount": 22,
@@ -855,9 +855,9 @@
         "dependencyFileCount": 1,
         "internalImportCount": 28,
         "exportedSymbolEstimate": 24,
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 5,
-        "expectedRelevantSymbolsAverage": 5,
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 4.5,
+        "expectedRelevantSymbolsAverage": 4.67,
         "maxFileLines": 49,
         "averageFileLines": 18.55,
         "packageDependencyCount": 2,