@dailephd/my-dev-kit-lab 0.5.0 → 0.5.1

package/CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
+## [0.5.1] - 2026-09-22
+
+Expanded warm-index benchmark suite.
+
+- Added the dedicated warm-index benchmark corpus `benchmarks/contracts/warm-index-benchmark-cases.json` with 12 tasks: six for `task-workflow-medium-ts` and six for `task-analytics-large-mixed`. Each project has one localized task, four cross-module tasks, and one broad-change negative control.
+- Added optional task-locality metadata (`localized`, `cross-module`, `broad-change`) to evaluation cases. Existing case files without it stay valid, invalid values are rejected, and the dedicated corpus requires it.
+- Every corpus task has an answer key with expected facts, plus expected files and symbols that must exactly match the answer key and must exist in the benchmark project.
+- `npm run verify:benchmarks` now validates the dedicated corpus, requires at least five tasks and every locality category for each warm-index benchmark project, and checks project-profile task statistics against the corpus. The medium and large/mixed profiles now record six tasks each (complexity scores 31 and 47 under the unchanged formula).
+- Regression tests prove the existing one-index-per-project warm-index runtime at six tasks per project: two index builds for the 12 tasks, task ordinals 1–6, amortized and cumulative metrics, per-project reset, strict-prefix availability, and isolation of single-task and single-project failures.
+- The existing warm-index report (schema `my-dev-kit-lab-warm-index-report-v1`) and the existing four plots cover the expanded suite: 2 projects, 12 tasks, and ordinals 1–6 per project.
+- The packed-package acceptance gate now checks that the corpus is in the tarball, readable from a clean install, and selectable through the installed CLI's existing `--cases` option.
+- No CLI command or flag, warm-index metric formula, report schema, or plot was added. The default cases file (`examples/token-savings-cases.json`) and the default my-dev-kit command are unchanged. Real-agent warm-index campaigns remain planned for v0.5.2.
+
 ## [0.5.0] - 2026-09-21
 
 Warm-index reuse experiment support.

package/README.md

@@ -6,16 +6,16 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.5.0 (warm-index reuse experiment support). The previous release, v0.4.9, added semantic native-select option selection for declarative browser tutorials through the value-only `select-option` action backed by Playwright `Locator.selectOption({ value })`. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
+The latest release is v0.5.1 (expanded warm-index benchmark suite). The previous release, v0.5.0, added warm-index reuse experiment support. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
 
-The current release includes v0.5.0 warm-index reuse. The package and installed CLI are version 0.5.0; v0.4.9 remains the previous release. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for the validated release state.
+The current release includes v0.5.1 expanded warm-index benchmarks. The package and installed CLI are version 0.5.1; v0.5.0 remains the previous release. v0.5.2 real-agent warm-index campaigns are planned next. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for the validated release state.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
 ## Current capabilities
 
 - **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
-- **Measure warm-index reuse (v0.5.0):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
+- **Measure warm-index reuse (v0.5.1):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. The v0.5.1 release adds a dedicated 12-task benchmark corpus (`benchmarks/contracts/warm-index-benchmark-cases.json`: six medium and six large/mixed tasks tagged `localized`, `cross-module`, or `broad-change`) for comparing warm-index behavior as task count grows. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
 - **Validate Android projects:** run nineteen static checks by default, with Gradle operations, external tools, and network access available only through explicit opt-in flags.
@@ -156,18 +156,18 @@ npm run experiment:run -- `
   --no-screenshot
 ```
 
-Run the warm-index reuse plugin deterministically with the fake my-dev-kit fixture (omit `--kit-command` to use the default `npx @dailephd/my-dev-kit@latest`):
+Run the warm-index reuse plugin over the dedicated expanded benchmark corpus released in v0.5.1, deterministically with the fake my-dev-kit fixture (omit `--kit-command` to use the default `npx @dailephd/my-dev-kit@latest`):
 
 ```bash
 npm run experiment:run -- \
   --experiment warm-index-reuse \
-  --cases tests/fixtures/warm-index-reuse/multi-task-cases.json \
+  --cases benchmarks/contracts/warm-index-benchmark-cases.json \
   --kit-command "node tests/fixtures/fake-my-dev-kit-cli.js" \
   --out lab-output/warm-index-reuse
 npm run generate-experiment-plots -- --experiment lab-output/warm-index-reuse --out lab-output/warm-index-plots
 ```
 
-The bundled `examples/token-savings-cases.json` currently has one task per benchmark project, so multi-task reuse needs a cases file with several tasks per project, such as the test fixture above; an expanded bundled warm-index benchmark suite is planned.
+The default cases file, `examples/token-savings-cases.json`, is a small compatibility corpus with one task per benchmark project and remains the default when `--cases` is omitted. For multi-task reuse, select the dedicated corpus explicitly as above: it runs six tasks for each of the two benchmark projects against one index per project. Add `--benchmark-project task-workflow-medium-ts` or `--case <id>` to narrow the selection.
 
 When `--target` is omitted, the experiment runs in self mode against my-dev-kit-lab. When `--target <path>` is provided, the lab remains the tool root and the target project is inspected separately. Generated experiment outputs stay under lab-controlled output directories by default, not inside the target project.
 
@@ -220,8 +220,8 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Claude does not expose token totals; token savings comparisons are unavailable for Claude runs
 - Codex may expose token totals but can produce timeouts or invalid-output runs
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework has two plugins: `context-strategy-comparison` and the current-release v0.5.0 `warm-index-reuse`. Later plugins such as incremental-change and context-window scaling are future roadmap work
-- `warm-index-reuse` uses the deterministic fake agent only; selectable Codex/Claude warm-index campaigns remain planned for v0.5.2, the expanded benchmark suite remains planned for v0.5.1, and warm-index screenshots/gallery integration are not implemented. Its component-duration sums are not wall-clock latency, and it calculates no token-savings percentage, break-even task, winner, or ranking
+- The generic experiment-plugin framework has two plugins: `context-strategy-comparison` and the v0.5.0-introduced `warm-index-reuse`. Later plugins such as incremental-change and context-window scaling are future roadmap work
+- `warm-index-reuse` uses the deterministic fake agent only; selectable Codex/Claude warm-index campaigns remain planned for v0.5.2, and warm-index screenshots/gallery integration are not implemented. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence, not real-agent evidence. Its component-duration sums are not wall-clock latency, and it calculates no token-savings percentage, break-even task, winner, or ranking
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Provider telemetry dashboards, semantic LLM judging, and cloud API billing integration are not yet implemented
 - The six new stage-context strategies have no CLI flags yet, are configured programmatically, and do not yet include plots, screenshots, or gallery integration

package/benchmarks/contracts/benchmark-project-profiles.json

@@ -581,11 +581,11 @@
         "answer-key correctness scoring"
       ],
       "taskStats": {
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 3,
-        "expectedRelevantSymbolsAverage": 4
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 3.83,
+        "expectedRelevantSymbolsAverage": 4.33
       },
-      "complexityScore": 30,
+      "complexityScore": 31,
       "complexityMetrics": {
         "fileCount": 20,
         "sourceFileCount": 11,
@@ -597,9 +597,9 @@
         "dependencyFileCount": 1,
         "internalImportCount": 17,
         "exportedSymbolEstimate": 21,
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 3,
-        "expectedRelevantSymbolsAverage": 4,
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 3.83,
+        "expectedRelevantSymbolsAverage": 4.33,
         "maxFileLines": 77,
         "averageFileLines": 26.31,
         "packageDependencyCount": 2,
@@ -839,11 +839,11 @@
         "answer-key correctness scoring"
       ],
       "taskStats": {
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 5,
-        "expectedRelevantSymbolsAverage": 5
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 4.5,
+        "expectedRelevantSymbolsAverage": 4.67
       },
-      "complexityScore": 48,
+      "complexityScore": 47,
       "complexityMetrics": {
         "fileCount": 35,
         "sourceFileCount": 22,
@@ -855,9 +855,9 @@
         "dependencyFileCount": 1,
         "internalImportCount": 28,
         "exportedSymbolEstimate": 24,
-        "taskCount": 1,
-        "expectedRelevantFilesAverage": 5,
-        "expectedRelevantSymbolsAverage": 5,
+        "taskCount": 6,
+        "expectedRelevantFilesAverage": 4.5,
+        "expectedRelevantSymbolsAverage": 4.67,
         "maxFileLines": 49,
         "averageFileLines": 18.55,
         "packageDependencyCount": 2,

package/benchmarks/contracts/warm-index-benchmark-cases.json

@@ -0,0 +1,764 @@
+[
+  {
+    "id": "warm-medium-import-dedupe",
+    "title": "Warm-index task workflow import and dedupe retrieval",
+    "benchmarkProject": "task-workflow-medium-ts",
+    "projectProfileRef": "task-workflow-medium-ts",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-workflow-medium-ts",
+    "sourceRoots": ["src", "tests"],
+    "query": "import tasks duplicate normalized title project validation deterministic id",
+    "expectedFiles": [
+      "src/services/importTasks.ts",
+      "src/store/taskStore.ts",
+      "src/validation/taskValidation.ts"
+    ],
+    "expectedSymbols": ["importTasks", "validateImportInput", "findDuplicate", "createTask"],
+    "answerKey": {
+      "expectedFiles": [
+        "src/services/importTasks.ts",
+        "src/store/taskStore.ts",
+        "src/validation/taskValidation.ts"
+      ],
+      "expectedSymbols": ["importTasks", "validateImportInput", "findDuplicate", "createTask"],
+      "expectedFacts": [
+        {
+          "id": "warm-medium-import-validates-title-and-project",
+          "text": "importTasks validates normalized task titles and project ids before inserting records.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-import-skips-duplicates-by-project-and-normalized-title",
+          "text": "importTasks skips duplicates when a task with the same normalized title already exists in the same project.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-import-keeps-deterministic-task-sequences",
+          "text": "New imported tasks still receive deterministic ids such as task-1 and task-2 from the shared store sequence.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 3,
+      "notes": "Cross-module retrieval spanning the import service, validation, and store logic."
+    },
+    "rawIncludeGlobs": ["src/**/*", "tests/**/*"],
+    "notes": "v0.5.1 warm-index seed case grounded in the existing task-workflow-import-dedup behavior."
+  },
+  {
+    "id": "warm-medium-create-project-task",
+    "title": "Warm-index create task with project metadata",
+    "benchmarkProject": "task-workflow-medium-ts",
+    "projectProfileRef": "task-workflow-medium-ts",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-workflow-medium-ts",
+    "sourceRoots": ["src", "tests"],
+    "query": "create task validate project title priority tags normalize tags deterministic id",
+    "expectedFiles": [
+      "src/services/createTask.ts",
+      "src/validation/taskValidation.ts",
+      "src/store/taskStore.ts",
+      "src/utils/deterministicId.ts"
+    ],
+    "expectedSymbols": [
+      "createTask",
+      "validateTaskTitle",
+      "validateProjectId",
+      "normalizeTags",
+      "TaskWorkflowStore",
+      "createDeterministicId"
+    ],
+    "answerKey": {
+      "expectedFiles": [
+        "src/services/createTask.ts",
+        "src/validation/taskValidation.ts",
+        "src/store/taskStore.ts",
+        "src/utils/deterministicId.ts"
+      ],
+      "expectedSymbols": [
+        "createTask",
+        "validateTaskTitle",
+        "validateProjectId",
+        "normalizeTags",
+        "TaskWorkflowStore",
+        "createDeterministicId"
+      ],
+      "expectedFacts": [
+        {
+          "id": "warm-medium-create-validates-title",
+          "text": "createTask validates the title with validateTaskTitle, which collapses whitespace and rejects an empty title with \"Task title must not be empty.\"",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-create-validates-project-before-storing",
+          "text": "createTask validates the projectId with validateProjectId and throws \"Unknown project id: <id>\" when the store has no such project, before the task is stored.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-create-normalizes-tags",
+          "text": "Tags are normalized with normalizeTags (trimmed, lowercased, empty values dropped, de-duplicated, and sorted) before the task is stored.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-create-uses-deterministic-sequence",
+          "text": "TaskWorkflowStore.createTask assigns the id with createDeterministicId(\"task\", sequence) from the store's shared incrementing sequence, producing ids such as task-1 and task-2.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-create-starts-incomplete",
+          "text": "A newly created task is stored with completed set to false.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 4,
+      "notes": "Cross-module: the create service composes validation, the store, and the deterministic id utility."
+    },
+    "rawIncludeGlobs": ["src/**/*", "tests/**/*"],
+    "notes": "v0.5.1 warm-index medium case for task creation."
+  },
+  {
+    "id": "warm-medium-complete-idempotent",
+    "title": "Warm-index idempotent task completion",
+    "benchmarkProject": "task-workflow-medium-ts",
+    "projectProfileRef": "task-workflow-medium-ts",
+    "taskLocality": "localized",
+    "targetRoot": "benchmarks/projects/task-workflow-medium-ts",
+    "sourceRoots": ["src", "tests"],
+    "query": "complete task by id idempotent completedAt unknown task",
+    "expectedFiles": ["src/services/completeTask.ts", "src/store/taskStore.ts"],
+    "expectedSymbols": ["completeTask", "TaskWorkflowStore"],
+    "answerKey": {
+      "expectedFiles": ["src/services/completeTask.ts", "src/store/taskStore.ts"],
+      "expectedSymbols": ["completeTask", "TaskWorkflowStore"],
+      "expectedFacts": [
+        {
+          "id": "warm-medium-complete-targets-task-by-id",
+          "text": "completeTask(store, taskId) changes one task by id through TaskWorkflowStore.updateTask.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-complete-sets-completed-at",
+          "text": "The first completion sets completed to true and completedAt to \"2026-02-01T00:00:00.000Z\".",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-complete-is-idempotent",
+          "text": "When the task is already completed the updater returns it unchanged, so completing it again does not rewrite completedAt.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-complete-unknown-task-throws",
+          "text": "An unknown task id throws \"Unknown task id: <id>\" from TaskWorkflowStore.updateTask.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 3,
+      "notes": "Localized: the behavior lives in one service plus the store update primitive."
+    },
+    "rawIncludeGlobs": ["src/**/*", "tests/**/*"],
+    "notes": "v0.5.1 warm-index medium localized case."
+  },
+  {
+    "id": "warm-medium-composite-filter",
+    "title": "Warm-index composite task filtering",
+    "benchmarkProject": "task-workflow-medium-ts",
+    "projectProfileRef": "task-workflow-medium-ts",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-workflow-medium-ts",
+    "sourceRoots": ["src", "tests"],
+    "query": "filter tasks project completed priority tag text query ordering",
+    "expectedFiles": ["src/services/filterTasks.ts", "src/models/task.ts", "src/store/taskStore.ts"],
+    "expectedSymbols": ["filterTasks", "TaskFilter", "TaskWorkflowStore"],
+    "answerKey": {
+      "expectedFiles": ["src/services/filterTasks.ts", "src/models/task.ts", "src/store/taskStore.ts"],
+      "expectedSymbols": ["filterTasks", "TaskFilter", "TaskWorkflowStore"],
+      "expectedFacts": [
+        {
+          "id": "warm-medium-filter-combines-active-predicates",
+          "text": "filterTasks keeps a task only when it satisfies every supplied TaskFilter field; fields that are not supplied do not filter.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-filter-project",
+          "text": "The projectId filter requires task.projectId to equal the requested project id.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-filter-completed-priority-tag",
+          "text": "The completed filter compares the boolean when defined, priority must match exactly, and the tag is trimmed and lowercased before checking task.tags.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-filter-text-query",
+          "text": "The query is trimmed and lowercased and matched as a substring of the lowercased title, notes, and tags joined with spaces.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-filter-read-only-ordering",
+          "text": "filterTasks only reads store.listTasks(), which returns task copies sorted by task id, and returns the matches in that order without mutating the store.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 4,
+      "notes": "Cross-module: filter semantics depend on the TaskFilter model and store listing order."
+    },
+    "rawIncludeGlobs": ["src/**/*", "tests/**/*"],
+    "notes": "v0.5.1 warm-index medium case for composite filtering."
+  },
+  {
+    "id": "warm-medium-project-summary",
+    "title": "Warm-index workflow project summary",
+    "benchmarkProject": "task-workflow-medium-ts",
+    "projectProfileRef": "task-workflow-medium-ts",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-workflow-medium-ts",
+    "sourceRoots": ["src", "tests"],
+    "query": "summarize tasks total open completed per project high priority open",
+    "expectedFiles": [
+      "src/services/summarizeTasks.ts",
+      "src/store/taskStore.ts",
+      "src/models/task.ts",
+      "src/models/project.ts"
+    ],
+    "expectedSymbols": ["summarizeTasks", "TaskSummary", "TaskWorkflowStore", "WorkflowProject"],
+    "answerKey": {
+      "expectedFiles": [
+        "src/services/summarizeTasks.ts",
+        "src/store/taskStore.ts",
+        "src/models/task.ts",
+        "src/models/project.ts"
+      ],
+      "expectedSymbols": ["summarizeTasks", "TaskSummary", "TaskWorkflowStore", "WorkflowProject"],
+      "expectedFacts": [
+        {
+          "id": "warm-medium-summary-total-open-completed",
+          "text": "summarizeTasks reports total, completed, and open counts across all tasks, where open equals total minus completed.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-summary-by-project",
+          "text": "byProject holds total, completed, and open counts for every project returned by store.listProjects(), including zero counts for projects without tasks, plus any task projectId not registered as a project.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-summary-high-priority-open",
+          "text": "highPriorityOpenTitles lists the titles of tasks that are not completed and have priority \"high\".",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-summary-read-only",
+          "text": "summarizeTasks reads copies from store.listTasks() and store.listProjects() and does not mutate stored tasks.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 3,
+      "notes": "Cross-module: the summary combines the store with task and project models."
+    },
+    "rawIncludeGlobs": ["src/**/*", "tests/**/*"],
+    "notes": "v0.5.1 warm-index medium case for project summaries."
+  },
+  {
+    "id": "warm-medium-broad-workflow-map",
+    "title": "Warm-index broad task workflow architecture map",
+    "benchmarkProject": "task-workflow-medium-ts",
+    "projectProfileRef": "task-workflow-medium-ts",
+    "taskLocality": "broad-change",
+    "targetRoot": "benchmarks/projects/task-workflow-medium-ts",
+    "sourceRoots": ["src", "tests"],
+    "query": "explain task creation import duplicate filtering completion summary validation store workflow",
+    "expectedFiles": [
+      "src/services/createTask.ts",
+      "src/services/importTasks.ts",
+      "src/services/filterTasks.ts",
+      "src/services/completeTask.ts",
+      "src/services/summarizeTasks.ts",
+      "src/store/taskStore.ts",
+      "src/validation/taskValidation.ts"
+    ],
+    "expectedSymbols": [
+      "createTask",
+      "importTasks",
+      "filterTasks",
+      "completeTask",
+      "summarizeTasks",
+      "TaskWorkflowStore",
+      "validateImportInput"
+    ],
+    "answerKey": {
+      "expectedFiles": [
+        "src/services/createTask.ts",
+        "src/services/importTasks.ts",
+        "src/services/filterTasks.ts",
+        "src/services/completeTask.ts",
+        "src/services/summarizeTasks.ts",
+        "src/store/taskStore.ts",
+        "src/validation/taskValidation.ts"
+      ],
+      "expectedSymbols": [
+        "createTask",
+        "importTasks",
+        "filterTasks",
+        "completeTask",
+        "summarizeTasks",
+        "TaskWorkflowStore",
+        "validateImportInput"
+      ],
+      "expectedFacts": [
+        {
+          "id": "warm-medium-broad-shared-storage",
+          "text": "Both the createTask service and importTasks store new tasks through TaskWorkflowStore.createTask, sharing its deterministic task id sequence.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-broad-import-adds-dedupe",
+          "text": "importTasks validates each item with validateImportInput and skips duplicates found by TaskWorkflowStore.findDuplicate (same project and normalized title), recording their ids in skippedDuplicates; plain task creation does not deduplicate.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-broad-filter-read-only",
+          "text": "filterTasks reads store.listTasks() and returns matching tasks without changing workflow state.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-broad-complete-targeted",
+          "text": "completeTask changes a single task by id through TaskWorkflowStore.updateTask.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-medium-broad-summary-aggregates",
+          "text": "summarizeTasks aggregates counts from the current tasks and projects held by the store.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "forbiddenWrongClaims": ["importTasks calls the createTask service to insert imported tasks."],
+      "minimumCorrectFacts": 4,
+      "notes": "Deliberately wide-scope negative control spanning every workflow service; it is not expected to fit a small localized context."
+    },
+    "rawIncludeGlobs": ["src/**/*", "tests/**/*"],
+    "notes": "v0.5.1 warm-index medium broad-change negative control."
+  },
+  {
+    "id": "warm-large-health-label",
+    "title": "Warm-index task analytics health label retrieval",
+    "benchmarkProject": "task-analytics-large-mixed",
+    "projectProfileRef": "task-analytics-large-mixed",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-analytics-large-mixed",
+    "sourceRoots": ["ts/src", "ts/tests", "py/task_analytics", "py/tests"],
+    "query": "completion rate stale tasks quality label health report analytics snapshot",
+    "expectedFiles": [
+      "ts/src/services/buildAnalyticsSnapshot.ts",
+      "ts/src/reporting/formatTaskHealthReport.ts",
+      "py/task_analytics/metrics.py",
+      "py/task_analytics/quality.py",
+      "py/task_analytics/reporting.py"
+    ],
+    "expectedSymbols": [
+      "buildAnalyticsSnapshot",
+      "formatTaskHealthReport",
+      "calculate_project_metrics",
+      "determine_quality_label",
+      "build_health_report"
+    ],
+    "answerKey": {
+      "expectedFiles": [
+        "ts/src/services/buildAnalyticsSnapshot.ts",
+        "ts/src/reporting/formatTaskHealthReport.ts",
+        "py/task_analytics/metrics.py",
+        "py/task_analytics/quality.py",
+        "py/task_analytics/reporting.py"
+      ],
+      "expectedSymbols": [
+        "buildAnalyticsSnapshot",
+        "formatTaskHealthReport",
+        "calculate_project_metrics",
+        "determine_quality_label",
+        "build_health_report"
+      ],
+      "expectedFacts": [
+        {
+          "id": "warm-large-snapshot-computes-completion-rate",
+          "text": "The analytics snapshot computes completionRate as completedTasks divided by totalTasks times 100, rounded to two decimals.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-snapshot-counts-stale-open-tasks",
+          "text": "Stale tasks count only open tasks whose updated day is at least ten days behind the current day.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-python-quality-labels-healthy-watch-risk",
+          "text": "The Python quality layer maps metrics to healthy, watch, or risk labels based on completion rate and stale task thresholds.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-python-reporting-renders-quality-lines",
+          "text": "The Python report writer emits one line per project with the quality label plus completion, open, and stale counts.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 3,
+      "notes": "Cross-module retrieval spanning TypeScript snapshot building and Python quality reporting."
+    },
+    "rawIncludeGlobs": ["ts/src/**/*", "ts/tests/**/*", "py/task_analytics/**/*", "py/tests/**/*"],
+    "notes": "v0.5.1 warm-index seed case grounded in the existing task-analytics-health-label behavior."
+  },
+  {
+    "id": "warm-large-ts-analytics-snapshot",
+    "title": "Warm-index TypeScript analytics snapshot",
+    "benchmarkProject": "task-analytics-large-mixed",
+    "projectProfileRef": "task-analytics-large-mixed",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-analytics-large-mixed",
+    "sourceRoots": ["ts/src", "ts/tests", "py/task_analytics", "py/tests"],
+    "query": "analytics snapshot project completion rate stale tasks average story points task store",
+    "expectedFiles": [
+      "ts/src/services/buildAnalyticsSnapshot.ts",
+      "ts/src/services/listTasksByProject.ts",
+      "ts/src/models/analyticsSnapshot.ts",
+      "ts/src/store/taskStore.ts"
+    ],
+    "expectedSymbols": ["buildAnalyticsSnapshot", "listTasksByProject", "ProjectSnapshot", "AnalyticsTaskStore"],
+    "answerKey": {
+      "expectedFiles": [
+        "ts/src/services/buildAnalyticsSnapshot.ts",
+        "ts/src/services/listTasksByProject.ts",
+        "ts/src/models/analyticsSnapshot.ts",
+        "ts/src/store/taskStore.ts"
+      ],
+      "expectedSymbols": ["buildAnalyticsSnapshot", "listTasksByProject", "ProjectSnapshot", "AnalyticsTaskStore"],
+      "expectedFacts": [
+        {
+          "id": "warm-large-snapshot-per-project-counts",
+          "text": "For each project from ProjectStore.list(), buildAnalyticsSnapshot loads the project's tasks with listTasksByProject (AnalyticsTaskStore.list() filtered by projectId) and records totalTasks, completedTasks, and openTasks as totalTasks minus completedTasks.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-snapshot-completion-rate-formula",
+          "text": "completionRate is completedTasks divided by totalTasks times 100, rounded to two decimals with Math.round(value * 100) / 100.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-snapshot-stale-condition",
+          "text": "staleTasks counts only incomplete tasks whose currentDay minus updatedDay is at least STALE_DAY_THRESHOLD (10).",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-snapshot-empty-project",
+          "text": "A project with no tasks gets zero counts, completionRate 0, and averageStoryPoints 0; otherwise averageStoryPoints is the mean story points rounded to two decimals.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-snapshot-totals-aggregate",
+          "text": "The snapshot totals sum totalTasks, completedTasks, openTasks, and staleTasks across the per-project snapshots.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 4,
+      "notes": "Cross-module: the snapshot service composes the task-listing service, the task store, and the snapshot model."
+    },
+    "rawIncludeGlobs": ["ts/src/**/*", "ts/tests/**/*", "py/task_analytics/**/*", "py/tests/**/*"],
+    "notes": "v0.5.1 warm-index large case for the TypeScript analytics snapshot."
+  },
+  {
+    "id": "warm-large-ts-leaderboard",
+    "title": "Warm-index deterministic project leaderboard",
+    "benchmarkProject": "task-analytics-large-mixed",
+    "projectProfileRef": "task-analytics-large-mixed",
+    "taskLocality": "localized",
+    "targetRoot": "benchmarks/projects/task-analytics-large-mixed",
+    "sourceRoots": ["ts/src", "ts/tests", "py/task_analytics", "py/tests"],
+    "query": "project leaderboard completion rate stale tasks tie break project id ranking",
+    "expectedFiles": ["ts/src/reporting/buildProjectLeaderboard.ts", "ts/src/models/analyticsSnapshot.ts"],
+    "expectedSymbols": ["buildProjectLeaderboard", "ProjectSnapshot"],
+    "answerKey": {
+      "expectedFiles": ["ts/src/reporting/buildProjectLeaderboard.ts", "ts/src/models/analyticsSnapshot.ts"],
+      "expectedSymbols": ["buildProjectLeaderboard", "ProjectSnapshot"],
+      "expectedFacts": [
+        {
+          "id": "warm-large-leaderboard-completion-rate-descending",
+          "text": "buildProjectLeaderboard sorts a copy of snapshot.projects by completionRate in descending order.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-leaderboard-stale-ascending",
+          "text": "Ties on completionRate are broken by staleTasks in ascending order.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-leaderboard-project-id-final",
+          "text": "Remaining ties are broken by projectId in ascending localeCompare order.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-leaderboard-one-based-ranks",
+          "text": "Each line is formatted as \"<rank>. <projectId> (<completionRate>% complete, <staleTasks> stale)\" with ranks starting at 1.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 3,
+      "notes": "Localized: the ranking rule lives in one reporting function over the ProjectSnapshot model."
+    },
+    "rawIncludeGlobs": ["ts/src/**/*", "ts/tests/**/*", "py/task_analytics/**/*", "py/tests/**/*"],
+    "notes": "v0.5.1 warm-index large localized case."
+  },
+  {
+    "id": "warm-large-python-parser-metrics",
+    "title": "Warm-index Python parser and project metrics",
+    "benchmarkProject": "task-analytics-large-mixed",
+    "projectProfileRef": "task-analytics-large-mixed",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-analytics-large-mixed",
+    "sourceRoots": ["ts/src", "ts/tests", "py/task_analytics", "py/tests"],
+    "query": "parse task rows calculate project metrics completion open stale story points",
+    "expectedFiles": ["py/task_analytics/parser.py", "py/task_analytics/models.py", "py/task_analytics/metrics.py"],
+    "expectedSymbols": ["parse_task_rows", "TaskRecord", "calculate_project_metrics", "ProjectMetrics"],
+    "answerKey": {
+      "expectedFiles": ["py/task_analytics/parser.py", "py/task_analytics/models.py", "py/task_analytics/metrics.py"],
+      "expectedSymbols": ["parse_task_rows", "TaskRecord", "calculate_project_metrics", "ProjectMetrics"],
+      "expectedFacts": [
+        {
+          "id": "warm-large-python-parser-task-record",
+          "text": "parse_task_rows converts each row into a TaskRecord with task_id and project_id as str, completed as bool, and story_points and updated_day as int.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-python-metrics-grouped-by-project",
+          "text": "calculate_project_metrics groups tasks by project_id and returns one ProjectMetrics per project in sorted project_id order.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-python-metrics-counts",
+          "text": "open_tasks is total_tasks minus completed_tasks, and stale_tasks counts incomplete tasks whose current_day minus updated_day is at least STALE_DAY_THRESHOLD (10).",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-python-metrics-completion-rate",
+          "text": "completion_rate is round((completed / total) * 100, 2), or 0.0 when a project has no tasks.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-python-metrics-average-story-points",
+          "text": "average_story_points is the mean story_points rounded with round(..., 2), or 0.0 when a project has no tasks.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 4,
+      "notes": "Cross-module: parsing, the record models, and metric calculation live in separate Python modules."
+    },
+    "rawIncludeGlobs": ["ts/src/**/*", "ts/tests/**/*", "py/task_analytics/**/*", "py/tests/**/*"],
+    "notes": "v0.5.1 warm-index large case for Python parsing and metrics."
+  },
+  {
+    "id": "warm-large-python-pipeline",
+    "title": "Warm-index Python analytics report pipeline",
+    "benchmarkProject": "task-analytics-large-mixed",
+    "projectProfileRef": "task-analytics-large-mixed",
+    "taskLocality": "cross-module",
+    "targetRoot": "benchmarks/projects/task-analytics-large-mixed",
+    "sourceRoots": ["ts/src", "ts/tests", "py/task_analytics", "py/tests"],
+    "query": "pipeline parse rows project metrics quality label health report",
+    "expectedFiles": [
+      "py/task_analytics/pipeline.py",
+      "py/task_analytics/parser.py",
+      "py/task_analytics/metrics.py",
+      "py/task_analytics/quality.py",
+      "py/task_analytics/reporting.py"
+    ],
+    "expectedSymbols": [
+      "build_report_from_rows",
+      "parse_task_rows",
+      "calculate_project_metrics",
+      "determine_quality_label",
+      "build_health_report"
+    ],
+    "answerKey": {
+      "expectedFiles": [
+        "py/task_analytics/pipeline.py",
+        "py/task_analytics/parser.py",
+        "py/task_analytics/metrics.py",
+        "py/task_analytics/quality.py",
+        "py/task_analytics/reporting.py"
+      ],
+      "expectedSymbols": [
+        "build_report_from_rows",
+        "parse_task_rows",
+        "calculate_project_metrics",
+        "determine_quality_label",
+        "build_health_report"
+      ],
+      "expectedFacts": [
+        {
+          "id": "warm-large-pipeline-parses-first",
+          "text": "build_report_from_rows(rows, current_day) first parses the rows with parse_task_rows.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-pipeline-metrics-before-report",
+          "text": "The parsed tasks go to calculate_project_metrics(tasks, current_day), and the resulting metrics go to build_health_report.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-pipeline-quality-label-in-report",
+          "text": "build_health_report calls determine_quality_label for each metric, which returns empty, healthy, watch, or risk from total_tasks, completion_rate, and stale_tasks thresholds.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-pipeline-deterministic-output",
+          "text": "The report is one line per project, \"<project_id>: <label> | completion=<rate>% | open=<n> | stale=<n>\", joined by newlines in sorted project_id order, so fixed rows and current_day produce the same output.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-pipeline-composes-owners",
+          "text": "pipeline.py only composes parse_task_rows, calculate_project_metrics, and build_health_report; quality rules stay in quality.py and are reached through reporting.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "minimumCorrectFacts": 4,
+      "notes": "Cross-module: the pipeline composes parser, metrics, reporting, and (through reporting) quality modules."
+    },
+    "rawIncludeGlobs": ["ts/src/**/*", "ts/tests/**/*", "py/task_analytics/**/*", "py/tests/**/*"],
+    "notes": "v0.5.1 warm-index large case for the Python report pipeline."
+  },
+  {
+    "id": "warm-large-broad-analytics-comparison",
+    "title": "Warm-index broad TypeScript and Python analytics comparison",
+    "benchmarkProject": "task-analytics-large-mixed",
+    "projectProfileRef": "task-analytics-large-mixed",
+    "taskLocality": "broad-change",
+    "targetRoot": "benchmarks/projects/task-analytics-large-mixed",
+    "sourceRoots": ["ts/src", "ts/tests", "py/task_analytics", "py/tests"],
+    "query": "compare TypeScript and Python project analytics snapshot metrics quality reporting pipeline leaderboard",
+    "expectedFiles": [
+      "ts/src/services/buildAnalyticsSnapshot.ts",
+      "ts/src/reporting/formatTaskHealthReport.ts",
+      "ts/src/reporting/buildProjectLeaderboard.ts",
+      "py/task_analytics/parser.py",
+      "py/task_analytics/metrics.py",
+      "py/task_analytics/quality.py",
+      "py/task_analytics/reporting.py",
+      "py/task_analytics/pipeline.py"
+    ],
+    "expectedSymbols": [
+      "buildAnalyticsSnapshot",
+      "formatTaskHealthReport",
+      "buildProjectLeaderboard",
+      "parse_task_rows",
+      "calculate_project_metrics",
+      "determine_quality_label",
+      "build_health_report",
+      "build_report_from_rows"
+    ],
+    "answerKey": {
+      "expectedFiles": [
+        "ts/src/services/buildAnalyticsSnapshot.ts",
+        "ts/src/reporting/formatTaskHealthReport.ts",
+        "ts/src/reporting/buildProjectLeaderboard.ts",
+        "py/task_analytics/parser.py",
+        "py/task_analytics/metrics.py",
+        "py/task_analytics/quality.py",
+        "py/task_analytics/reporting.py",
+        "py/task_analytics/pipeline.py"
+      ],
+      "expectedSymbols": [
+        "buildAnalyticsSnapshot",
+        "formatTaskHealthReport",
+        "buildProjectLeaderboard",
+        "parse_task_rows",
+        "calculate_project_metrics",
+        "determine_quality_label",
+        "build_health_report",
+        "build_report_from_rows"
+      ],
+      "expectedFacts": [
+        {
+          "id": "warm-large-broad-typescript-side",
+          "text": "The TypeScript side builds analytics snapshots with buildAnalyticsSnapshot and presents them through formatTaskHealthReport, which uses buildProjectLeaderboard for ranking.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-broad-python-side",
+          "text": "The Python side parses rows (parse_task_rows), derives project metrics (calculate_project_metrics), assigns quality labels (determine_quality_label), and builds reports (build_health_report), composed by build_report_from_rows.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-broad-comparable-evidence",
+          "text": "Both sides expose comparable per-project health evidence: total, completed, open, and stale task counts plus a completion rate rounded to two decimals, using a stale threshold of 10 days.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-broad-independent-implementations",
+          "text": "The TypeScript and Python code are independent parallel benchmark implementations, not a runtime TypeScript-to-Python call chain.",
+          "weight": 1,
+          "required": true
+        },
+        {
+          "id": "warm-large-broad-no-cross-invocation",
+          "text": "Neither implementation imports, spawns, or calls the other; TypeScript imports stay within ts/src and Python imports stay within the task_analytics package.",
+          "weight": 1,
+          "required": true
+        }
+      ],
+      "forbiddenWrongClaims": [
+        "The TypeScript implementation invokes the Python analytics pipeline.",
+        "Python is a runtime backend for the TypeScript analytics implementation.",
+        "The TypeScript or Python implementation directly imports or calls the other."
+      ],
+      "minimumCorrectFacts": 4,
+      "notes": "Deliberately wide-scope negative control spanning both language implementations; it is not expected to fit a small localized context."
+    },
+    "rawIncludeGlobs": ["ts/src/**/*", "ts/tests/**/*", "py/task_analytics/**/*", "py/tests/**/*"],
+    "notes": "v0.5.1 warm-index large broad-change negative control."
+  }
+]

package/dist/src/evaluation/benchmarkMetadata.js

@@ -1,7 +1,8 @@
 import { existsSync } from "node:fs";
 import { readFile } from "node:fs/promises";
 import path from "node:path";
-import { PROJECT_COMPLEXITY_FORMULA, calculateProjectComplexityScore } from "./projectComplexity.js";
+import { PROJECT_COMPLEXITY_FORMULA, calculateProjectComplexityScore, roundToTwo } from "./projectComplexity.js";
+import { TASK_LOCALITIES } from "./types.js";
 export const REQUIRED_BENCHMARK_PROJECT_IDS = [
     "todo-ts",
     "todo-python",
@@ -11,6 +12,8 @@ export const REQUIRED_BENCHMARK_PROJECT_IDS = [
     "task-analytics-large-mixed"
 ];
 export const VALID_COMPLEXITY_LEVELS = new Set(["small", "medium", "large", "mixed-language"]);
+export const REQUIRED_WARM_INDEX_BENCHMARK_PROJECT_IDS = ["task-workflow-medium-ts", "task-analytics-large-mixed"];
+export const MIN_WARM_INDEX_TASKS_PER_PROJECT = 5;
 export async function readBenchmarkProjectProfiles(profilesPath, repoRoot = process.cwd()) {
     let parsed;
     try {
@@ -95,6 +98,193 @@ export function validateAnswerKey(answerKey, label) {
     }
     return errors;
 }
+export function validateTaskLocality(value, label) {
+    if (typeof value === "string" && TASK_LOCALITIES.includes(value)) {
+        return [];
+    }
+    return [`${label}: taskLocality must be one of ${TASK_LOCALITIES.join(", ")} (received ${JSON.stringify(value)}).`];
+}
+export function deriveWarmIndexTaskStats(cases, projectId) {
+    const projectCases = cases.filter((benchmarkCase) => benchmarkCase?.benchmarkProject === projectId);
+    if (projectCases.length === 0) {
+        return { taskCount: 0, expectedRelevantFilesAverage: 0, expectedRelevantSymbolsAverage: 0 };
+    }
+    const fileCounts = projectCases.map((benchmarkCase) => (Array.isArray(benchmarkCase.expectedFiles) ? benchmarkCase.expectedFiles.length : 0));
+    const symbolCounts = projectCases.map((benchmarkCase) => Array.isArray(benchmarkCase.expectedSymbols) ? benchmarkCase.expectedSymbols.length : 0);
+    return {
+        taskCount: projectCases.length,
+        expectedRelevantFilesAverage: roundToTwo(fileCounts.reduce((total, count) => total + count, 0) / projectCases.length),
+        expectedRelevantSymbolsAverage: roundToTwo(symbolCounts.reduce((total, count) => total + count, 0) / projectCases.length)
+    };
+}
+export function validateWarmIndexBenchmarkCases(cases, profiles, repoRoot = process.cwd()) {
+    const errors = [];
+    const profilesById = new Map(profiles.map((profile) => [profile.projectId, profile]));
+    const ids = new Set();
+    const sourceRootsByProject = new Map();
+    cases.forEach((benchmarkCase, index) => {
+        if (!benchmarkCase || typeof benchmarkCase !== "object") {
+            errors.push(`warm-index case at index ${index}: must be an object.`);
+            return;
+        }
+        const id = typeof benchmarkCase.id === "string" && benchmarkCase.id.length > 0 ? benchmarkCase.id : undefined;
+        const label = id === undefined ? `warm-index case at index ${index}` : `warm-index case ${id}`;
+        if (id === undefined) {
+            errors.push(`${label}: id must be a nonempty string.`);
+        }
+        else if (ids.has(id)) {
+            errors.push(`${label}: duplicate case id.`);
+        }
+        else {
+            ids.add(id);
+        }
+        const projectId = typeof benchmarkCase.benchmarkProject === "string" && benchmarkCase.benchmarkProject.length > 0
+            ? benchmarkCase.benchmarkProject
+            : undefined;
+        if (projectId === undefined) {
+            errors.push(`${label}: benchmarkProject must be a nonempty string.`);
+        }
+        if (typeof benchmarkCase.projectProfileRef !== "string" || benchmarkCase.projectProfileRef.length === 0) {
+            errors.push(`${label}: missing projectProfileRef.`);
+        }
+        else if (benchmarkCase.projectProfileRef !== projectId) {
+            errors.push(`${label}: projectProfileRef ${benchmarkCase.projectProfileRef} must equal benchmarkProject ${projectId ?? "<missing>"}.`);
+        }
+        const profile = projectId === undefined ? undefined : profilesById.get(projectId);
+        if (projectId !== undefined && profile === undefined) {
+            errors.push(`${label}: unknown benchmark project profile ${projectId}.`);
+        }
+        if (benchmarkCase.taskLocality === undefined) {
+            errors.push(`${label}: missing taskLocality; expected one of ${TASK_LOCALITIES.join(", ")}.`);
+        }
+        else {
+            errors.push(...validateTaskLocality(benchmarkCase.taskLocality, label));
+        }
+        const targetRoot = typeof benchmarkCase.targetRoot === "string" && benchmarkCase.targetRoot.length > 0 ? benchmarkCase.targetRoot : undefined;
+        if (targetRoot === undefined) {
+            errors.push(`${label}: targetRoot must be a nonempty string.`);
+        }
+        else if (profile !== undefined && normalizeRelativeRoot(targetRoot) !== normalizeRelativeRoot(profile.rootPath ?? "")) {
+            errors.push(`${label}: targetRoot ${targetRoot} does not match profile ${profile.projectId} rootPath ${profile.rootPath}.`);
+        }
+        if (!Array.isArray(benchmarkCase.sourceRoots) || benchmarkCase.sourceRoots.length === 0) {
+            errors.push(`${label}: sourceRoots must be a nonempty array.`);
+        }
+        else {
+            if (profile !== undefined) {
+                errors.push(...validateWarmIndexSourceRoots(benchmarkCase.sourceRoots, profile, label));
+            }
+            if (projectId !== undefined) {
+                const reference = sourceRootsByProject.get(projectId);
+                if (reference === undefined) {
+                    sourceRootsByProject.set(projectId, { caseLabel: label, sourceRoots: benchmarkCase.sourceRoots });
+                }
+                else if (!sameOrderedValues(reference.sourceRoots, benchmarkCase.sourceRoots)) {
+                    errors.push(`${label}: sourceRoots ${JSON.stringify(benchmarkCase.sourceRoots)} must match the ordered sourceRoots ${JSON.stringify(reference.sourceRoots)} of ${reference.caseLabel} for project ${projectId}.`);
+                }
+            }
+        }
+        const expectedFiles = Array.isArray(benchmarkCase.expectedFiles) ? benchmarkCase.expectedFiles : undefined;
+        const expectedSymbols = Array.isArray(benchmarkCase.expectedSymbols) ? benchmarkCase.expectedSymbols : undefined;
+        if (expectedFiles === undefined || expectedFiles.length === 0) {
+            errors.push(`${label}: expectedFiles must be a nonempty array.`);
+        }
+        if (expectedSymbols === undefined || expectedSymbols.length === 0) {
+            errors.push(`${label}: expectedSymbols must be a nonempty array.`);
+        }
+        if (benchmarkCase.answerKey === undefined) {
+            errors.push(`${label}: missing answerKey.`);
+        }
+        else {
+            errors.push(...validateAnswerKey(benchmarkCase.answerKey, label));
+            const answerKey = benchmarkCase.answerKey;
+            if (answerKey && typeof answerKey === "object") {
+                if (expectedFiles !== undefined && Array.isArray(answerKey.expectedFiles) && !sameOrderedValues(expectedFiles, answerKey.expectedFiles)) {
+                    errors.push(`${label}: expectedFiles must exactly match answerKey.expectedFiles (same values in the same order).`);
+                }
+                if (expectedSymbols !== undefined &&
+                    Array.isArray(answerKey.expectedSymbols) &&
+                    !sameOrderedValues(expectedSymbols, answerKey.expectedSymbols)) {
+                    errors.push(`${label}: expectedSymbols must exactly match answerKey.expectedSymbols (same values in the same order).`);
+                }
+            }
+        }
+        if (expectedFiles !== undefined && targetRoot !== undefined) {
+            for (const expectedFile of expectedFiles) {
+                if (typeof expectedFile !== "string" || expectedFile.length === 0 || path.isAbsolute(expectedFile) || expectedFile.includes("..")) {
+                    errors.push(`${label}: expected file must be a safe relative path: ${String(expectedFile)}.`);
+                }
+                else if (!existsSync(path.resolve(repoRoot, targetRoot, expectedFile))) {
+                    errors.push(`${label}: expected file does not exist in ${projectId ?? targetRoot}: ${expectedFile}.`);
+                }
+            }
+        }
+    });
+    const representedProjectIds = [
+        ...new Set(cases
+            .map((benchmarkCase) => benchmarkCase?.benchmarkProject)
+            .filter((projectId) => typeof projectId === "string" && projectId.length > 0))
+    ].sort();
+    for (const projectId of representedProjectIds) {
+        const profile = profilesById.get(projectId);
+        if (profile === undefined || !profile.complexityMetrics || typeof profile.complexityMetrics !== "object") {
+            continue;
+        }
+        const derived = deriveWarmIndexTaskStats(cases, projectId);
+        for (const field of ["taskCount", "expectedRelevantFilesAverage", "expectedRelevantSymbolsAverage"]) {
+            if (profile.complexityMetrics[field] !== derived[field]) {
+                errors.push(`profile ${projectId}: complexityMetrics.${field} ${profile.complexityMetrics[field]} does not match warm-index corpus value ${derived[field]}.`);
+            }
+        }
+    }
+    return errors;
+}
+export function validateWarmIndexBenchmarkSuiteCoverage(cases) {
+    const errors = [];
+    for (const projectId of REQUIRED_WARM_INDEX_BENCHMARK_PROJECT_IDS) {
+        const projectCases = cases.filter((benchmarkCase) => benchmarkCase?.benchmarkProject === projectId);
+        if (projectCases.length === 0) {
+            errors.push(`warm-index suite: missing required benchmark project ${projectId}.`);
+            continue;
+        }
+        if (projectCases.length < MIN_WARM_INDEX_TASKS_PER_PROJECT) {
+            errors.push(`warm-index suite: project ${projectId} has ${projectCases.length} cases; at least ${MIN_WARM_INDEX_TASKS_PER_PROJECT} are required.`);
+        }
+        for (const locality of TASK_LOCALITIES) {
+            if (!projectCases.some((benchmarkCase) => benchmarkCase.taskLocality === locality)) {
+                errors.push(`warm-index suite: project ${projectId} has no ${locality} case.`);
+            }
+        }
+    }
+    return errors;
+}
+function validateWarmIndexSourceRoots(sourceRoots, profile, label) {
+    const errors = [];
+    const allowedRoots = [...(profile.sourceRoots ?? []), ...(profile.testRoots ?? [])];
+    const seen = new Set();
+    for (const sourceRoot of sourceRoots) {
+        if (seen.has(sourceRoot)) {
+            errors.push(`${label}: duplicate source root ${JSON.stringify(sourceRoot)}.`);
+            continue;
+        }
+        seen.add(sourceRoot);
+        if (typeof sourceRoot !== "string" || !allowedRoots.includes(sourceRoot)) {
+            errors.push(`${label}: unknown source root ${JSON.stringify(sourceRoot)} for profile ${profile.projectId}; allowed roots: ${allowedRoots.join(", ")}.`);
+        }
+    }
+    for (const allowedRoot of allowedRoots) {
+        if (!seen.has(allowedRoot)) {
+            errors.push(`${label}: missing profile source/test root ${allowedRoot} for profile ${profile.projectId}.`);
+        }
+    }
+    return errors;
+}
+function normalizeRelativeRoot(value) {
+    return path.posix.normalize(value.replace(/\\/g, "/")).replace(/\/+$/, "");
+}
+function sameOrderedValues(left, right) {
+    return left.length === right.length && left.every((value, index) => value === right[index]);
+}
 export function validateBenchmarkProjectProfiles(profiles, repoRoot = process.cwd()) {
     const errors = [];
     const ids = new Set(profiles.map((profile) => profile.projectId));

package/dist/src/evaluation/projectComplexity.js

@@ -73,7 +73,7 @@ export function computeProjectComplexityMetrics(projectRoot, fileTree, taskStats
 function normalize(value, cap) {
     return Math.min(value / cap, 1);
 }
-function roundToTwo(value) {
+export function roundToTwo(value) {
     return Math.round(value * 100) / 100;
 }
 function sum(values) {

package/dist/src/evaluation/readEvaluationCases.js

@@ -1,7 +1,7 @@
 import { readFile } from "node:fs/promises";
 import path from "node:path";
 import { resolveWithinRoot } from "../core/pathSafety.js";
-import { validateAnswerKey } from "./benchmarkMetadata.js";
+import { validateAnswerKey, validateTaskLocality } from "./benchmarkMetadata.js";
 export async function readEvaluationCases(casesPath, repoRoot = process.cwd(), options = {}) {
     let parsed;
     try {
@@ -36,6 +36,12 @@ export async function readEvaluationCases(casesPath, repoRoot = process.cwd(), o
             throw new Error(`Duplicate evaluation case id: ${input.id}`);
         }
         ids.add(input.id);
+        if (candidate.taskLocality !== undefined) {
+            const localityErrors = validateTaskLocality(candidate.taskLocality, `evaluation case ${input.id}`);
+            if (localityErrors.length > 0) {
+                throw new Error(localityErrors.join("\n"));
+            }
+        }
         if (candidate.answerKey !== undefined) {
             const answerKeyErrors = validateAnswerKey(candidate.answerKey, `evaluation case ${input.id}`);
             if (answerKeyErrors.length > 0) {

package/dist/src/evaluation/types.js

@@ -1 +1 @@
-export {};
+export const TASK_LOCALITIES = ["localized", "cross-module", "broad-change"];

package/docs/METRICS.md

@@ -137,6 +137,15 @@ Each entry below uses the same fields: **Meaning**, **Appears in** (the artifact
   Interpretation: human-readable size category.
   Caveat: coarse label; use the score and metrics for detail.
 
+In the released v0.5.1 benchmark suite, the task statistics (`taskCount`, `expectedRelevantFilesAverage`, `expectedRelevantSymbolsAverage`) of the two warm-index benchmark projects are derived from the dedicated corpus `benchmarks/contracts/warm-index-benchmark-cases.json`: the averages are the mean `expectedFiles` and `expectedSymbols` counts of that project's cases, rounded to two decimals. `npm run verify:benchmarks` fails if a profile disagrees with the corpus. The `benchmark-project-complexity-v1` formula, its weights, and its caps are unchanged; the current values are:
+
+| Project | `taskCount` | `expectedRelevantFilesAverage` | `expectedRelevantSymbolsAverage` | `complexityScore` |
+|---|---|---|---|---|
+| `task-workflow-medium-ts` | 6 | 3.83 | 4.33 | 31 |
+| `task-analytics-large-mixed` | 6 | 4.5 | 4.67 | 47 |
+
+Task locality (`localized`, `cross-module`, `broad-change`) is benchmark metadata on each corpus case. It is not a metric, score, or ranking, and it does not enter the complexity formula.
+
 ## Prompt complexity metrics
 
 - `promptChars`
@@ -444,7 +453,9 @@ Implemented in `src/evaluation/stageContextMetrics` and composed once per run by
 
 ## Warm-index reuse metrics
 
-Implemented in the current release v0.5.0. The `warm-index-reuse` plugin calculates these metrics once, in `src/experiments/plugins/warmIndexReuse/metrics.ts`, from bounded execution summaries and bounded fake-agent evidence. The report section (`report.warmIndexReuse` in `report.json`, and the text and HTML "Warm Index Reuse Evidence" sections) and the four warm-index plots render these precomputed values; they never recalculate a formula.
+Introduced in v0.5.0. The `warm-index-reuse` plugin calculates these metrics once, in `src/experiments/plugins/warmIndexReuse/metrics.ts`, from bounded execution summaries and bounded fake-agent evidence. The report section (`report.warmIndexReuse` in `report.json`, and the text and HTML "Warm Index Reuse Evidence" sections) and the four warm-index plots render these precomputed values; they never recalculate a formula.
+
+The released v0.5.1 expanded benchmark suite adds no warm-index metric and changes no formula: over the dedicated 12-case corpus the same metrics run through task ordinal 6 in each project, and amortization and cumulative sums restart for each project. No token-savings percentage, break-even task, winner, speedup, ranking, or provider billing metric is calculated.
 
 **Availability contract.** Every warm-index metric carries `availability`, `value`, `unit`, `source`, `reason`, and `tokenCountMethod`:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dailephd/my-dev-kit-lab",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "type": "module",
   "description": "Evidence, benchmark, and evaluation companion for my-dev-kit.",
   "bin": {