@dailephd/my-dev-kit-lab 0.7.2 → 0.8.1

package/CHANGELOG.md

@@ -4,7 +4,39 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
-## [0.7.2] - 2026-10-03
+## [0.8.1] - 2026-10-05
+
+### Retrieval query strategy comparison
+
+- Added the experimental `retrieval-query-strategy-comparison` plugin with seven fixed variants: `keyword-search`, `symbol-lookup`, `graph-neighborhood`, `source-slice`, `data-model-graph`, `model-view-lineage`, and `combined-graph-guided`. It runs no coding agent.
+- Bundled mode reuses the frozen 12-case warm-index corpus; every selected case runs all seven strategies, and `--case` and `--benchmark-project` narrow cases or projects, never strategies.
+- Added strategy-neutral identity evidence and preserved the v0.8.0 retrieval-quality formulas through one generic identity-evidence metric owner.
+- Added balanced file and symbol F1: `2TP / (2TP + FP + FN)`.
+- Added matched-complete-case macro aggregation across the `overall`, `localized`, `cross-module`, and `broad-change` scopes.
+- Added four-objective Pareto comparison: maximize mean file F1, maximize mean symbol F1, maximize mean fact coverage, and minimize mean retrieved tokens.
+- A unique best strategy exists only for a single-member Pareto front; otherwise the report shows a tradeoff. There is no composite score and no total ranking.
+- Added `retrieval-query-strategy-comparison-execution.json` and `retrieval-query-strategy-comparison-analysis.json`, plus a typed `report.json`, `report.txt`, and `report.html` presentation of the precomputed analysis.
+- Added external-local `LocalRepositorySubject` execution: a private index per configured case, exact exclusions, isolated semantic copies, an eligible-universe check, target immutability, failure atomicity, science before redaction, and privacy-safe durable output.
+- Registered list, describe, and run support in the installed CLI with exact bundled and external-local argument boundaries.
+- Extended packed-package acceptance to cover discovery, bundled execution, reports, rejection and failure paths, privacy, immutability, and real `@dailephd/my-dev-kit@1.12.5` external-local compatibility.
+- Validated the legitimate 0.8.0 to 0.8.1 package metadata transition in a disposable candidate copy with no transition defects.
+- Not included: context packs, coding-agent evaluation, automatic strategy selection, scalar or composite scoring, ordinal ranking, and plots or screenshots.
+
+## [0.8.0] - 2026-10-04
+
+### Retrieval precision/recall
+
+- Added `retrieval-evidence-v1`, a deterministic observation of the existing my-dev-kit search, lookup, slice, and source lifecycle (retrieved files and symbols with `available`, `partial`, or `unavailable` status, safe repository-relative paths, fail-closed parsing, and no source-body scraping).
+- Added a deterministic retrieval-quality metric engine: file precision and recall, symbol precision and recall, fact coverage from explicit `factIds`-to-context mappings, irrelevant context ratio, retrieved token count, and the missed files, missed symbols, and uncovered fact IDs, with explicit `available`, `unavailable`, and `not-applicable` states. It produces no ranking or winner.
+- Added the `retrieval-precision-recall` experiment plugin with one `my-dev-kit-retrieval` variant and a bundled 12-case corpus (the warm-index benchmark cases, with explicit fact mappings) over two benchmark projects, one index per project and one retrieval per case. No agent is invoked. `experiment run --experiment retrieval-precision-recall` accepts `--out`, `--case`, `--benchmark-project`, and `--kit-command`.
+- Added the `retrieval-precision-recall-execution.json` artifact and specialized deterministic JSON, HTML, and text reports.
+- Added external-local support through the existing `LocalRepositorySubject` architecture: `--target <repository>` with `--local-subject-config <path>` runs one private index per configured case from its exact source roots, excludes Git-ignored and oversized files, enforces the eligible file universe, treats the repository as read-only, removes private scratch before persistence, and writes `local-repository-subject-manifest.json`. A failed external run writes no normal artifact, report, or manifest family.
+- Durable external-local output redacts the repository path, tool and output roots, file, symbol, and fact identities, case titles, and retrieval warning text, and preserves exact numeric evidence.
+- Extended the packed-package acceptance gate (`npm run verify:packed-package`) to cover installed plugin discovery, bundled and external-local execution, privacy, failure paths, and package and target immutability.
+- Corrected cross-platform portability of the retrieval-precision-recall verification: the packed-package gate now checks private-scratch containment against the physical output root (Windows short-name aliases), and the related unit and report tests use platform-native paths and separate runtime state from durable output.
+- Not included: ranked retrieval metrics, retrieval-strategy comparison, strategy ranking or winner selection, context packs, and real-agent evaluation.
+
+## [0.7.2] - 2026-10-04
 
 ### Local-repository experiments
 

package/README.md

@@ -6,7 +6,9 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.7.2 (local-repository experiments), and the published package and installed CLI are version 0.7.2. The previous release is v0.7.1 (synthetic large-repository generator). The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
+The latest release is v0.8.1 (retrieval query strategy comparison), and the published package and installed CLI are version 0.8.1. The previous release is v0.8.0 (retrieval precision/recall). The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
+
+v0.8.2 (context-pack generation experiments) is the next planned milestone.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
@@ -19,7 +21,9 @@ v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--vers
 - **Compare no-refresh, partial-refresh, and full-refresh retrieval after a controlled change (v0.6.3, released; retained in the current package):** the `incremental-change-staleness` experiment plugin runs six frozen controlled-change scenario families (unrelated-file, local-implementation, exported-symbol, public-API, import-graph, and test-only changes) against four disposable target copies — `stale-index` (no refresh), `changed-files-refresh` and `affected-neighborhood-refresh` (real partial refreshes performed by my-dev-kit 1.12.5 through `index --incremental --refresh-scope`, selected internally with no Lab-level flag), and `full-refresh` (a complete new post-change index, used as a comparison reference rather than a preferred treatment). Each partial treatment truthfully records whether upstream applied the requested partial scope or fell back to a full rebuild (`APPLIED_PARTIAL` or `FALLBACK_FULL`); a fallback is not counted as partial-refresh evidence. Three comparisons against `full-refresh` are persisted with retrieval, deterministic fake-agent correctness, and required-file evidence in `incremental-change-staleness-execution.json` (schema `my-dev-kit-lab-incremental-change-staleness-execution-v2`) and the existing `report.json`/`report.txt`/`report.html`; historical v0.6.2 (V1) artifacts stay readable. There is no winner, ranking, or safety claim, and the plugin is self-only (`--target` is not accepted). `--kit-command` also accepts `incremental-change-staleness` and now defaults to my-dev-kit 1.12.5. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md) and [docs/METRICS.md](docs/METRICS.md).
 - **Measure context-window scaling (v0.7.0, released):** the self-only `context-window-scaling` plugin compares `raw-full-file` and `my-dev-kit-guided` over 8k, 16k, 32k, 64k, or custom positive-integer estimated-token budgets. It records fit, utilization, deterministic fake-agent correctness/success, and omitted expected relevant files, with JSON/text/HTML reports and three SVG plots. `plots generate` reads `context-window-scaling-execution.json`. Token counts are a character-based estimate, not provider-tokenizer or billing data; fake-agent correctness is context-independent in the current harness. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md), [docs/METRICS.md](docs/METRICS.md), and [docs/COMMANDS.md](docs/COMMANDS.md).
 - **Run context-window scaling over synthetic repositories (v0.7.1, released):** `context-window-scaling` uses the bundled fixed corpus by default, or `--synthetic-config <path>` (a `SyntheticRepositoryConfigV1` JSON file) to generate disposable deterministic TypeScript/Python repositories beneath the run output. The option is a command input selector, not plugin scientific config; it is mutually exclusive with `--case`, and the scientific semantics, execution artifact, reports, and plots are unchanged. See [docs/COMMANDS.md](docs/COMMANDS.md) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md).
-- **Run context-window scaling over a local Git repository (v0.7.2, current release):** `context-window-scaling` can also run against an explicitly selected local Git repository with `--target <repository> --local-subject-config <path>` (a `LocalRepositorySubjectConfigV1` JSON file). The Lab treats the repository as read-only and compares its state before and after the run, builds contexts only from files Git does not ignore (ignored, oversized (over 1 MiB), symbolic-link, and other unsafe entries are excluded), requires the output directory to be outside the repository, and keeps the repository path, source text, and file names out of durable artifacts (file identities appear as numbered `<redacted file N>` placeholders). A `local-repository-subject-manifest.json` records the logical subject ID, the full Git commit, the branch, and aggregate size metadata. This measures operational context fit only, not retrieval precision, recall, or ranking quality. It is part of the current package. See [docs/COMMANDS.md](docs/COMMANDS.md) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md).
+- **Run context-window scaling over a local Git repository (v0.7.2, released):** `context-window-scaling` can also run against an explicitly selected local Git repository with `--target <repository> --local-subject-config <path>` (a `LocalRepositorySubjectConfigV1` JSON file). The Lab treats the repository as read-only and compares its state before and after the run, builds contexts only from files Git does not ignore (ignored, oversized (over 1 MiB), symbolic-link, and other unsafe entries are excluded), requires the output directory to be outside the repository, and keeps the repository path, source text, and file names out of durable artifacts (file identities appear as numbered `<redacted file N>` placeholders). A `local-repository-subject-manifest.json` records the logical subject ID, the full Git commit, the branch, and aggregate size metadata. This measures operational context fit only, not retrieval precision, recall, or ranking quality. It is part of the current package. See [docs/COMMANDS.md](docs/COMMANDS.md) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md).
+- **Measure retrieval precision and recall (v0.8.0, released):** the `retrieval-precision-recall` plugin runs the existing my-dev-kit search, lookup, slice, and source lifecycle for each case and compares the retrieved files and symbols with a deterministic answer key, without invoking any agent. It reports file precision and recall, symbol precision and recall, fact coverage (from explicit fact-to-context mappings), irrelevant context ratio, the estimated retrieved token count, and the missed and irrelevant identities, as `retrieval-precision-recall-execution.json` plus `report.json`, `report.html`, and `report.txt`. By default it runs a bundled 12-case corpus over two benchmark projects against the Lab itself. It can also run over an explicitly selected local Git repository with `--target <repository> --local-subject-config <path>`; in that mode the repository is treated as read-only and durable outputs redact the repository path and file, symbol, and fact identities while keeping the numeric evidence. These are set-based measures with no ranking, winner, or composite score. See [docs/COMMANDS.md](docs/COMMANDS.md), [docs/WORKFLOWS.md](docs/WORKFLOWS.md), and [docs/METRICS.md](docs/METRICS.md).
+- **Compare retrieval query strategies (v0.8.1, released):** the `retrieval-query-strategy-comparison` plugin compares seven fixed my-dev-kit retrieval strategies without running any coding agent: `keyword-search`, `symbol-lookup`, `graph-neighborhood`, `source-slice`, `data-model-graph`, `model-view-lineage`, and `combined-graph-guided`. In bundled mode it uses the same frozen 12-case warm-index benchmark corpus as `retrieval-precision-recall`, and every selected case receives all seven strategies (filters narrow cases or projects, never strategies). File and symbol quality keeps the inherited v0.8.0 set-based semantics and adds balanced file and symbol F1 (`2TP / (2TP + FP + FN)`); fact coverage and the estimated retrieved token count complete the four comparison objectives. Strategies are compared over matched complete cases only, as macro means for four scopes (`overall`, `localized`, `cross-module`, `broad-change`, reusing each case's existing task-locality label), using Pareto dominance on mean file F1, mean symbol F1, mean fact coverage (all maximized) and mean retrieved tokens (minimized). A best strategy is named only when the Pareto front has a single member; when several strategies remain nondominated the report states a tradeoff. There is no scalar or composite score and no ordinal ranking. The plugin can also run over an explicitly selected local Git repository through the existing `LocalRepositorySubject` boundary with `--target <repository> --local-subject-config <path>`; the repository stays read-only and durable output redacts private file, symbol, fact, semantic-node, warning, and case-title identities while preserving the numeric results. It writes `retrieval-query-strategy-comparison-execution.json`, `retrieval-query-strategy-comparison-analysis.json`, and `report.json`, `report.html`, and `report.txt`. No context-pack behavior is included. See [docs/COMMANDS.md](docs/COMMANDS.md), [docs/WORKFLOWS.md](docs/WORKFLOWS.md), and [docs/METRICS.md](docs/METRICS.md#retrieval-query-strategy-comparison-metrics-v081).
 - **Assess affected-neighborhood evidence (v0.6.1, released; retained in the current package):** using the baseline my-dev-kit graph of the same prepared warm index (loaded once per session), each task's confirmed modified or missing indexed files, and the baseline symbols those files contain, are mapped to graph nodes. The affected neighborhood is those seed nodes plus their direct one-hop neighbors (edges of any kind, either direction). The task's `expectedFiles` and file-scoped `expectedSymbols` are mapped to graph nodes and compared with that neighborhood, which yields six numeric warm-side metrics (`changedFileCount`, `changedSymbolCount`, `affectedNodeCount`, `affectedEdgeCount`, `taskOverlapCount`, `taskOverlapPercent`) plus categorical evidence: a `related`, `unrelated`, or `unknown` relationship and a `recommended`, `not-indicated`, or `unknown` `reindexRecommendation`. Incomplete evidence is reported as `unknown`, never as unrelated. The assessment is recorded additively in `warm-index-execution.json` and shown in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: retrieval, execution status, and provider status are unchanged, nothing is reindexed automatically, and no CLI flag, experiment plugin, plot, or gallery item was added. See [docs/METRICS.md](docs/METRICS.md#affected-neighborhood-metrics-v061) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
@@ -239,7 +243,9 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Claude and Codex token totals are used only when the CLI/run exposes them; missing provider token evidence stays explicitly unavailable (never zero, and never substituted with an estimated context-token count)
 - Codex and Claude campaign runs can produce timeouts, invalid-output, agent-unavailable, or agent-limit-reached outcomes; these are structured provider states, not warm-index infrastructure failures
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework has four registered plugins: `context-strategy-comparison`, `warm-index-reuse`, `incremental-change-staleness`, and `context-window-scaling`. The v0.7.0 context-window-scaling plugin is released; v0.7.1 lets it also run over caller-supplied synthetic repositories; v0.7.2 lets it run over an explicitly selected local Git repository; retrieval precision/recall and agent-success plugins remain planned.
+- The generic experiment-plugin framework has six registered plugins: `context-strategy-comparison`, `warm-index-reuse`, `incremental-change-staleness`, `context-window-scaling`, `retrieval-precision-recall`, and `retrieval-query-strategy-comparison`. The v0.7.0 context-window-scaling plugin is released; v0.7.1 lets it also run over caller-supplied synthetic repositories; v0.7.2 lets it run over an explicitly selected local Git repository. The fifth, `retrieval-precision-recall`, was added in v0.8.0. The sixth, `retrieval-query-strategy-comparison`, was added in v0.8.1; agent-success plugins remain planned.
+- `retrieval-precision-recall` (v0.8.0) measures set-based retrieval quality, not ranked retrieval: it has no ranking metrics, no retrieval-strategy comparison of its own (the separate `retrieval-query-strategy-comparison` plugin added in v0.8.1 compares strategies), no winner or composite score, and no real agents. Its lifecycle expands only the top search candidate. An external-local run fails as a whole if any case fails a safety or execution check, and then writes no normal artifact or report family. Durable external-local identities are redacted, although the console summary of a successful run can still show the physical output directory. Platform path-length limits can make a run with very long paths fail cleanly instead of executing.
+- `retrieval-query-strategy-comparison` (v0.8.1) runs no coding agent and does not evaluate answer quality. Its retrieved token count is a context-size estimate, not provider billing telemetry. Semantic strategies (`data-model-graph`, `model-view-lineage`) can truthfully be partial or unavailable for a case, and a case is then excluded from the matched cross-strategy comparison for every strategy while its per-strategy evidence is kept. The Pareto front is a set of nondominated strategies in canonical strategy order, not an ordinal ranking. The plugin reports no significance test or confidence interval, selects no strategy automatically, and generates no context pack. External-local mode rejects `--case` and `--benchmark-project`, builds one private base index per configured case, and fails as a whole, writing no normal artifact or report family, if any safety or privacy check fails.
 - `warm-index-reuse` without `--campaign` still uses the deterministic fake agent only. With `--campaign`, exactly one real provider (Codex or Claude) runs per campaign preset — there is no multi-agent matrix, retry, provider switching, or scheduler. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence when run without `--campaign`. No warm-index mode calculates a token-savings percentage, break-even task, winner, or ranking, and component-duration sums are not wall-clock latency
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Context-window scaling uses estimated tokens (`ceil(characters / 4)`) and experiment budgets, not actual provider tokenizer or model-window limits. Its deterministic fake-agent correctness check does not measure semantic sensitivity to the supplied context.

package/benchmarks/contracts/warm-index-benchmark-cases.json

@@ -41,6 +41,32 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "src/services/importTasks.ts",
+          "symbols": ["importTasks"],
+          "required": true,
+          "factIds": ["warm-medium-import-validates-title-and-project", "warm-medium-import-skips-duplicates-by-project-and-normalized-title", "warm-medium-import-keeps-deterministic-task-sequences"]
+        },
+        {
+          "file": "src/validation/taskValidation.ts",
+          "symbols": ["validateImportInput"],
+          "required": true,
+          "factIds": ["warm-medium-import-validates-title-and-project"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["findDuplicate"],
+          "required": true,
+          "factIds": ["warm-medium-import-skips-duplicates-by-project-and-normalized-title"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["createTask"],
+          "required": true,
+          "factIds": ["warm-medium-import-keeps-deterministic-task-sequences"]
+        }
+      ],
       "minimumCorrectFacts": 3,
       "notes": "Cross-module retrieval spanning the import service, validation, and store logic."
     },
@@ -117,6 +143,44 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "src/services/createTask.ts",
+          "symbols": ["createTask"],
+          "required": true,
+          "factIds": ["warm-medium-create-validates-title", "warm-medium-create-validates-project-before-storing", "warm-medium-create-normalizes-tags", "warm-medium-create-starts-incomplete"]
+        },
+        {
+          "file": "src/validation/taskValidation.ts",
+          "symbols": ["validateTaskTitle"],
+          "required": true,
+          "factIds": ["warm-medium-create-validates-title"]
+        },
+        {
+          "file": "src/validation/taskValidation.ts",
+          "symbols": ["validateProjectId"],
+          "required": true,
+          "factIds": ["warm-medium-create-validates-project-before-storing"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["TaskWorkflowStore"],
+          "required": true,
+          "factIds": ["warm-medium-create-validates-project-before-storing", "warm-medium-create-uses-deterministic-sequence"]
+        },
+        {
+          "file": "src/validation/taskValidation.ts",
+          "symbols": ["normalizeTags"],
+          "required": true,
+          "factIds": ["warm-medium-create-normalizes-tags"]
+        },
+        {
+          "file": "src/utils/deterministicId.ts",
+          "symbols": ["createDeterministicId"],
+          "required": true,
+          "factIds": ["warm-medium-create-uses-deterministic-sequence"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Cross-module: the create service composes validation, the store, and the deterministic id utility."
     },
@@ -163,6 +227,20 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "src/services/completeTask.ts",
+          "symbols": ["completeTask"],
+          "required": true,
+          "factIds": ["warm-medium-complete-targets-task-by-id", "warm-medium-complete-sets-completed-at", "warm-medium-complete-is-idempotent"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["TaskWorkflowStore"],
+          "required": true,
+          "factIds": ["warm-medium-complete-targets-task-by-id", "warm-medium-complete-unknown-task-throws"]
+        }
+      ],
       "minimumCorrectFacts": 3,
       "notes": "Localized: the behavior lives in one service plus the store update primitive."
     },
@@ -215,6 +293,26 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "src/services/filterTasks.ts",
+          "symbols": ["filterTasks"],
+          "required": true,
+          "factIds": ["warm-medium-filter-combines-active-predicates", "warm-medium-filter-project", "warm-medium-filter-completed-priority-tag", "warm-medium-filter-text-query", "warm-medium-filter-read-only-ordering"]
+        },
+        {
+          "file": "src/models/task.ts",
+          "symbols": ["TaskFilter"],
+          "required": true,
+          "factIds": ["warm-medium-filter-combines-active-predicates"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["TaskWorkflowStore"],
+          "required": true,
+          "factIds": ["warm-medium-filter-read-only-ordering"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Cross-module: filter semantics depend on the TaskFilter model and store listing order."
     },
@@ -271,6 +369,32 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "src/services/summarizeTasks.ts",
+          "symbols": ["summarizeTasks"],
+          "required": true,
+          "factIds": ["warm-medium-summary-total-open-completed", "warm-medium-summary-by-project", "warm-medium-summary-high-priority-open", "warm-medium-summary-read-only"]
+        },
+        {
+          "file": "src/models/task.ts",
+          "symbols": ["TaskSummary"],
+          "required": true,
+          "factIds": ["warm-medium-summary-total-open-completed"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["TaskWorkflowStore"],
+          "required": true,
+          "factIds": ["warm-medium-summary-by-project", "warm-medium-summary-read-only"]
+        },
+        {
+          "file": "src/models/project.ts",
+          "symbols": ["WorkflowProject"],
+          "required": true,
+          "factIds": ["warm-medium-summary-by-project"]
+        }
+      ],
       "minimumCorrectFacts": 3,
       "notes": "Cross-module: the summary combines the store with task and project models."
     },
@@ -356,6 +480,50 @@
         }
       ],
       "forbiddenWrongClaims": ["importTasks calls the createTask service to insert imported tasks."],
+      "expectedContextTargets": [
+        {
+          "file": "src/services/createTask.ts",
+          "symbols": ["createTask"],
+          "required": true,
+          "factIds": ["warm-medium-broad-shared-storage"]
+        },
+        {
+          "file": "src/services/importTasks.ts",
+          "symbols": ["importTasks"],
+          "required": true,
+          "factIds": ["warm-medium-broad-shared-storage", "warm-medium-broad-import-adds-dedupe"]
+        },
+        {
+          "file": "src/store/taskStore.ts",
+          "symbols": ["TaskWorkflowStore"],
+          "required": true,
+          "factIds": ["warm-medium-broad-shared-storage", "warm-medium-broad-import-adds-dedupe", "warm-medium-broad-filter-read-only", "warm-medium-broad-complete-targeted", "warm-medium-broad-summary-aggregates"]
+        },
+        {
+          "file": "src/validation/taskValidation.ts",
+          "symbols": ["validateImportInput"],
+          "required": true,
+          "factIds": ["warm-medium-broad-import-adds-dedupe"]
+        },
+        {
+          "file": "src/services/filterTasks.ts",
+          "symbols": ["filterTasks"],
+          "required": true,
+          "factIds": ["warm-medium-broad-filter-read-only"]
+        },
+        {
+          "file": "src/services/completeTask.ts",
+          "symbols": ["completeTask"],
+          "required": true,
+          "factIds": ["warm-medium-broad-complete-targeted"]
+        },
+        {
+          "file": "src/services/summarizeTasks.ts",
+          "symbols": ["summarizeTasks"],
+          "required": true,
+          "factIds": ["warm-medium-broad-summary-aggregates"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Deliberately wide-scope negative control spanning every workflow service; it is not expected to fit a small localized context."
     },
@@ -426,6 +594,26 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "ts/src/services/buildAnalyticsSnapshot.ts",
+          "symbols": ["buildAnalyticsSnapshot"],
+          "required": true,
+          "factIds": ["warm-large-snapshot-computes-completion-rate", "warm-large-snapshot-counts-stale-open-tasks"]
+        },
+        {
+          "file": "py/task_analytics/quality.py",
+          "symbols": ["determine_quality_label"],
+          "required": true,
+          "factIds": ["warm-large-python-quality-labels-healthy-watch-risk", "warm-large-python-reporting-renders-quality-lines"]
+        },
+        {
+          "file": "py/task_analytics/reporting.py",
+          "symbols": ["build_health_report"],
+          "required": true,
+          "factIds": ["warm-large-python-reporting-renders-quality-lines"]
+        }
+      ],
       "minimumCorrectFacts": 3,
       "notes": "Cross-module retrieval spanning TypeScript snapshot building and Python quality reporting."
     },
@@ -488,6 +676,32 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "ts/src/services/buildAnalyticsSnapshot.ts",
+          "symbols": ["buildAnalyticsSnapshot"],
+          "required": true,
+          "factIds": ["warm-large-snapshot-per-project-counts", "warm-large-snapshot-completion-rate-formula", "warm-large-snapshot-stale-condition", "warm-large-snapshot-empty-project", "warm-large-snapshot-totals-aggregate"]
+        },
+        {
+          "file": "ts/src/services/listTasksByProject.ts",
+          "symbols": ["listTasksByProject"],
+          "required": true,
+          "factIds": ["warm-large-snapshot-per-project-counts"]
+        },
+        {
+          "file": "ts/src/store/taskStore.ts",
+          "symbols": ["AnalyticsTaskStore"],
+          "required": true,
+          "factIds": ["warm-large-snapshot-per-project-counts"]
+        },
+        {
+          "file": "ts/src/models/analyticsSnapshot.ts",
+          "symbols": ["ProjectSnapshot"],
+          "required": true,
+          "factIds": ["warm-large-snapshot-per-project-counts", "warm-large-snapshot-empty-project", "warm-large-snapshot-totals-aggregate"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Cross-module: the snapshot service composes the task-listing service, the task store, and the snapshot model."
     },
@@ -534,6 +748,14 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "ts/src/reporting/buildProjectLeaderboard.ts",
+          "symbols": ["buildProjectLeaderboard"],
+          "required": true,
+          "factIds": ["warm-large-leaderboard-completion-rate-descending", "warm-large-leaderboard-stale-ascending", "warm-large-leaderboard-project-id-final", "warm-large-leaderboard-one-based-ranks"]
+        }
+      ],
       "minimumCorrectFacts": 3,
       "notes": "Localized: the ranking rule lives in one reporting function over the ProjectSnapshot model."
     },
@@ -586,6 +808,32 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "py/task_analytics/parser.py",
+          "symbols": ["parse_task_rows"],
+          "required": true,
+          "factIds": ["warm-large-python-parser-task-record"]
+        },
+        {
+          "file": "py/task_analytics/models.py",
+          "symbols": ["TaskRecord"],
+          "required": true,
+          "factIds": ["warm-large-python-parser-task-record"]
+        },
+        {
+          "file": "py/task_analytics/metrics.py",
+          "symbols": ["calculate_project_metrics"],
+          "required": true,
+          "factIds": ["warm-large-python-metrics-grouped-by-project", "warm-large-python-metrics-counts", "warm-large-python-metrics-completion-rate", "warm-large-python-metrics-average-story-points"]
+        },
+        {
+          "file": "py/task_analytics/models.py",
+          "symbols": ["ProjectMetrics"],
+          "required": true,
+          "factIds": ["warm-large-python-metrics-grouped-by-project"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Cross-module: parsing, the record models, and metric calculation live in separate Python modules."
     },
@@ -662,6 +910,38 @@
           "required": true
         }
       ],
+      "expectedContextTargets": [
+        {
+          "file": "py/task_analytics/pipeline.py",
+          "symbols": ["build_report_from_rows"],
+          "required": true,
+          "factIds": ["warm-large-pipeline-parses-first", "warm-large-pipeline-metrics-before-report", "warm-large-pipeline-composes-owners"]
+        },
+        {
+          "file": "py/task_analytics/parser.py",
+          "symbols": ["parse_task_rows"],
+          "required": true,
+          "factIds": ["warm-large-pipeline-parses-first", "warm-large-pipeline-composes-owners"]
+        },
+        {
+          "file": "py/task_analytics/metrics.py",
+          "symbols": ["calculate_project_metrics"],
+          "required": true,
+          "factIds": ["warm-large-pipeline-metrics-before-report", "warm-large-pipeline-composes-owners"]
+        },
+        {
+          "file": "py/task_analytics/reporting.py",
+          "symbols": ["build_health_report"],
+          "required": true,
+          "factIds": ["warm-large-pipeline-metrics-before-report", "warm-large-pipeline-quality-label-in-report", "warm-large-pipeline-deterministic-output", "warm-large-pipeline-composes-owners"]
+        },
+        {
+          "file": "py/task_analytics/quality.py",
+          "symbols": ["determine_quality_label"],
+          "required": true,
+          "factIds": ["warm-large-pipeline-quality-label-in-report"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Cross-module: the pipeline composes parser, metrics, reporting, and (through reporting) quality modules."
     },
@@ -755,6 +1035,56 @@
         "Python is a runtime backend for the TypeScript analytics implementation.",
         "The TypeScript or Python implementation directly imports or calls the other."
       ],
+      "expectedContextTargets": [
+        {
+          "file": "ts/src/services/buildAnalyticsSnapshot.ts",
+          "symbols": ["buildAnalyticsSnapshot"],
+          "required": true,
+          "factIds": ["warm-large-broad-typescript-side", "warm-large-broad-comparable-evidence", "warm-large-broad-independent-implementations", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "ts/src/reporting/formatTaskHealthReport.ts",
+          "symbols": ["formatTaskHealthReport"],
+          "required": true,
+          "factIds": ["warm-large-broad-typescript-side", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "ts/src/reporting/buildProjectLeaderboard.ts",
+          "symbols": ["buildProjectLeaderboard"],
+          "required": true,
+          "factIds": ["warm-large-broad-typescript-side", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "py/task_analytics/parser.py",
+          "symbols": ["parse_task_rows"],
+          "required": true,
+          "factIds": ["warm-large-broad-python-side", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "py/task_analytics/metrics.py",
+          "symbols": ["calculate_project_metrics"],
+          "required": true,
+          "factIds": ["warm-large-broad-python-side", "warm-large-broad-comparable-evidence", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "py/task_analytics/quality.py",
+          "symbols": ["determine_quality_label"],
+          "required": true,
+          "factIds": ["warm-large-broad-python-side", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "py/task_analytics/reporting.py",
+          "symbols": ["build_health_report"],
+          "required": true,
+          "factIds": ["warm-large-broad-python-side", "warm-large-broad-no-cross-invocation"]
+        },
+        {
+          "file": "py/task_analytics/pipeline.py",
+          "symbols": ["build_report_from_rows"],
+          "required": true,
+          "factIds": ["warm-large-broad-python-side", "warm-large-broad-independent-implementations", "warm-large-broad-no-cross-invocation"]
+        }
+      ],
       "minimumCorrectFacts": 4,
       "notes": "Deliberately wide-scope negative control spanning both language implementations; it is not expected to fit a small localized context."
     },

package/dist/src/cli/help.js

@@ -156,7 +156,7 @@ export function renderExperimentRunHelp() {
         "  --case <ids>                                       Comma-separated case ids to run",
         "  --benchmark-project <ids>                          Comma-separated benchmark project ids to run",
         "",
-        "my-dev-kit command override (warm-index-reuse, incremental-change-staleness, and context-window-scaling):",
+        "my-dev-kit command override (warm-index-reuse, incremental-change-staleness, context-window-scaling, retrieval-precision-recall, and retrieval-query-strategy-comparison):",
         "  --kit-command <command>                            my-dev-kit command used by the selected experiment plugin",
         "                                                      (defaults: warm-index-reuse and context-window-scaling use",
         "                                                      npx @dailephd/my-dev-kit@latest;",
@@ -188,6 +188,37 @@ export function renderExperimentRunHelp() {
         "                                                      only, not retrieval precision, recall, or ranking quality.",
         "                                                      Generate plots with \"plots generate --experiment <run dir>\".",
         "",
+        "retrieval-precision-recall only:",
+        "                                                      Bundled mode (default) runs the frozen 12-case corpus and accepts",
+        "                                                      only --out, --case, --benchmark-project and --kit-command.",
+        "                                                      External-local mode requires an explicit --target together with",
+        "                                                      --local-subject-config and accepts only --out, --target,",
+        "                                                      --local-subject-config and --kit-command; --case and",
+        "                                                      --benchmark-project are rejected because the local-subject config",
+        "                                                      owns the case set. Each configured case needs a complete answer",
+        "                                                      key with explicit factIds mappings. One private index is built per",
+        "                                                      case; the repository is never modified and durable output omits its",
+        "                                                      path, source text, and file, symbol and fact identities.",
+        "",
+        "retrieval-query-strategy-comparison only:",
+        "                                                      Bundled mode (default) runs the frozen 12-case corpus and accepts",
+        "                                                      only --out, --case, --benchmark-project and --kit-command; --case",
+        "                                                      and --benchmark-project narrow the case set. All seven retrieval",
+        "                                                      query strategies always run for every case; there is no strategy",
+        "                                                      option. External-local mode requires an explicit --target together",
+        "                                                      with --local-subject-config and accepts only --out, --target,",
+        "                                                      --local-subject-config and --kit-command; --case and",
+        "                                                      --benchmark-project are rejected because the local-subject config",
+        "                                                      owns the case set. One private base index is built per configured",
+        "                                                      case; the semantic strategies use isolated copies of it; the",
+        "                                                      repository is never modified and durable output withholds private",
+        "                                                      file, symbol, fact, semantic-node, warning and case-title",
+        "                                                      identities. Reports compare the strategies for overall, localized,",
+        "                                                      cross-module and broad-change cases. A best strategy is named only",
+        "                                                      when the Pareto front has a single member; multiple nondominated",
+        "                                                      strategies are reported as a tradeoff. No scalar score or total",
+        "                                                      ranking exists.",
+        "",
         "warm-index-reuse only:",
         "                                                      Legacy/default mode uses the deterministic fake agent only;",
         "                                                      agent options below are not accepted by this plugin.",

package/dist/src/commands/runExperimentDescribeCommand.js

@@ -133,6 +133,12 @@ function describeTargetBehavior(plugin) {
     if (plugin.metadata.id === "context-window-scaling") {
         return "Three subject modes. Bundled (default): the bundled fixed scaling corpus against the Lab itself. Synthetic: caller-supplied deterministic synthetic repositories via --synthetic-config <path> (a SyntheticRepositoryConfigV1 JSON file, mutually exclusive with --case) written beneath the experiment output directory. External local repository: an explicitly selected local Git worktree via --target <path> together with --local-subject-config <path> (a LocalRepositorySubjectConfigV1 JSON file); the repository is never modified, the output directory must be outside it, and durable output omits the repository path, source text, and file names. Operational context-fit measurement only; no retrieval precision, recall, or ranking quality is measured.";
     }
+    if (plugin.metadata.id === "retrieval-precision-recall") {
+        return "Two subject modes. Bundled (default): the frozen bundled 12-case warm-index corpus against the Lab itself, filterable with --case and --benchmark-project. External local repository: an explicitly selected local Git worktree via --target <path> together with --local-subject-config <path> (a LocalRepositorySubjectConfigV1 JSON file whose cases carry a complete retrieval answer key with explicit fact-to-context mappings); the repository is never modified, the output directory must be outside it, one private index is built per configured case, and durable output withholds the repository path, source text, and file, symbol and fact identities. Bundled filters are not accepted in external mode.";
+    }
+    if (plugin.metadata.id === "retrieval-query-strategy-comparison") {
+        return "Two subject modes. Bundled (default): the frozen 12-case warm-index corpus compares all seven retrieval query strategies and may be narrowed with --case and --benchmark-project. External local repository: an explicitly selected local Git worktree via --target <path> together with --local-subject-config <path>; the local-subject config owns the case set, one private base index is built per configured case, core strategies share that base, semantic strategies use isolated copies, the repository is never modified, and durable output withholds private file, symbol, fact, semantic-node, warning and case-title identities while preserving numeric scientific results. Bundled filters are not accepted in external-local mode.";
+    }
     if (supportsExternal) {
         return "Runs against the current lab repository when --target is omitted, or against an explicit local target project when --target <path> is provided. Outputs stay under lab-controlled output directories by default.";
     }
@@ -156,13 +162,22 @@ function describeExamples(plugin) {
         `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id}${agentOptions}`
     ];
     // Only advertise an explicit --target example for plugins that support external targets.
-    if (plugin.metadata.supportedTargets.includes("external-local") && plugin.metadata.id !== "context-window-scaling") {
+    if (plugin.metadata.supportedTargets.includes("external-local") &&
+        plugin.metadata.id !== "context-window-scaling" &&
+        plugin.metadata.id !== "retrieval-precision-recall" &&
+        plugin.metadata.id !== "retrieval-query-strategy-comparison") {
         examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target "Z:\\Users\\newuser\\Projects\\my-dev-kit-v1"${agentOptions}${screenshotOption}`);
     }
     // Only advertise campaign examples for plugins that declare the campaignPreset config field.
     if (hasConfigField(plugin, "campaignPreset")) {
         examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign codex-full --include-real-agents --out <dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign claude-full --include-real-agents --case warm-medium-complete-idempotent --out <dir>`);
     }
+    if (plugin.metadata.id === "retrieval-precision-recall") {
+        examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --case <case-id> --out <run-dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target <local-git-repository> --local-subject-config <path-to-local-subject-config.json> --out <run-dir-outside-the-repository>`);
+    }
+    if (plugin.metadata.id === "retrieval-query-strategy-comparison") {
+        examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --case <case-id> --out <run-dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target <local-git-repository> --local-subject-config <path-to-local-subject-config.json> --out <run-dir-outside-the-repository>`);
+    }
     if (plugin.metadata.id === "context-window-scaling") {
         examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --synthetic-config <path-to-config.json> --out <run-dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target <local-git-repository> --local-subject-config <path-to-local-subject-config.json> --out <run-dir-outside-the-repository>`);
     }