@dailephd/my-dev-kit-lab 0.6.2 → 0.6.3

package/CHANGELOG.md

@@ -4,6 +4,24 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
+## [0.6.3] - 2026-09-30
+
+### Four-treatment partial-refresh experiments
+
+- Replaced CodeQL-flagged polynomial regular-expression parsing paths with linear scanning and strengthened credential evidence fingerprints with context-salted scrypt derivation.
+
+- Extended the `incremental-change-staleness` plugin from two treatments to four, in fixed order: `stale-index` (`my-dev-kit-no-refresh`), `changed-files-refresh` (`my-dev-kit-changed-files-refresh`), `affected-neighborhood-refresh` (`my-dev-kit-affected-neighborhood-refresh`), and `full-refresh` (`my-dev-kit-full-refresh`). No new experiment plugin was added.
+- Changed the default kit command of `incremental-change-staleness` from `npx @dailephd/my-dev-kit@1.12.4` to `npx @dailephd/my-dev-kit@1.12.5`, the first published version with `index --incremental --refresh-scope changed-files|affected-neighborhood`. Lab requests the scope internally; no Lab-level `--refresh-scope` flag exists.
+- Added a four-treatment lifecycle: four isolated disposable targets, equivalent trusted baselines, containment-checked whole-directory index clones for the two partial treatments, a mutation barrier, and per-treatment freshness validation. `stale-index` performs no post-mutation index invocation.
+- Added truthful refresh evidence: upstream `requestedScope`, `appliedScope`, `selectionStatus`, `fallbackReason`, fresh-extraction, reused-file, forced-neighbor, and seed/affected counts, and a Lab `refreshRealization` of `NO_REFRESH`, `APPLIED_PARTIAL`, `FALLBACK_FULL`, or `FULL_REFRESH`. A partial treatment that falls back is `not-comparable-as-partial-refresh`.
+- Added exactly three reference comparisons against `full-refresh`: `stale-index` (unchanged v0.6.2 stale-risk vocabulary) and the two partial treatments (`observed-regression-relative-to-full`, `no-observed-regression-relative-to-full`, `inconclusive`, `not-comparable-as-partial-refresh`). There is no winner, ranking, safety score, or automatic refresh policy.
+- New runs write execution schema `my-dev-kit-lab-incremental-change-staleness-execution-v2` and report schema `my-dev-kit-lab-incremental-change-staleness-report-v2` (JSON, text, HTML); historical V1 artifacts and reports remain readable. Added additive numeric `upstream-refresh-*` metrics for the partial treatments, kept separate from the Lab affected-neighborhood metrics.
+- `experiment run --help` now states that `--target` applies only to plugins that support external targets and that self-only plugins reject an explicit external `--target`.
+- Automatic warm-index campaign report screenshots are best-effort presentation evidence: failed capture stays explicitly failed with its warning and no fabricated PNG, but does not fail an otherwise successful campaign. Required campaign, report, plot, and gallery failures remain fatal.
+- Updated the packed-package acceptance gate (`npm run verify:packed-package`) to pin the real registry package `@dailephd/my-dev-kit@1.12.5` and to verify all six scenarios with four treatments, applied partial refresh, operational discrimination, and installed-package, upstream-package, and canonical-benchmark immutability.
+- Added repository-controlled GitHub CodeQL advanced-setup analysis for JavaScript/TypeScript and Actions; the release candidate passed exact-SHA analysis with no open applicable alerts.
+- No new plugin, `graph-diff` dependency, true-symbol-diff metric, multi-hop traversal, automatic refresh policy, external-target support, or new plot, screenshot, or gallery item was added.
+
 ## [0.6.2] - 2026-09-27
 
 ### Incremental-change and staleness plugin

package/README.md

@@ -6,7 +6,7 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.6.2 (incremental-change and staleness experiments), and the published package and installed CLI are version 0.6.2. The previous release is v0.6.1 (affected-neighborhood experiments). Partial-refresh work is planned for v0.6.3. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
+The latest release is v0.6.3 (four-treatment partial-refresh experiments), and the published package and installed CLI are version 0.6.3. The previous release is v0.6.2 (incremental-change and staleness experiments). The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
@@ -14,10 +14,10 @@ v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--vers
 
 - **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
 - **Measure warm-index reuse (v0.5.0–v0.5.1, released):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. The v0.5.1 release adds a dedicated 12-task benchmark corpus (`benchmarks/contracts/warm-index-benchmark-cases.json`: six medium and six large/mixed tasks tagged `localized`, `cross-module`, or `broad-change`) for comparing warm-index behavior as task count grows. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry.
-- **Run selectable real-agent warm-index campaigns (v0.5.2, released):** `--campaign codex-full`, `claude-full`, or `codex-timeout-isolation` (with `--include-real-agents`) runs exactly one provider — Codex or Claude — against the exact raw and warm contexts the experiment already measured, over the frozen production corpus each preset owns. Provider outcomes (`completed`, `failed`, `timeout`, `invalid-output`, `agent-unavailable`, `agent-limit-reached`, `skipped`) are classified and reported separately from warm-index infrastructure status, so a campaign can complete its indexing/retrieval work even when provider evidence is partial or unavailable. A campaign run automatically produces the report, the same four warm-index SVG charts, a best-effort report screenshot (skipped without a browser runtime, never fatal), and a three-item gallery (report, plots, bounded execution evidence). Legacy runs without `--campaign` keep the existing deterministic fake-agent, report-only behavior unchanged. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
+- **Run selectable real-agent warm-index campaigns (v0.5.2, released):** `--campaign codex-full`, `claude-full`, or `codex-timeout-isolation` (with `--include-real-agents`) runs exactly one provider — Codex or Claude — against the exact raw and warm contexts the experiment already measured, over the frozen production corpus each preset owns. Provider outcomes (`completed`, `failed`, `timeout`, `invalid-output`, `agent-unavailable`, `agent-limit-reached`, `skipped`) are classified and reported separately from warm-index infrastructure status, so a campaign can complete its indexing/retrieval work even when provider evidence is partial or unavailable. A campaign run automatically produces the report, the same four warm-index SVG charts, and a three-item gallery (report, plots, bounded execution evidence). Its report screenshot is best-effort presentation evidence: unavailable browser runtime or failed capture is reported explicitly, no PNG is fabricated, and neither outcome fails an otherwise successful campaign. Legacy runs without `--campaign` keep the existing deterministic fake-agent, report-only behavior unchanged. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
 - **Record index freshness evidence (v0.6.0):** a warm-index run captures an index snapshot after its one index build — the exact indexed file set from the my-dev-kit index contract, each with SHA-256 content identity, size, and modified-time metadata, plus the my-dev-kit version and index-command evidence — and assesses freshness immediately before each task's warm retrieval as `fresh`, `stale`, `partially-stale`, or `unknown`. The evidence appears in `warm-index-execution.json` and in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: it covers only files the snapshot lists (not the whole repository), never changes retrieval, execution or provider status, and adds no CLI flag, metric, plot, gallery item, or reindex recommendation in v0.6.0. See [docs/METRICS.md](docs/METRICS.md#index-freshness-evidence-v060) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
-- **Compare stale-index vs full-refresh retrieval after a controlled change (`v0.6.2`):** the `incremental-change-staleness` experiment plugin runs six frozen controlled-change scenario families (unrelated-file, local-implementation, exported-symbol, public-API, import-graph, and test-only changes) against two disposable target copies — one kept on its pre-change baseline index (`stale-index`) and one fully reindexed after the same deterministic change (`full-refresh`) — and persists retrieval, deterministic fake-agent correctness, required-file, and a conservative stale-risk classification (`observed-stale-regression`, `no-observed-stale-regression`, or `inconclusive`) in `incremental-change-staleness-execution.json` and the existing `report.json`/`report.txt`/`report.html`. `--kit-command` also accepts `incremental-change-staleness`. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md) and [docs/METRICS.md](docs/METRICS.md).
-- **Assess affected-neighborhood evidence (v0.6.1, current):** using the baseline my-dev-kit graph of the same prepared warm index (loaded once per session), each task's confirmed modified or missing indexed files, and the baseline symbols those files contain, are mapped to graph nodes. The affected neighborhood is those seed nodes plus their direct one-hop neighbors (edges of any kind, either direction). The task's `expectedFiles` and file-scoped `expectedSymbols` are mapped to graph nodes and compared with that neighborhood, which yields six numeric warm-side metrics (`changedFileCount`, `changedSymbolCount`, `affectedNodeCount`, `affectedEdgeCount`, `taskOverlapCount`, `taskOverlapPercent`) plus categorical evidence: a `related`, `unrelated`, or `unknown` relationship and a `recommended`, `not-indicated`, or `unknown` `reindexRecommendation`. Incomplete evidence is reported as `unknown`, never as unrelated. The assessment is recorded additively in `warm-index-execution.json` and shown in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: retrieval, execution status, and provider status are unchanged, nothing is reindexed automatically, and no CLI flag, experiment plugin, plot, or gallery item was added. See [docs/METRICS.md](docs/METRICS.md#affected-neighborhood-metrics-v061) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
+- **Compare no-refresh, partial-refresh, and full-refresh retrieval after a controlled change (v0.6.3, current release):** the `incremental-change-staleness` experiment plugin runs six frozen controlled-change scenario families (unrelated-file, local-implementation, exported-symbol, public-API, import-graph, and test-only changes) against four disposable target copies — `stale-index` (no refresh), `changed-files-refresh` and `affected-neighborhood-refresh` (real partial refreshes performed by my-dev-kit 1.12.5 through `index --incremental --refresh-scope`, selected internally with no Lab-level flag), and `full-refresh` (a complete new post-change index, used as a comparison reference rather than a preferred treatment). Each partial treatment truthfully records whether upstream applied the requested partial scope or fell back to a full rebuild (`APPLIED_PARTIAL` or `FALLBACK_FULL`); a fallback is not counted as partial-refresh evidence. Three comparisons against `full-refresh` are persisted with retrieval, deterministic fake-agent correctness, and required-file evidence in `incremental-change-staleness-execution.json` (schema `my-dev-kit-lab-incremental-change-staleness-execution-v2`) and the existing `report.json`/`report.txt`/`report.html`; historical v0.6.2 (V1) artifacts stay readable. There is no winner, ranking, or safety claim, and the plugin is self-only (`--target` is not accepted). `--kit-command` also accepts `incremental-change-staleness` and now defaults to my-dev-kit 1.12.5. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md) and [docs/METRICS.md](docs/METRICS.md).
+- **Assess affected-neighborhood evidence (v0.6.1, released; retained in the current package):** using the baseline my-dev-kit graph of the same prepared warm index (loaded once per session), each task's confirmed modified or missing indexed files, and the baseline symbols those files contain, are mapped to graph nodes. The affected neighborhood is those seed nodes plus their direct one-hop neighbors (edges of any kind, either direction). The task's `expectedFiles` and file-scoped `expectedSymbols` are mapped to graph nodes and compared with that neighborhood, which yields six numeric warm-side metrics (`changedFileCount`, `changedSymbolCount`, `affectedNodeCount`, `affectedEdgeCount`, `taskOverlapCount`, `taskOverlapPercent`) plus categorical evidence: a `related`, `unrelated`, or `unknown` relationship and a `recommended`, `not-indicated`, or `unknown` `reindexRecommendation`. Incomplete evidence is reported as `unknown`, never as unrelated. The assessment is recorded additively in `warm-index-execution.json` and shown in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: retrieval, execution status, and provider status are unchanged, nothing is reindexed automatically, and no CLI flag, experiment plugin, plot, or gallery item was added. See [docs/METRICS.md](docs/METRICS.md#affected-neighborhood-metrics-v061) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
 - **Validate Android projects:** run nineteen static checks by default, with Gradle operations, external tools, and network access available only through explicit opt-in flags.
@@ -182,9 +182,9 @@ npm run experiment:run -- \
   --out lab-output/warm-index-campaign
 ```
 
-This automatically writes `report.json`/`.txt`/`.html`, the four warm-index charts under `plots/`, a best-effort `report.png`, and a three-item `gallery/` (report, plots, bounded execution evidence) beneath `--out`. `--timeout-ms` overrides the preset's default per-agent timeout (240000 ms). `codex-timeout-isolation` narrows the corpus to three larger cases for exercising provider timeout/partial-outcome handling without running the full 12-case corpus.
+This automatically writes `report.json`/`.txt`/`.html`, the four warm-index charts under `plots/`, an optional best-effort `report.png`, and a three-item `gallery/` (report, plots, bounded execution evidence) beneath `--out`. Screenshot capture failure remains `failed`, is reported in the command output and gallery warning, and does not fail an otherwise successful campaign or fabricate a PNG. Required experiment, report, plot, and gallery failures remain fatal. `--timeout-ms` overrides the preset's default per-agent timeout (240000 ms). `codex-timeout-isolation` narrows the corpus to three larger cases for exercising provider timeout/partial-outcome handling without running the full 12-case corpus.
 
-When `--target` is omitted, the experiment runs in self mode against my-dev-kit-lab. When `--target <path>` is provided, the lab remains the tool root and the target project is inspected separately. Generated experiment outputs stay under lab-controlled output directories by default, not inside the target project.
+For experiment plugins that support explicit targets — `context-strategy-comparison` and ordinary (non-campaign) `warm-index-reuse` runs — omitting `--target` uses self mode against my-dev-kit-lab. When those plugins receive `--target <path>`, the lab remains the tool root and the target project is inspected separately. `incremental-change-staleness` does not accept `--target`; it always runs against the lab's bundled benchmark projects. Generated experiment outputs stay under lab-controlled output directories by default, not inside an inspected target project.
 
 ---
 
@@ -236,10 +236,11 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Claude and Codex token totals are used only when the CLI/run exposes them; missing provider token evidence stays explicitly unavailable (never zero, and never substituted with an estimated context-token count)
 - Codex and Claude campaign runs can produce timeouts, invalid-output, agent-unavailable, or agent-limit-reached outcomes; these are structured provider states, not warm-index infrastructure failures
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework has three registered plugins: `context-strategy-comparison`, the v0.5.0-introduced `warm-index-reuse`, and the `v0.6.2` `incremental-change-staleness` plugin (current). Later plugins such as context-window scaling remain future roadmap work
+- The generic experiment-plugin framework has three registered plugins: `context-strategy-comparison`, the v0.5.0-introduced `warm-index-reuse`, and the `incremental-change-staleness` plugin (introduced in v0.6.2; extended to four treatments in current release v0.6.3). Later plugins such as context-window scaling remain future roadmap work
 - `warm-index-reuse` without `--campaign` still uses the deterministic fake agent only. With `--campaign`, exactly one real provider (Codex or Claude) runs per campaign preset — there is no multi-agent matrix, retry, provider switching, or scheduler. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence when run without `--campaign`. No warm-index mode calculates a token-savings percentage, break-even task, winner, or ranking, and component-duration sums are not wall-clock latency
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Released v0.6.1 affected-neighborhood evidence is bounded to the baseline graph of the prepared index and to the snapshot-represented files: a changed symbol means the symbol was present in a confirmed changed indexed file, not that its own text was proven to change; `not-indicated` means only that this bounded analysis found no supporting evidence for reindexing, not that skipping a reindex is safe; unresolved or ambiguous expected symbols keep a task at `unknown` rather than unrelated
+- v0.6.3 (current release) partial-refresh evidence is scoped to the executed deterministic scenarios and the configured my-dev-kit 1.12.5-or-later tool. A partial treatment can fall back to a full rebuild and is then reported as not comparable as partial refresh, so an applied partial refresh in one run does not guarantee the next; `no-observed-regression-relative-to-full` is not a safety claim, and full refresh is a reference rather than a winner. Correctness remains deterministic fake-agent evidence, and `incremental-change-staleness` accepts no external `--target`.
 - Provider telemetry dashboards, semantic LLM judging, and cloud API billing integration are not yet implemented
 - The six new stage-context strategies have no CLI flags yet, are configured programmatically, and do not yet include plots, screenshots, or gallery integration
 - The published upstream artifacts the stage-context strategies read do not expose considered-but-unselected reads or unnecessary-read evidence; those metrics report `unavailable` rather than zero
@@ -258,7 +259,7 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 
 Android validation uses `--profile android`. Its default path is static and non-destructive: it starts zero Gradle processes, zero external tools, and zero network operations. Only confirmed `SecurityFinding` records can become audit issues; Android `CandidateEvidence` remains review-only evidence.
 
-Optional scanners are reported as `skipped` when unavailable, never as passed. The framework does not provide runtime isolation proof, device or APK/AAB analysis, signing verification, Play Console validation, automatic fixes, or manual pentesting. Manual pentest remains deferred until post-v1/version TBD.
+Optional local scanners are reported as `skipped` when unavailable, never as passed. This local availability rule does not make CodeQL coverage optional for release readiness: the exact candidate must pass the repository-controlled GitHub CodeQL workflow and have its Code Scanning alerts reviewed. The framework does not provide runtime isolation proof, device or APK/AAB analysis, signing verification, Play Console validation, automatic fixes, or manual pentesting. Manual pentest remains deferred until post-v1/version TBD.
 
 See [COMMANDS.md](docs/COMMANDS.md) for exact syntax and [Security Validation Framework](docs/security-validation-framework.md) for checks, evidence semantics, verdicts, and limitations.
 

package/dist/src/cli/help.js

@@ -143,7 +143,10 @@ export function renderExperimentRunHelp() {
         "  --experiment <id>   Registered experiment plugin id (see \"my-dev-kit-lab experiment list\")",
         "",
         "Common options (all plugins):",
-        "  --target <path>                                    Local target project path (default: self)",
+        "  --target <path>                                    Local target project path (default: self); only for",
+        "                                                      plugins that support external targets. Self-only",
+        "                                                      plugins (for example incremental-change-staleness)",
+        "                                                      reject an explicit external --target.",
         "  --out <dir>                                         Output directory for experiment artifacts",
         "                                                      (default: beneath the workspace when installed)",
         "  --cases <path>                                      Path to the evaluation cases file",
@@ -153,10 +156,12 @@ export function renderExperimentRunHelp() {
         "  --case <ids>                                       Comma-separated case ids to run",
         "  --benchmark-project <ids>                          Comma-separated benchmark project ids to run",
         "",
+        "my-dev-kit command override (warm-index-reuse and incremental-change-staleness):",
+        "  --kit-command <command>                            my-dev-kit command used by the selected experiment plugin",
+        "                                                      (defaults: warm-index-reuse uses npx @dailephd/my-dev-kit@latest;",
+        "                                                      incremental-change-staleness uses npx @dailephd/my-dev-kit@1.12.5)",
+        "",
         "warm-index-reuse only:",
-        "  --kit-command <command>                            my-dev-kit command used to build one index per benchmark",
-        "                                                      project and retrieve per task",
-        "                                                      (default: npx @dailephd/my-dev-kit@latest)",
         "                                                      Legacy/default mode uses the deterministic fake agent only;",
         "                                                      agent options below are not accepted by this plugin.",
         "  --campaign <preset>                                 Runs the selected real-agent campaign preset: codex-full,",
@@ -173,11 +178,13 @@ export function renderExperimentRunHelp() {
         "                                                      replaced by a context-size estimate. A campaign run",
         "                                                      automatically produces the report JSON/text/HTML, the",
         "                                                      four warm-index SVG plots, a best-effort report PNG",
-        "                                                      screenshot, and a gallery manifest/index. A missing",
-        "                                                      Playwright/browser runtime makes screenshot capture skip",
-        "                                                      without invalidating the campaign; an unexpected",
-        "                                                      screenshot failure makes the command unsuccessful after",
-        "                                                      preserving the report, plots, and gallery. Legacy",
+        "                                                      screenshot, and a gallery manifest/index. The automatic",
+        "                                                      campaign report screenshot is best-effort presentation",
+        "                                                      evidence: skipped or failed capture is reported explicitly",
+        "                                                      (failed remains failed), no PNG is fabricated, and",
+        "                                                      screenshot status alone does not fail an otherwise",
+        "                                                      successful campaign command. Core experiment, report,",
+        "                                                      plot, and gallery failures remain fatal. Legacy",
         "                                                      non-campaign warm-index runs retain their existing",
         "                                                      report-only behavior.",
         "",

package/dist/src/commands/runExperimentDescribeCommand.js

@@ -147,9 +147,12 @@ function describeExamples(plugin) {
     const screenshotOption = hasConfigField(plugin, "agents") ? " --no-screenshot" : "";
     const examples = [
         `my-dev-kit-lab experiment describe --experiment ${plugin.metadata.id}`,
-        `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id}${agentOptions}`,
-        `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target "Z:\\Users\\newuser\\Projects\\my-dev-kit-v1"${agentOptions}${screenshotOption}`
+        `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id}${agentOptions}`
     ];
+    // Only advertise an explicit --target example for plugins that support external targets.
+    if (plugin.metadata.supportedTargets.includes("external-local")) {
+        examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target "Z:\\Users\\newuser\\Projects\\my-dev-kit-v1"${agentOptions}${screenshotOption}`);
+    }
     // Only advertise campaign examples for plugins that declare the campaignPreset config field.
     if (hasConfigField(plugin, "campaignPreset")) {
         examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign codex-full --include-real-agents --out <dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign claude-full --include-real-agents --case warm-medium-complete-idempotent --out <dir>`);

package/dist/src/commands/runExperimentRunCommand.js

@@ -76,7 +76,6 @@ export async function runExperimentRunCommandFromArgs(argv, options = {}) {
         // Automatic presentation (report already produced above) applies only to warm-index-reuse
         // campaign runs; legacy fake-agent warm-index runs and context-strategy-comparison stay
         // report-only, unchanged.
-        let screenshotFailed = false;
         if (args.experimentId === warmIndexReusePlugin.metadata.id && args.config.campaignPreset) {
             const executionArtifactPath = readMetadataString(result.metadata?.executionArtifactPath);
             const campaignAgentId = readMetadataString(result.metadata?.campaignAgentId);
@@ -106,7 +105,6 @@ export async function runExperimentRunCommandFromArgs(argv, options = {}) {
             else {
                 outputLines.push("Screenshot: failed");
                 outputLines.push(presentation.screenshot.error ?? "Screenshot capture failed.");
-                screenshotFailed = true;
             }
             outputLines.push(`Gallery manifest: ${presentation.gallery.manifestPath}`);
             outputLines.push(`Gallery index: ${presentation.gallery.indexPath}`);
@@ -114,8 +112,6 @@ export async function runExperimentRunCommandFromArgs(argv, options = {}) {
         console.log(outputLines.join("\n"));
         if (result.status === "failed")
             return 1;
-        if (screenshotFailed)
-            return 1;
         return 0;
     }
     catch (error) {

package/dist/src/evaluation/benchmarkMetadata.js

@@ -280,7 +280,11 @@ function validateWarmIndexSourceRoots(sourceRoots, profile, label) {
     return errors;
 }
 function normalizeRelativeRoot(value) {
-    return path.posix.normalize(value.replace(/\\/g, "/")).replace(/\/+$/, "");
+    const normalized = path.posix.normalize(value.replaceAll("\\", "/"));
+    let end = normalized.length;
+    while (end > 0 && normalized[end - 1] === "/")
+        end -= 1;
+    return normalized.slice(0, end);
 }
 function sameOrderedValues(left, right) {
     return left.length === right.length && left.every((value, index) => value === right[index]);

package/dist/src/evaluation/parseAgentAnswer.js

@@ -77,11 +77,24 @@ function collectJsonCandidates(text) {
     if (trimmed.startsWith("{") && trimmed.endsWith("}")) {
         candidates.push(trimmed);
     }
-    for (const match of text.matchAll(/```(?:json)?\s*([\s\S]*?)```/gi)) {
-        const body = match[1]?.trim();
+    let cursor = 0;
+    while (cursor < text.length) {
+        const opening = text.indexOf("```", cursor);
+        if (opening < 0)
+            break;
+        let bodyStart = opening + 3;
+        if (text.slice(bodyStart, bodyStart + 4).toLowerCase() === "json")
+            bodyStart += 4;
+        while (bodyStart < text.length && isWhitespace(text[bodyStart]))
+            bodyStart += 1;
+        const closing = text.indexOf("```", bodyStart);
+        if (closing < 0)
+            break;
+        const body = text.slice(bodyStart, closing).trim();
         if (body?.startsWith("{")) {
             candidates.push(body);
         }
+        cursor = closing + 3;
     }
     return candidates;
 }
@@ -89,39 +102,103 @@ function collectFieldValues(text) {
     const fields = new Map();
     let currentKey;
     for (const line of text.split(/\r?\n/)) {
-        const fieldMatch = line.match(/^\s*(?:[-*]\s*)?(?:[*_`]{0,2})([A-Za-z][A-Za-z0-9 _-]{1,40})(?:[*_`]{0,2})\s*:\s*(.*)$/);
-        if (fieldMatch) {
-            currentKey = normalizeKey(fieldMatch[1]);
+        const field = parseStructuredFieldLine(line);
+        if (field) {
+            currentKey = normalizeKey(field.label);
             const current = fields.get(currentKey) ?? [];
-            current.push(stripMarkupOnlyValue(fieldMatch[2] ?? ""));
+            current.push(stripMarkupOnlyValue(field.value));
             fields.set(currentKey, current);
             continue;
         }
-        if (currentKey && (/^\s+[-*]?\s*\S/.test(line) || /^\s*[-*]\s+\S/.test(line))) {
-            fields.get(currentKey)?.push(line.trim().replace(/^[-*]\s*/, ""));
+        if (currentKey) {
+            const continuation = parseFieldContinuation(line);
+            if (continuation !== undefined)
+                fields.get(currentKey)?.push(continuation);
         }
     }
     return fields;
 }
+function isWhitespace(character) {
+    return character.trim().length === 0;
+}
+function parseStructuredFieldLine(line) {
+    let start = 0;
+    while (start < line.length && isWhitespace(line[start]))
+        start += 1;
+    if (line[start] === "-" || line[start] === "*") {
+        start += 1;
+        while (start < line.length && isWhitespace(line[start]))
+            start += 1;
+    }
+    let leadingMarkup = 0;
+    while (start < line.length && leadingMarkup < 2 && "*_`".includes(line[start])) {
+        start += 1;
+        leadingMarkup += 1;
+    }
+    const colon = line.indexOf(":", start);
+    if (colon < 0)
+        return undefined;
+    let labelEnd = colon;
+    let trailingMarkup = 0;
+    while (labelEnd > start && trailingMarkup < 2 && "*_`".includes(line[labelEnd - 1])) {
+        labelEnd -= 1;
+        trailingMarkup += 1;
+    }
+    const label = line.slice(start, labelEnd);
+    if (label.length < 2 || label.length > 41 || !isAsciiLetter(label[0]))
+        return undefined;
+    for (let index = 1; index < label.length; index += 1) {
+        const character = label[index];
+        if (!isAsciiLetter(character) && !(character >= "0" && character <= "9") && character !== " " && character !== "_" && character !== "-")
+            return undefined;
+    }
+    let valueStart = colon + 1;
+    while (valueStart < line.length && isWhitespace(line[valueStart]))
+        valueStart += 1;
+    return { label, value: line.slice(valueStart) };
+}
+function isAsciiLetter(character) {
+    return (character >= "a" && character <= "z") || (character >= "A" && character <= "Z");
+}
+function parseFieldContinuation(line) {
+    let index = 0;
+    while (index < line.length && isWhitespace(line[index]))
+        index += 1;
+    const indented = index > 0;
+    if (index === line.length)
+        return undefined;
+    if (line[index] === "-" || line[index] === "*") {
+        index += 1;
+        const whitespaceStart = index;
+        while (index < line.length && isWhitespace(line[index]))
+            index += 1;
+        if ((!indented && index === whitespaceStart) || index === line.length)
+            return undefined;
+        return line.slice(index).trim();
+    }
+    if (!indented)
+        return undefined;
+    return line.slice(index).trim();
+}
 function parseMarkdownSection(text, headings) {
     const lines = text.split(/\r?\n/);
     const values = [];
     let inSection = false;
     for (const line of lines) {
-        const heading = line.match(/^\s*#{1,6}\s*(.+?)\s*$/);
-        if (heading) {
-            inSection = headings.some((candidate) => normalizeKey(candidate) === normalizeKey(heading[1]));
+        const heading = parseMarkdownHeading(line);
+        if (heading !== undefined) {
+            inSection = headings.some((candidate) => normalizeKey(candidate) === normalizeKey(heading));
             continue;
         }
         if (!inSection) {
             continue;
         }
-        if (/^\s*#{1,6}\s+/.test(line)) {
+        if (isMarkdownHeading(line)) {
             break;
         }
-        const bullet = line.match(/^\s*[-*]\s+(.+)$/);
-        if (bullet) {
-            values.push(bullet[1].trim());
+        const bullet = parseMarkdownBullet(line);
+        if (bullet !== undefined) {
+            values.push(bullet);
         }
         else if (line.includes(",")) {
             values.push(...splitListValues([line]));
@@ -129,6 +206,41 @@ function parseMarkdownSection(text, headings) {
     }
     return unique(values);
 }
+function parseMarkdownHeading(line) {
+    let index = 0;
+    while (index < line.length && isWhitespace(line[index]))
+        index += 1;
+    const hashStart = index;
+    while (index < line.length && line[index] === "#" && index - hashStart < 6)
+        index += 1;
+    if (index === hashStart || (index < line.length && line[index] === "#"))
+        return undefined;
+    const heading = line.slice(index).trim();
+    return heading.length > 0 ? heading : undefined;
+}
+function isMarkdownHeading(line) {
+    let index = 0;
+    while (index < line.length && isWhitespace(line[index]))
+        index += 1;
+    const start = index;
+    while (index < line.length && line[index] === "#" && index - start < 6)
+        index += 1;
+    return index > start && index < line.length && isWhitespace(line[index]);
+}
+function parseMarkdownBullet(line) {
+    let index = 0;
+    while (index < line.length && isWhitespace(line[index]))
+        index += 1;
+    if (line[index] !== "-" && line[index] !== "*")
+        return undefined;
+    index += 1;
+    if (index >= line.length || !isWhitespace(line[index]))
+        return undefined;
+    while (index < line.length && isWhitespace(line[index]))
+        index += 1;
+    const content = line.slice(index).trim();
+    return content.length > 0 ? content : undefined;
+}
 function enrichFacts(parsed, text, answerKey) {
     parsed.expectedFactsFound = normalizeFactMatches(parsed.expectedFactsFound, text, answerKey);
     if (parsed.answerText.length === 0) {
@@ -168,15 +280,51 @@ function splitListValues(values) {
         .filter(Boolean));
 }
 function cleanListItem(value) {
-    const codeSpan = value.match(/`([^`]+)`/);
-    const cleaned = (codeSpan?.[1] ?? value)
-        .trim()
-        .replace(/^[-*]\s*/, "")
-        .replace(/^["'`]+|["'`.]+$/g, "")
-        .replace(/\s+[--]\s+.*$/g, "")
-        .replace(/\s+[-–—]\s+.*$/g, "")
-        .trim();
-    return cleaned;
+    let source = value;
+    let searchFrom = 0;
+    while (searchFrom < value.length) {
+        const open = value.indexOf("`", searchFrom);
+        if (open < 0)
+            break;
+        const close = value.indexOf("`", open + 1);
+        if (close < 0)
+            break;
+        if (close > open + 1) {
+            source = value.slice(open + 1, close);
+            break;
+        }
+        searchFrom = close + 1;
+    }
+    let start = 0;
+    let end = source.length;
+    while (start < end && isWhitespace(source[start]))
+        start += 1;
+    while (end > start && isWhitespace(source[end - 1]))
+        end -= 1;
+    if (source[start] === "-" || source[start] === "*") {
+        start += 1;
+        while (start < end && isWhitespace(source[start]))
+            start += 1;
+    }
+    while (start < end && (source[start] === '"' || source[start] === "'" || source[start] === "`"))
+        start += 1;
+    while (end > start && (source[end - 1] === '"' || source[end - 1] === "'" || source[end - 1] === "`" || source[end - 1] === "."))
+        end -= 1;
+    let whitespaceStart = -1;
+    for (let index = start; index < end; index += 1) {
+        const character = source[index];
+        if (isWhitespace(character)) {
+            if (whitespaceStart < 0)
+                whitespaceStart = index;
+            continue;
+        }
+        if ((character === "-" || character === "–" || character === "—") && whitespaceStart >= start && index + 1 < end && isWhitespace(source[index + 1])) {
+            end = whitespaceStart;
+            break;
+        }
+        whitespaceStart = -1;
+    }
+    return source.slice(start, end).trim();
 }
 function stripMarkupOnlyValue(value) {
     return /^[*_`\s]+$/.test(value) ? "" : value;

package/dist/src/evaluation/runMyDevKitRetrieval.js

@@ -55,13 +55,118 @@ function skippedRetrieval(caseId, warnings, commands, durationMs) {
         durationMs
     };
 }
+const APPLIED_SCOPES = ["none", "changed-files", "affected-neighborhood", "full"];
+const SELECTION_STATUSES = ["not-needed", "applied", "fallback-full"];
+const NEIGHBORHOOD_COUNT_KEYS = ["seedFileCount", "seedSymbolCount", "affectedNodeCount", "affectedEdgeCount"];
+function isNonNegativeInteger(value) {
+    return typeof value === "number" && Number.isInteger(value) && value >= 0;
+}
+/**
+ * Validates the per-invocation `incrementalRefresh` object from `my-dev-kit index --incremental --json`
+ * stdout against the scope Lab requested. Returns validated evidence or an explicit error; never coerces.
+ * The fallback reason is intentionally an open string: upstream reason codes may evolve additively.
+ */
+export function parseIncrementalRefreshEvidence(stdout, requestedScope) {
+    const payload = parseJsonIfPossible(stdout);
+    if (payload === undefined)
+        return { ok: false, error: "index stdout is not valid JSON" };
+    if (!payload || typeof payload !== "object" || Array.isArray(payload)) {
+        return { ok: false, error: "index JSON result is not an object" };
+    }
+    const raw = payload.incrementalRefresh;
+    if (!raw || typeof raw !== "object" || Array.isArray(raw)) {
+        return { ok: false, error: "index JSON result has no incrementalRefresh object" };
+    }
+    const r = raw;
+    if (r.requestedScope !== requestedScope) {
+        return {
+            ok: false,
+            error: `incrementalRefresh.requestedScope ${JSON.stringify(r.requestedScope)} does not match requested scope ${requestedScope}`
+        };
+    }
+    if (!APPLIED_SCOPES.includes(r.appliedScope)) {
+        return { ok: false, error: `incrementalRefresh.appliedScope ${JSON.stringify(r.appliedScope)} is not a known value` };
+    }
+    if (!SELECTION_STATUSES.includes(r.selectionStatus)) {
+        return { ok: false, error: `incrementalRefresh.selectionStatus ${JSON.stringify(r.selectionStatus)} is not a known value` };
+    }
+    if (r.fallbackReason !== null && typeof r.fallbackReason !== "string") {
+        return { ok: false, error: "incrementalRefresh.fallbackReason must be a string or null" };
+    }
+    for (const key of NEIGHBORHOOD_COUNT_KEYS) {
+        if (r[key] !== null && !isNonNegativeInteger(r[key])) {
+            return { ok: false, error: `incrementalRefresh.${key} must be a non-negative integer or null` };
+        }
+    }
+    for (const key of ["forcedNeighborReanalysisFileCount", "freshExtractionFileCount", "reusedFileCount"]) {
+        if (!isNonNegativeInteger(r[key])) {
+            return { ok: false, error: `incrementalRefresh.${key} must be a non-negative integer` };
+        }
+    }
+    if (!Array.isArray(r.forcedNeighborSample) || r.forcedNeighborSample.some((item) => typeof item !== "string")) {
+        return { ok: false, error: "incrementalRefresh.forcedNeighborSample must be an array of strings" };
+    }
+    const appliedScope = r.appliedScope;
+    const selectionStatus = r.selectionStatus;
+    const fallbackReason = r.fallbackReason;
+    if (selectionStatus === "applied") {
+        if (appliedScope !== requestedScope) {
+            return { ok: false, error: "incrementalRefresh applied status requires appliedScope equal to requestedScope" };
+        }
+        if (fallbackReason !== null)
+            return { ok: false, error: "incrementalRefresh applied status requires a null fallbackReason" };
+    }
+    else if (selectionStatus === "fallback-full") {
+        if (appliedScope !== "full")
+            return { ok: false, error: "incrementalRefresh fallback-full status requires appliedScope full" };
+        if (fallbackReason === null || fallbackReason.length === 0) {
+            return { ok: false, error: "incrementalRefresh fallback-full status requires a nonempty fallbackReason" };
+        }
+    }
+    else {
+        if (appliedScope !== "none")
+            return { ok: false, error: "incrementalRefresh not-needed status requires appliedScope none" };
+        if (fallbackReason !== null)
+            return { ok: false, error: "incrementalRefresh not-needed status requires a null fallbackReason" };
+    }
+    if (requestedScope === "affected-neighborhood" && selectionStatus === "applied") {
+        for (const key of NEIGHBORHOOD_COUNT_KEYS) {
+            if (r[key] === null) {
+                return { ok: false, error: `incrementalRefresh.${key} must be numeric for an applied affected-neighborhood refresh` };
+            }
+        }
+    }
+    return {
+        ok: true,
+        evidence: {
+            requestedScope,
+            appliedScope,
+            selectionStatus,
+            fallbackReason,
+            seedFileCount: r.seedFileCount,
+            seedSymbolCount: r.seedSymbolCount,
+            affectedNodeCount: r.affectedNodeCount,
+            affectedEdgeCount: r.affectedEdgeCount,
+            forcedNeighborReanalysisFileCount: r.forcedNeighborReanalysisFileCount,
+            forcedNeighborSample: [...r.forcedNeighborSample],
+            freshExtractionFileCount: r.freshExtractionFileCount,
+            reusedFileCount: r.reusedFileCount
+        }
+    };
+}
 /**
  * Runs exactly one my-dev-kit `index` command for the target and writes the index only to
  * `indexDir`. Performs no search/lookup/slice/source work. With `requireKit` a failed index
  * throws; otherwise the failure is returned as `ok: false` with a warning.
+ *
+ * `mode` defaults to a full index. An incremental mode adds `--incremental --refresh-scope <scope>`
+ * and requires valid per-invocation `incrementalRefresh` JSON evidence on stdout; a successful
+ * process without that evidence is not a usable incremental build. Upstream full-fallback and
+ * no-change outcomes are valid evidence and are returned as-is.
  */
 export async function buildMyDevKitIndex(options) {
     const warnings = [];
+    const mode = options.mode ?? { kind: "full" };
     const command = await runMeasuredCommand({
         commandId: "index",
         commandString: options.kitCommand,
@@ -74,22 +179,30 @@ export async function buildMyDevKitIndex(options) {
             ...options.target.sourceRoots.flatMap((sourceRoot) => ["--src", sourceRoot]),
             "--out",
             options.indexDir,
+            ...(mode.kind === "incremental" ? ["--incremental", "--refresh-scope", mode.refreshScope] : []),
             "--json"
         ]
     });
+    const result = { indexDir: options.indexDir, durationMs: command.durationMs, command, mode };
     if (!command.ok) {
         if (options.requireKit) {
             throw new Error(command.error || `my-dev-kit index failed with exit code ${command.exitCode}`);
         }
         warnings.push("my-dev-kit index command was unavailable or failed.");
+        return { ...result, ok: false, warnings, incrementalRefresh: null };
     }
-    return {
-        ok: command.ok,
-        indexDir: options.indexDir,
-        durationMs: command.durationMs,
-        warnings,
-        command
-    };
+    if (mode.kind === "full") {
+        return { ...result, ok: true, warnings, incrementalRefresh: null };
+    }
+    const parsed = parseIncrementalRefreshEvidence(command.stdout, mode.refreshScope);
+    if (!parsed.ok) {
+        const message = `my-dev-kit incremental index (${mode.refreshScope}) returned no valid incrementalRefresh evidence: ${parsed.error}`;
+        if (options.requireKit)
+            throw new Error(message);
+        warnings.push(message);
+        return { ...result, ok: false, warnings, incrementalRefresh: null };
+    }
+    return { ...result, ok: true, warnings, incrementalRefresh: parsed.evidence };
 }
 /**
  * Runs `<kit-command> --version` once through the shared measured-command runner and returns