@dailephd/my-dev-kit-lab 0.6.1 → 0.6.3

package/CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
+## [0.6.3] - 2026-09-30
+
+### Four-treatment partial-refresh experiments
+
+- Replaced CodeQL-flagged polynomial regular-expression parsing paths with linear scanning and strengthened credential evidence fingerprints with context-salted scrypt derivation.
+
+- Extended the `incremental-change-staleness` plugin from two treatments to four, in fixed order: `stale-index` (`my-dev-kit-no-refresh`), `changed-files-refresh` (`my-dev-kit-changed-files-refresh`), `affected-neighborhood-refresh` (`my-dev-kit-affected-neighborhood-refresh`), and `full-refresh` (`my-dev-kit-full-refresh`). No new experiment plugin was added.
+- Changed the default kit command of `incremental-change-staleness` from `npx @dailephd/my-dev-kit@1.12.4` to `npx @dailephd/my-dev-kit@1.12.5`, the first published version with `index --incremental --refresh-scope changed-files|affected-neighborhood`. Lab requests the scope internally; no Lab-level `--refresh-scope` flag exists.
+- Added a four-treatment lifecycle: four isolated disposable targets, equivalent trusted baselines, containment-checked whole-directory index clones for the two partial treatments, a mutation barrier, and per-treatment freshness validation. `stale-index` performs no post-mutation index invocation.
+- Added truthful refresh evidence: upstream `requestedScope`, `appliedScope`, `selectionStatus`, `fallbackReason`, fresh-extraction, reused-file, forced-neighbor, and seed/affected counts, and a Lab `refreshRealization` of `NO_REFRESH`, `APPLIED_PARTIAL`, `FALLBACK_FULL`, or `FULL_REFRESH`. A partial treatment that falls back is `not-comparable-as-partial-refresh`.
+- Added exactly three reference comparisons against `full-refresh`: `stale-index` (unchanged v0.6.2 stale-risk vocabulary) and the two partial treatments (`observed-regression-relative-to-full`, `no-observed-regression-relative-to-full`, `inconclusive`, `not-comparable-as-partial-refresh`). There is no winner, ranking, safety score, or automatic refresh policy.
+- New runs write execution schema `my-dev-kit-lab-incremental-change-staleness-execution-v2` and report schema `my-dev-kit-lab-incremental-change-staleness-report-v2` (JSON, text, HTML); historical V1 artifacts and reports remain readable. Added additive numeric `upstream-refresh-*` metrics for the partial treatments, kept separate from the Lab affected-neighborhood metrics.
+- `experiment run --help` now states that `--target` applies only to plugins that support external targets and that self-only plugins reject an explicit external `--target`.
+- Automatic warm-index campaign report screenshots are best-effort presentation evidence: failed capture stays explicitly failed with its warning and no fabricated PNG, but does not fail an otherwise successful campaign. Required campaign, report, plot, and gallery failures remain fatal.
+- Updated the packed-package acceptance gate (`npm run verify:packed-package`) to pin the real registry package `@dailephd/my-dev-kit@1.12.5` and to verify all six scenarios with four treatments, applied partial refresh, operational discrimination, and installed-package, upstream-package, and canonical-benchmark immutability.
+- Added repository-controlled GitHub CodeQL advanced-setup analysis for JavaScript/TypeScript and Actions; the release candidate passed exact-SHA analysis with no open applicable alerts.
+- No new plugin, `graph-diff` dependency, true-symbol-diff metric, multi-hop traversal, automatic refresh policy, external-target support, or new plot, screenshot, or gallery item was added.
+
+## [0.6.2] - 2026-09-27
+
+### Incremental-change and staleness plugin
+
+- Added the `incremental-change-staleness` experiment plugin (`src/experiments/plugins/incrementalChangeStaleness/`), registered alongside `context-strategy-comparison` and `warm-index-reuse` with registry status `experimental` and exactly two treatments, `stale-index` and `full-refresh`.
+- Added a separate versioned scenario catalog (`benchmarks/contracts/incremental-change-staleness-scenarios.json`, schema `1.0.0`) with six frozen controlled-change scenarios (`U1` unrelated-file, `L2` local-implementation, `E1` exported-symbol, `P1` public-API, `I1` import-graph, `T1` test-only), a bounded declarative mutation contract (relative path, pre/post SHA-256, ordered literal exact-preimage replacements, exactly-one-match validation), and an `inherit`/`scenario` answer-key policy reusing the existing benchmark answer-key vocabulary.
+- Added disposable treatment targets: each scenario runs against two independent disposable copies of the same immutable benchmark project, proving pre-mutation and post-mutation controlled-source equivalence before and after the shared deterministic mutation.
+- Added the stale/full-refresh index lifecycle: both treatments build a baseline index/snapshot/graph before mutation; `stale-index` keeps retrieving from that baseline index, while `full-refresh` builds a distinct complete post-mutation index and retrieves from it. Affected-neighborhood evidence (the released v0.6.1 six numeric metrics, `relationship`, and `reindexRecommendation`) is asserted symmetric across both treatments before retrieval; the six-metric conversion is now shared through `src/evaluation/affectedNeighborhoodMetrics.ts`.
+- Added deterministic fake-agent retrieval/correctness evidence, required-file evidence (`present`/`missing`/`unknown`), a `correctnessRelation` and `requiredFileEvidenceRelation` (`stale-worse`/`same`/`stale-better`/`unknown`), and a conservative `staleRiskClassification` (`observed-stale-regression`/`no-observed-stale-regression`/`inconclusive`) with no numeric stale-risk score, no treatment winner, and no safety verdict.
+- Added the persisted execution artifact `incremental-change-staleness-execution.json` (schema `my-dev-kit-lab-incremental-change-staleness-execution-v1`) and plugin `report.json`/`report.txt`/`report.html` presentation built from that persisted evidence only, with fixed limitations text.
+- `experiment run --kit-command` now also accepts `incremental-change-staleness`, in addition to `warm-index-reuse`; it remains rejected for `context-strategy-comparison`.
+- Validated the exact npm tarball end to end against the real published `@dailephd/my-dev-kit@1.12.4`: all six scenarios ran through the installed CLI, producing the execution artifact and all three reports while keeping the installed package, canonical benchmarks, and source checkout immutable.
+- No partial refresh, no `graph-diff` dependency, no true-symbol-diff metric, and no new plot, screenshot, gallery item, or top-level CLI command were added.
+
 ## [0.6.1] - 2026-09-26
 
 ### Affected-neighborhood experiments

package/README.md

@@ -6,7 +6,7 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.6.1 (affected-neighborhood experiments), and the package and installed CLI are version 0.6.1. The previous release is v0.6.0 (index freshness and changed-file detection). Version v0.6.2 (incremental-change and staleness plugin) is the next planned version. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
+The latest release is v0.6.3 (four-treatment partial-refresh experiments), and the published package and installed CLI are version 0.6.3. The previous release is v0.6.2 (incremental-change and staleness experiments). The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
@@ -14,9 +14,10 @@ v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--vers
 
 - **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
 - **Measure warm-index reuse (v0.5.0–v0.5.1, released):** the `warm-index-reuse` experiment plugin builds one my-dev-kit index per benchmark project, reuses it across that project's tasks, and compares every task with a matched `raw-full-file` baseline. Reports separate one-time index-build cost from per-task retrieval cost, show amortized index cost and cumulative measurements, and include deterministic fake-agent correctness and token evidence; `plots generate` renders four warm-index SVG charts. The v0.5.1 release adds a dedicated 12-task benchmark corpus (`benchmarks/contracts/warm-index-benchmark-cases.json`: six medium and six large/mixed tasks tagged `localized`, `cross-module`, or `broad-change`) for comparing warm-index behavior as task count grows. Estimated context tokens (a character-based context-size estimate) and fake-agent total tokens (simulated harness telemetry) are reported separately, and neither is provider billing telemetry.
-- **Run selectable real-agent warm-index campaigns (v0.5.2, released):** `--campaign codex-full`, `claude-full`, or `codex-timeout-isolation` (with `--include-real-agents`) runs exactly one provider — Codex or Claude — against the exact raw and warm contexts the experiment already measured, over the frozen production corpus each preset owns. Provider outcomes (`completed`, `failed`, `timeout`, `invalid-output`, `agent-unavailable`, `agent-limit-reached`, `skipped`) are classified and reported separately from warm-index infrastructure status, so a campaign can complete its indexing/retrieval work even when provider evidence is partial or unavailable. A campaign run automatically produces the report, the same four warm-index SVG charts, a best-effort report screenshot (skipped without a browser runtime, never fatal), and a three-item gallery (report, plots, bounded execution evidence). Legacy runs without `--campaign` keep the existing deterministic fake-agent, report-only behavior unchanged. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
+- **Run selectable real-agent warm-index campaigns (v0.5.2, released):** `--campaign codex-full`, `claude-full`, or `codex-timeout-isolation` (with `--include-real-agents`) runs exactly one provider — Codex or Claude — against the exact raw and warm contexts the experiment already measured, over the frozen production corpus each preset owns. Provider outcomes (`completed`, `failed`, `timeout`, `invalid-output`, `agent-unavailable`, `agent-limit-reached`, `skipped`) are classified and reported separately from warm-index infrastructure status, so a campaign can complete its indexing/retrieval work even when provider evidence is partial or unavailable. A campaign run automatically produces the report, the same four warm-index SVG charts, and a three-item gallery (report, plots, bounded execution evidence). Its report screenshot is best-effort presentation evidence: unavailable browser runtime or failed capture is reported explicitly, no PNG is fabricated, and neither outcome fails an otherwise successful campaign. Legacy runs without `--campaign` keep the existing deterministic fake-agent, report-only behavior unchanged. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment) and [docs/METRICS.md](docs/METRICS.md#warm-index-reuse-metrics).
 - **Record index freshness evidence (v0.6.0):** a warm-index run captures an index snapshot after its one index build — the exact indexed file set from the my-dev-kit index contract, each with SHA-256 content identity, size, and modified-time metadata, plus the my-dev-kit version and index-command evidence — and assesses freshness immediately before each task's warm retrieval as `fresh`, `stale`, `partially-stale`, or `unknown`. The evidence appears in `warm-index-execution.json` and in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: it covers only files the snapshot lists (not the whole repository), never changes retrieval, execution or provider status, and adds no CLI flag, metric, plot, gallery item, or reindex recommendation in v0.6.0. See [docs/METRICS.md](docs/METRICS.md#index-freshness-evidence-v060) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
-- **Assess affected-neighborhood evidence (v0.6.1, current):** using the baseline my-dev-kit graph of the same prepared warm index (loaded once per session), each task's confirmed modified or missing indexed files, and the baseline symbols those files contain, are mapped to graph nodes. The affected neighborhood is those seed nodes plus their direct one-hop neighbors (edges of any kind, either direction). The task's `expectedFiles` and file-scoped `expectedSymbols` are mapped to graph nodes and compared with that neighborhood, which yields six numeric warm-side metrics (`changedFileCount`, `changedSymbolCount`, `affectedNodeCount`, `affectedEdgeCount`, `taskOverlapCount`, `taskOverlapPercent`) plus categorical evidence: a `related`, `unrelated`, or `unknown` relationship and a `recommended`, `not-indicated`, or `unknown` `reindexRecommendation`. Incomplete evidence is reported as `unknown`, never as unrelated. The assessment is recorded additively in `warm-index-execution.json` and shown in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: retrieval, execution status, and provider status are unchanged, nothing is reindexed automatically, and no CLI flag, experiment plugin, plot, or gallery item was added. See [docs/METRICS.md](docs/METRICS.md#affected-neighborhood-metrics-v061) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
+- **Compare no-refresh, partial-refresh, and full-refresh retrieval after a controlled change (v0.6.3, current release):** the `incremental-change-staleness` experiment plugin runs six frozen controlled-change scenario families (unrelated-file, local-implementation, exported-symbol, public-API, import-graph, and test-only changes) against four disposable target copies — `stale-index` (no refresh), `changed-files-refresh` and `affected-neighborhood-refresh` (real partial refreshes performed by my-dev-kit 1.12.5 through `index --incremental --refresh-scope`, selected internally with no Lab-level flag), and `full-refresh` (a complete new post-change index, used as a comparison reference rather than a preferred treatment). Each partial treatment truthfully records whether upstream applied the requested partial scope or fell back to a full rebuild (`APPLIED_PARTIAL` or `FALLBACK_FULL`); a fallback is not counted as partial-refresh evidence. Three comparisons against `full-refresh` are persisted with retrieval, deterministic fake-agent correctness, and required-file evidence in `incremental-change-staleness-execution.json` (schema `my-dev-kit-lab-incremental-change-staleness-execution-v2`) and the existing `report.json`/`report.txt`/`report.html`; historical v0.6.2 (V1) artifacts stay readable. There is no winner, ranking, or safety claim, and the plugin is self-only (`--target` is not accepted). `--kit-command` also accepts `incremental-change-staleness` and now defaults to my-dev-kit 1.12.5. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md) and [docs/METRICS.md](docs/METRICS.md).
+- **Assess affected-neighborhood evidence (v0.6.1, released; retained in the current package):** using the baseline my-dev-kit graph of the same prepared warm index (loaded once per session), each task's confirmed modified or missing indexed files, and the baseline symbols those files contain, are mapped to graph nodes. The affected neighborhood is those seed nodes plus their direct one-hop neighbors (edges of any kind, either direction). The task's `expectedFiles` and file-scoped `expectedSymbols` are mapped to graph nodes and compared with that neighborhood, which yields six numeric warm-side metrics (`changedFileCount`, `changedSymbolCount`, `affectedNodeCount`, `affectedEdgeCount`, `taskOverlapCount`, `taskOverlapPercent`) plus categorical evidence: a `related`, `unrelated`, or `unknown` relationship and a `recommended`, `not-indicated`, or `unknown` `reindexRecommendation`. Incomplete evidence is reported as `unknown`, never as unrelated. The assessment is recorded additively in `warm-index-execution.json` and shown in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: retrieval, execution status, and provider status are unchanged, nothing is reindexed automatically, and no CLI flag, experiment plugin, plot, or gallery item was added. See [docs/METRICS.md](docs/METRICS.md#affected-neighborhood-metrics-v061) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
 - **Validate Android projects:** run nineteen static checks by default, with Gradle operations, external tools, and network access available only through explicit opt-in flags.
@@ -181,9 +182,9 @@ npm run experiment:run -- \
   --out lab-output/warm-index-campaign
 ```
 
-This automatically writes `report.json`/`.txt`/`.html`, the four warm-index charts under `plots/`, a best-effort `report.png`, and a three-item `gallery/` (report, plots, bounded execution evidence) beneath `--out`. `--timeout-ms` overrides the preset's default per-agent timeout (240000 ms). `codex-timeout-isolation` narrows the corpus to three larger cases for exercising provider timeout/partial-outcome handling without running the full 12-case corpus.
+This automatically writes `report.json`/`.txt`/`.html`, the four warm-index charts under `plots/`, an optional best-effort `report.png`, and a three-item `gallery/` (report, plots, bounded execution evidence) beneath `--out`. Screenshot capture failure remains `failed`, is reported in the command output and gallery warning, and does not fail an otherwise successful campaign or fabricate a PNG. Required experiment, report, plot, and gallery failures remain fatal. `--timeout-ms` overrides the preset's default per-agent timeout (240000 ms). `codex-timeout-isolation` narrows the corpus to three larger cases for exercising provider timeout/partial-outcome handling without running the full 12-case corpus.
 
-When `--target` is omitted, the experiment runs in self mode against my-dev-kit-lab. When `--target <path>` is provided, the lab remains the tool root and the target project is inspected separately. Generated experiment outputs stay under lab-controlled output directories by default, not inside the target project.
+For experiment plugins that support explicit targets — `context-strategy-comparison` and ordinary (non-campaign) `warm-index-reuse` runs — omitting `--target` uses self mode against my-dev-kit-lab. When those plugins receive `--target <path>`, the lab remains the tool root and the target project is inspected separately. `incremental-change-staleness` does not accept `--target`; it always runs against the lab's bundled benchmark projects. Generated experiment outputs stay under lab-controlled output directories by default, not inside an inspected target project.
 
 ---
 
@@ -235,10 +236,11 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Claude and Codex token totals are used only when the CLI/run exposes them; missing provider token evidence stays explicitly unavailable (never zero, and never substituted with an estimated context-token count)
 - Codex and Claude campaign runs can produce timeouts, invalid-output, agent-unavailable, or agent-limit-reached outcomes; these are structured provider states, not warm-index infrastructure failures
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework has two plugins: `context-strategy-comparison` and the v0.5.0-introduced `warm-index-reuse`. Later plugins such as incremental-change and context-window scaling are future roadmap work
+- The generic experiment-plugin framework has three registered plugins: `context-strategy-comparison`, the v0.5.0-introduced `warm-index-reuse`, and the `incremental-change-staleness` plugin (introduced in v0.6.2; extended to four treatments in current release v0.6.3). Later plugins such as context-window scaling remain future roadmap work
 - `warm-index-reuse` without `--campaign` still uses the deterministic fake agent only. With `--campaign`, exactly one real provider (Codex or Claude) runs per campaign preset — there is no multi-agent matrix, retry, provider switching, or scheduler. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence when run without `--campaign`. No warm-index mode calculates a token-savings percentage, break-even task, winner, or ranking, and component-duration sums are not wall-clock latency
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Released v0.6.1 affected-neighborhood evidence is bounded to the baseline graph of the prepared index and to the snapshot-represented files: a changed symbol means the symbol was present in a confirmed changed indexed file, not that its own text was proven to change; `not-indicated` means only that this bounded analysis found no supporting evidence for reindexing, not that skipping a reindex is safe; unresolved or ambiguous expected symbols keep a task at `unknown` rather than unrelated
+- v0.6.3 (current release) partial-refresh evidence is scoped to the executed deterministic scenarios and the configured my-dev-kit 1.12.5-or-later tool. A partial treatment can fall back to a full rebuild and is then reported as not comparable as partial refresh, so an applied partial refresh in one run does not guarantee the next; `no-observed-regression-relative-to-full` is not a safety claim, and full refresh is a reference rather than a winner. Correctness remains deterministic fake-agent evidence, and `incremental-change-staleness` accepts no external `--target`.
 - Provider telemetry dashboards, semantic LLM judging, and cloud API billing integration are not yet implemented
 - The six new stage-context strategies have no CLI flags yet, are configured programmatically, and do not yet include plots, screenshots, or gallery integration
 - The published upstream artifacts the stage-context strategies read do not expose considered-but-unselected reads or unnecessary-read evidence; those metrics report `unavailable` rather than zero
@@ -257,7 +259,7 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 
 Android validation uses `--profile android`. Its default path is static and non-destructive: it starts zero Gradle processes, zero external tools, and zero network operations. Only confirmed `SecurityFinding` records can become audit issues; Android `CandidateEvidence` remains review-only evidence.
 
-Optional scanners are reported as `skipped` when unavailable, never as passed. The framework does not provide runtime isolation proof, device or APK/AAB analysis, signing verification, Play Console validation, automatic fixes, or manual pentesting. Manual pentest remains deferred until post-v1/version TBD.
+Optional local scanners are reported as `skipped` when unavailable, never as passed. This local availability rule does not make CodeQL coverage optional for release readiness: the exact candidate must pass the repository-controlled GitHub CodeQL workflow and have its Code Scanning alerts reviewed. The framework does not provide runtime isolation proof, device or APK/AAB analysis, signing verification, Play Console validation, automatic fixes, or manual pentesting. Manual pentest remains deferred until post-v1/version TBD.
 
 See [COMMANDS.md](docs/COMMANDS.md) for exact syntax and [Security Validation Framework](docs/security-validation-framework.md) for checks, evidence semantics, verdicts, and limitations.
 

package/benchmarks/contracts/incremental-change-staleness-scenarios.json

@@ -0,0 +1,360 @@
+{
+  "schemaVersion": "1.0.0",
+  "scenarios": [
+    {
+      "id": "U1",
+      "category": "unrelated-file-change",
+      "benchmarkProjectId": "task-analytics-large-mixed",
+      "baseCaseId": "warm-large-ts-leaderboard",
+      "answerPolicy": "inherit",
+      "expectedRelationship": "unrelated",
+      "mutation": {
+        "files": [
+          {
+            "path": "py/task_analytics/quality.py",
+            "expectedPreSha256": "5faab1bf018de6e647a11a2edffd641a51c0c05b399cb6f58a2c22b0ff035554",
+            "operations": [
+              {
+                "expectedPreimage": "metric.completion_rate >= 80 and metric.stale_tasks == 0",
+                "replacement": "metric.completion_rate >= 81 and metric.stale_tasks == 0"
+              }
+            ],
+            "expectedPostSha256": "9f59d256710a9345eb6e0f768e45c4676041b4cac215cd84cd1075cfd730ab18"
+          }
+        ]
+      },
+      "notes": "Negative control across the intentionally separate Python and TypeScript implementations. The follow-up TypeScript leaderboard query/answer is unaffected by this Python-only threshold change, so this record states the scenario planning expectation as unrelated. Runtime classification in later batches must still produce unknown instead of unrelated when evidence is incomplete, per v0.6.1 semantics; Batch 1 performs no runtime classification."
+    },
+    {
+      "id": "L2",
+      "category": "local-implementation-change",
+      "benchmarkProjectId": "task-workflow-medium-ts",
+      "baseCaseId": "warm-medium-complete-idempotent",
+      "answerPolicy": "scenario",
+      "scenarioQuery": "complete task by id after the completedAt timestamp change idempotent unknown task",
+      "scenarioAnswerKey": {
+        "expectedFiles": [
+          "src/services/completeTask.ts",
+          "src/store/taskStore.ts"
+        ],
+        "expectedSymbols": [
+          "completeTask",
+          "TaskWorkflowStore"
+        ],
+        "expectedFacts": [
+          {
+            "id": "warm062-l2-sets-new-completed-at",
+            "text": "The first completion sets completed to true and completedAt to \"2026-03-01T00:00:00.000Z\".",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-l2-still-idempotent",
+            "text": "When the task is already completed the updater returns it unchanged, so completing it again does not rewrite completedAt.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-l2-unknown-task-throws",
+            "text": "An unknown task id still throws \"Unknown task id: <id>\" from TaskWorkflowStore.updateTask.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-l2-targets-by-id",
+            "text": "completeTask(store, taskId) still changes exactly one task by id through TaskWorkflowStore.updateTask.",
+            "weight": 1,
+            "required": true
+          }
+        ],
+        "minimumCorrectFacts": 4,
+        "notes": "v0.6.2 L2 scenario: local implementation change to the completedAt literal. Exported name, callable signature, idempotence, and unknown-task behavior are preserved."
+      },
+      "mutation": {
+        "files": [
+          {
+            "path": "src/services/completeTask.ts",
+            "expectedPreSha256": "27f6833abb1c5f8fa7226ec8a4857ba267dd48272d7cf094c7a3462e425ca64c",
+            "operations": [
+              {
+                "expectedPreimage": "\"2026-02-01T00:00:00.000Z\"",
+                "replacement": "\"2026-03-01T00:00:00.000Z\""
+              }
+            ],
+            "expectedPostSha256": "0d148ddab0dbd43b9bc8eb40a09de9dfd4567b9097fb0f028b74ebf921798cc8"
+          }
+        ]
+      },
+      "notes": "Local implementation change: only the fixed completedAt literal changes."
+    },
+    {
+      "id": "E1",
+      "category": "exported-symbol-change",
+      "benchmarkProjectId": "task-analytics-large-mixed",
+      "baseCaseId": "warm-large-python-pipeline",
+      "answerPolicy": "scenario",
+      "scenarioQuery": "pipeline parse rows project metrics quality label health report after the healthy completion-rate boundary changed",
+      "scenarioAnswerKey": {
+        "expectedFiles": [
+          "py/task_analytics/pipeline.py",
+          "py/task_analytics/parser.py",
+          "py/task_analytics/metrics.py",
+          "py/task_analytics/quality.py",
+          "py/task_analytics/reporting.py"
+        ],
+        "expectedSymbols": [
+          "build_report_from_rows",
+          "parse_task_rows",
+          "calculate_project_metrics",
+          "determine_quality_label",
+          "build_health_report"
+        ],
+        "expectedFacts": [
+          {
+            "id": "warm062-e1-new-healthy-boundary",
+            "text": "determine_quality_label now returns \"healthy\" only when completion_rate is at least 85 (previously 80) and stale_tasks is 0.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-e1-signature-unchanged",
+            "text": "determine_quality_label keeps its exported name and single-argument signature; only its healthy-boundary behavior changed.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-e1-role-in-pipeline",
+            "text": "build_health_report still calls determine_quality_label for each metric produced by calculate_project_metrics inside build_report_from_rows.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-e1-deterministic-output",
+            "text": "The report remains one line per project in sorted project_id order, so fixed rows and current_day still produce the same output under the new boundary.",
+            "weight": 1,
+            "required": true
+          }
+        ],
+        "minimumCorrectFacts": 3,
+        "notes": "v0.6.2 E1 scenario: exported-symbol change. Export presence, symbol name, and callable signature remain stable; externally observable behavior changes."
+      },
+      "mutation": {
+        "files": [
+          {
+            "path": "py/task_analytics/quality.py",
+            "expectedPreSha256": "5faab1bf018de6e647a11a2edffd641a51c0c05b399cb6f58a2c22b0ff035554",
+            "operations": [
+              {
+                "expectedPreimage": "metric.completion_rate >= 80 and metric.stale_tasks == 0",
+                "replacement": "metric.completion_rate >= 85 and metric.stale_tasks == 0"
+              }
+            ],
+            "expectedPostSha256": "9cd4df5fd2fd29b99a1e82b744a7bd7afddb123a2fe1d861c1edcce21a32998c"
+          }
+        ]
+      },
+      "notes": "Same canonical quality.py file as U1, but with an independent scenario-specific mutation to threshold 85 instead of 81."
+    },
+    {
+      "id": "P1",
+      "category": "public-api-change",
+      "benchmarkProjectId": "task-analytics-large-mixed",
+      "baseCaseId": "warm-large-python-pipeline",
+      "answerPolicy": "scenario",
+      "scenarioQuery": "calculate_project_metrics signature stale_day_threshold parameter default behavior for callers that omit it",
+      "scenarioAnswerKey": {
+        "expectedFiles": [
+          "py/task_analytics/metrics.py",
+          "py/task_analytics/pipeline.py"
+        ],
+        "expectedSymbols": [
+          "calculate_project_metrics",
+          "STALE_DAY_THRESHOLD"
+        ],
+        "expectedFacts": [
+          {
+            "id": "warm062-p1-new-optional-param",
+            "text": "calculate_project_metrics(tasks, current_day, stale_day_threshold=STALE_DAY_THRESHOLD) now accepts an optional stale_day_threshold parameter.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-p1-default-preserved",
+            "text": "The parameter defaults to the existing STALE_DAY_THRESHOLD constant (10), so callers that omit it keep identical prior behavior.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-p1-predicate-uses-param",
+            "text": "The stale-task predicate now compares current_day minus task.updated_day against stale_day_threshold instead of the module constant directly.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-p1-pipeline-caller-unaffected",
+            "text": "pipeline.py build_report_from_rows calls calculate_project_metrics(tasks, current_day) without the new argument, so its behavior is unchanged.",
+            "weight": 1,
+            "required": true
+          }
+        ],
+        "minimumCorrectFacts": 3,
+        "notes": "v0.6.2 P1 scenario: public API change. Existing callers that omit the new argument preserve prior default behavior; the callable contract itself changes."
+      },
+      "mutation": {
+        "files": [
+          {
+            "path": "py/task_analytics/metrics.py",
+            "expectedPreSha256": "da2bfce46ec51016f87b3456a201e69a9d71721002e1781b44fdc59a96650e91",
+            "operations": [
+              {
+                "expectedPreimage": "def calculate_project_metrics(tasks, current_day):",
+                "replacement": "def calculate_project_metrics(tasks, current_day, stale_day_threshold=STALE_DAY_THRESHOLD):"
+              },
+              {
+                "expectedPreimage": "current_day - task.updated_day >= STALE_DAY_THRESHOLD",
+                "replacement": "current_day - task.updated_day >= stale_day_threshold"
+              }
+            ],
+            "expectedPostSha256": "c91af3c5549606c522a6cef8264d4da1a6febed0a58757ad5a96cafcb3ed58c9"
+          }
+        ]
+      },
+      "notes": "Two ordered exact replacements: the signature gains the optional parameter first, then the stale predicate is switched to use it."
+    },
+    {
+      "id": "I1",
+      "category": "import-graph-change",
+      "benchmarkProjectId": "task-analytics-large-mixed",
+      "baseCaseId": "warm-large-ts-analytics-snapshot",
+      "answerPolicy": "scenario",
+      "scenarioQuery": "analytics snapshot project completion rate stale tasks average story points task store after buildAnalyticsSnapshot task-listing dependency changed",
+      "scenarioAnswerKey": {
+        "expectedFiles": [
+          "ts/src/services/buildAnalyticsSnapshot.ts",
+          "ts/src/models/analyticsSnapshot.ts",
+          "ts/src/store/taskStore.ts"
+        ],
+        "expectedSymbols": [
+          "buildAnalyticsSnapshot",
+          "ProjectSnapshot",
+          "AnalyticsTaskStore"
+        ],
+        "expectedFacts": [
+          {
+            "id": "warm062-i1-no-longer-imports-helper",
+            "text": "buildAnalyticsSnapshot no longer imports or calls listTasksByProject.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-i1-inline-filtering",
+            "text": "buildAnalyticsSnapshot now obtains a target projects tasks directly with taskStore.list().filter((task) => task.projectId === project.id).",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-i1-output-unchanged",
+            "text": "The inline filtering is behaviorally identical to the removed helper, so totalTasks, completedTasks, openTasks, staleTasks, and averageStoryPoints per project are unchanged.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-i1-totals-aggregate-unchanged",
+            "text": "The snapshot totals still sum totalTasks, completedTasks, openTasks, and staleTasks across the per-project snapshots.",
+            "weight": 1,
+            "required": true
+          }
+        ],
+        "minimumCorrectFacts": 3,
+        "notes": "v0.6.2 I1 scenario: import-graph change. The intended snapshot output remains equivalent while the dependency edge on listTasksByProject disappears."
+      },
+      "mutation": {
+        "files": [
+          {
+            "path": "ts/src/services/buildAnalyticsSnapshot.ts",
+            "expectedPreSha256": "928af15041715ffa19ae3674e8d4f82e1c5f694cfa2bad1fea03db31bce6cedd",
+            "operations": [
+              {
+                "expectedPreimage": "import { listTasksByProject } from \"./listTasksByProject.js\";\r\n",
+                "replacement": ""
+              },
+              {
+                "expectedPreimage": "const tasks = listTasksByProject(taskStore, project.id);",
+                "replacement": "const tasks = taskStore.list().filter((task) => task.projectId === project.id);"
+              }
+            ],
+            "expectedPostSha256": "1ab0616f11107a2ffabfc1d7ee2b0614d9c9c58cd62fb461313a87c9754570b1"
+          }
+        ]
+      },
+      "notes": "Two ordered exact replacements: remove the now-unused import, then inline the equivalent filtering that listTasksByProject itself performed."
+    },
+    {
+      "id": "T1",
+      "category": "test-only-change",
+      "benchmarkProjectId": "task-analytics-large-mixed",
+      "baseCaseId": "warm-large-python-pipeline",
+      "answerPolicy": "scenario",
+      "scenarioQuery": "In py/tests/test_quality.py, what updated_day value does task-1 use in test_determine_quality_label, and what quality label does the test assert after calculating project metrics?",
+      "scenarioAnswerKey": {
+        "expectedFiles": [
+          "py/tests/test_quality.py"
+        ],
+        "expectedSymbols": [
+          "determine_quality_label",
+          "test_determine_quality_label"
+        ],
+        "expectedFacts": [
+          {
+            "id": "warm062-t1-task1-updated-day-nine",
+            "text": "task-1 row in test_determine_quality_label now uses updated_day 9 (previously 8).",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-t1-both-rows-completed",
+            "text": "Both fixture rows remain completed, so total_tasks stays 2 and completed_tasks stays 2.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-t1-completion-rate-unchanged",
+            "text": "The computed project completion_rate therefore remains 100.0.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-t1-no-stale-task-introduced",
+            "text": "current_day (15) minus the new updated_day (9) is 6, still below STALE_DAY_THRESHOLD (10), so stale_tasks remains 0.",
+            "weight": 1,
+            "required": true
+          },
+          {
+            "id": "warm062-t1-still-healthy",
+            "text": "The test still asserts determine_quality_label(metrics[0]) == \"healthy\"; production source and behavior are unchanged.",
+            "weight": 1,
+            "required": true
+          }
+        ],
+        "minimumCorrectFacts": 4,
+        "notes": "v0.6.2 T1 scenario: test-only change. Only the task-1 updated_day test input changes; production source in quality.py, metrics.py, and parser.py is unchanged, and the healthy-label assertion still passes."
+      },
+      "mutation": {
+        "files": [
+          {
+            "path": "py/tests/test_quality.py",
+            "expectedPreSha256": "b5610a2720012532baf6d18e114f375424008e0daadbc27291bfca2ce6316875",
+            "operations": [
+              {
+                "expectedPreimage": "{\"task_id\": \"task-1\", \"project_id\": \"alpha\", \"completed\": True, \"story_points\": 5, \"updated_day\": 8}",
+                "replacement": "{\"task_id\": \"task-1\", \"project_id\": \"alpha\", \"completed\": True, \"story_points\": 5, \"updated_day\": 9}"
+              }
+            ],
+            "expectedPostSha256": "eda7f39eed5b08d615b02a01b3b05655ed2aad06b65c39ad0ea09e9b93992ea5"
+          }
+        ]
+      },
+      "notes": "Test-only change: exactly one bounded test input literal changes. Production source and production behavior are unchanged; the query/answer targets the changed test evidence itself."
+    }
+  ]
+}

package/dist/src/cli/help.js

@@ -143,7 +143,10 @@ export function renderExperimentRunHelp() {
         "  --experiment <id>   Registered experiment plugin id (see \"my-dev-kit-lab experiment list\")",
         "",
         "Common options (all plugins):",
-        "  --target <path>                                    Local target project path (default: self)",
+        "  --target <path>                                    Local target project path (default: self); only for",
+        "                                                      plugins that support external targets. Self-only",
+        "                                                      plugins (for example incremental-change-staleness)",
+        "                                                      reject an explicit external --target.",
         "  --out <dir>                                         Output directory for experiment artifacts",
         "                                                      (default: beneath the workspace when installed)",
         "  --cases <path>                                      Path to the evaluation cases file",
@@ -153,10 +156,12 @@ export function renderExperimentRunHelp() {
         "  --case <ids>                                       Comma-separated case ids to run",
         "  --benchmark-project <ids>                          Comma-separated benchmark project ids to run",
         "",
+        "my-dev-kit command override (warm-index-reuse and incremental-change-staleness):",
+        "  --kit-command <command>                            my-dev-kit command used by the selected experiment plugin",
+        "                                                      (defaults: warm-index-reuse uses npx @dailephd/my-dev-kit@latest;",
+        "                                                      incremental-change-staleness uses npx @dailephd/my-dev-kit@1.12.5)",
+        "",
         "warm-index-reuse only:",
-        "  --kit-command <command>                            my-dev-kit command used to build one index per benchmark",
-        "                                                      project and retrieve per task",
-        "                                                      (default: npx @dailephd/my-dev-kit@latest)",
         "                                                      Legacy/default mode uses the deterministic fake agent only;",
         "                                                      agent options below are not accepted by this plugin.",
         "  --campaign <preset>                                 Runs the selected real-agent campaign preset: codex-full,",
@@ -173,11 +178,13 @@ export function renderExperimentRunHelp() {
         "                                                      replaced by a context-size estimate. A campaign run",
         "                                                      automatically produces the report JSON/text/HTML, the",
         "                                                      four warm-index SVG plots, a best-effort report PNG",
-        "                                                      screenshot, and a gallery manifest/index. A missing",
-        "                                                      Playwright/browser runtime makes screenshot capture skip",
-        "                                                      without invalidating the campaign; an unexpected",
-        "                                                      screenshot failure makes the command unsuccessful after",
-        "                                                      preserving the report, plots, and gallery. Legacy",
+        "                                                      screenshot, and a gallery manifest/index. The automatic",
+        "                                                      campaign report screenshot is best-effort presentation",
+        "                                                      evidence: skipped or failed capture is reported explicitly",
+        "                                                      (failed remains failed), no PNG is fabricated, and",
+        "                                                      screenshot status alone does not fail an otherwise",
+        "                                                      successful campaign command. Core experiment, report,",
+        "                                                      plot, and gallery failures remain fatal. Legacy",
         "                                                      non-campaign warm-index runs retain their existing",
         "                                                      report-only behavior.",
         "",

package/dist/src/commands/runExperimentDescribeCommand.js

@@ -147,9 +147,12 @@ function describeExamples(plugin) {
     const screenshotOption = hasConfigField(plugin, "agents") ? " --no-screenshot" : "";
     const examples = [
         `my-dev-kit-lab experiment describe --experiment ${plugin.metadata.id}`,
-        `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id}${agentOptions}`,
-        `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target "Z:\\Users\\newuser\\Projects\\my-dev-kit-v1"${agentOptions}${screenshotOption}`
+        `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id}${agentOptions}`
     ];
+    // Only advertise an explicit --target example for plugins that support external targets.
+    if (plugin.metadata.supportedTargets.includes("external-local")) {
+        examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --target "Z:\\Users\\newuser\\Projects\\my-dev-kit-v1"${agentOptions}${screenshotOption}`);
+    }
     // Only advertise campaign examples for plugins that declare the campaignPreset config field.
     if (hasConfigField(plugin, "campaignPreset")) {
         examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign codex-full --include-real-agents --out <dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign claude-full --include-real-agents --case warm-medium-complete-idempotent --out <dir>`);

package/dist/src/commands/runExperimentRunCommand.js

@@ -2,7 +2,7 @@ import path from "node:path";
 import { parseAgentCommandTemplate } from "../agents/index.js";
 import { parseAgentId } from "../agents/agentRegistry.js";
 import { readBenchmarkProjectProfiles, readEvaluationCases } from "../evaluation/index.js";
-import { contextStrategyComparisonPlugin, createDefaultExperimentPluginRegistry, getWarmIndexCampaignPreset, parseWarmIndexCampaignPresetId, resolveExperimentTarget, runExperiment, warmIndexReusePlugin } from "../experiments/index.js";
+import { contextStrategyComparisonPlugin, createDefaultExperimentPluginRegistry, getWarmIndexCampaignPreset, incrementalChangeStalenessPlugin, parseWarmIndexCampaignPresetId, resolveExperimentTarget, runExperiment, warmIndexReusePlugin } from "../experiments/index.js";
 import { buildDefaultExperimentOutputRoot } from "../experiments/outputPaths.js";
 import { parsePromptComplexityLevel, parsePromptStrategy } from "../prompts/index.js";
 import { writePluginExperimentReports } from "../report/index.js";
@@ -76,7 +76,6 @@ export async function runExperimentRunCommandFromArgs(argv, options = {}) {
         // Automatic presentation (report already produced above) applies only to warm-index-reuse
         // campaign runs; legacy fake-agent warm-index runs and context-strategy-comparison stay
         // report-only, unchanged.
-        let screenshotFailed = false;
         if (args.experimentId === warmIndexReusePlugin.metadata.id && args.config.campaignPreset) {
             const executionArtifactPath = readMetadataString(result.metadata?.executionArtifactPath);
             const campaignAgentId = readMetadataString(result.metadata?.campaignAgentId);
@@ -106,7 +105,6 @@ export async function runExperimentRunCommandFromArgs(argv, options = {}) {
             else {
                 outputLines.push("Screenshot: failed");
                 outputLines.push(presentation.screenshot.error ?? "Screenshot capture failed.");
-                screenshotFailed = true;
             }
             outputLines.push(`Gallery manifest: ${presentation.gallery.manifestPath}`);
             outputLines.push(`Gallery index: ${presentation.gallery.indexPath}`);
@@ -114,8 +112,6 @@ export async function runExperimentRunCommandFromArgs(argv, options = {}) {
         console.log(outputLines.join("\n"));
         if (result.status === "failed")
             return 1;
-        if (screenshotFailed)
-            return 1;
         return 0;
     }
     catch (error) {
@@ -220,8 +216,9 @@ export function parseRunExperimentArgs(argv) {
     if (!experimentId) {
         throw new Error("Usage: --experiment <id> [--target <path>] [--out <directory>]");
     }
-    if (kitCommand !== undefined && experimentId !== warmIndexReusePlugin.metadata.id) {
-        throw new Error(`--kit-command is only supported for --experiment ${warmIndexReusePlugin.metadata.id}.`);
+    const KIT_COMMAND_PLUGIN_IDS = [warmIndexReusePlugin.metadata.id, incrementalChangeStalenessPlugin.metadata.id];
+    if (kitCommand !== undefined && !KIT_COMMAND_PLUGIN_IDS.includes(experimentId)) {
+        throw new Error(`--kit-command is only supported for --experiment ${KIT_COMMAND_PLUGIN_IDS.join(" or ")}.`);
     }
     if (campaignPreset !== undefined) {
         if (experimentId !== warmIndexReusePlugin.metadata.id) {