@dailephd/my-dev-kit-lab 0.7.0 → 0.7.1

package/CHANGELOG.md

@@ -4,6 +4,15 @@ All notable changes to my-dev-kit-lab are documented here.
 
 ## [Unreleased]
 
+## [0.7.1] - 2026-10-02
+
+### Synthetic large-repository generator
+
+- Added deterministic synthetic TypeScript and Python repository generation under `src/evaluation/syntheticRepository/`: a strict `SyntheticRepositoryConfigV1`, seed- and identity-based pure planning, a versioned generator-owned PRNG, exact source/test/symbol/import/depth/repeated-pattern dimensions, and localized, cross-module, and broad-change task answer keys.
+- Added contained, idempotent materialization with a per-repository generation manifest (deterministic file hashes and metrics) and a read-only manifest verifier, plus an in-memory `EvaluationCase` adapter that reuses the existing raw baseline and scoring.
+- Added `--synthetic-config <path>` to `experiment run --experiment context-window-scaling` so the experiment can run over generated repositories placed beneath the run output; it is a command input selector (mutually exclusive with `--case`), not a plugin config field. Without it, the bundled v0.7.0 fixed corpus is used unchanged.
+- Extended help, `experiment list`/`describe` text, and installed-package acceptance to cover the synthetic input for both languages. The `context-window-scaling` treatments, budget semantics, correctness/success semantics, execution artifact, reports, and plots are unchanged.
+
 ## [0.7.0] - 2026-10-01
 
 ### Context-window scaling

package/README.md

@@ -6,7 +6,7 @@ my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev
 
 Whole-ecosystem workflows are centralized in [my-dev-kit/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md). Its [command-surface compatibility map](https://github.com/dailephd/my-dev-kit/blob/main/docs/ECOSYSTEM_DEVELOPMENT_WORKFLOWS.md#915-command-surface-compatibility-map) documents which Lab outputs can safely seed my-dev-kit investigation, Observer reference work, or Orchestrator evidence, and which look-alike artifacts are not direct inputs.
 
-The latest release is v0.7.0 (context-window scaling), and the published package and installed CLI are version 0.7.0. The previous release is v0.6.3 (four-treatment partial-refresh experiments). The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
+The latest release is v0.7.1 (synthetic large-repository generator), and the published package and installed CLI are version 0.7.1. The previous release is v0.7.0 (context-window scaling), and v0.7.2 (real/local external repository experiments) is planned after the current release. The v0.4.5 context-integrity evaluation remains intentionally frozen against the published `@dailephd/my-dev-kit@1.10.4` and `@dailephd/my-dev-kit-orchestrator@1.2.3` contracts; those versions are historical validation baselines, not a statement that they are the ecosystem's current releases. See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for current state.
 
 v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--version`, `security validate`, `audit`, the `experiment` family, `report render`, `plots generate`, `gallery build`, `demo final`, and the historical direct final-demo invocation form), a writable lab workspace model kept separate from the installed package and the inspected target, and a permanent packed-tarball installation/execution acceptance gate (`npm run verify:packed-package`). See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) for status detail and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the runtime path model. The "Installed CLI" section below documents the shipped command surface; the source-checkout `npm run` workflow in Quickstart remains available for contributors.
 
@@ -18,6 +18,7 @@ v0.4.6 adds a supported `my-dev-kit-lab` installed CLI router (`--help`, `--vers
 - **Record index freshness evidence (v0.6.0):** a warm-index run captures an index snapshot after its one index build — the exact indexed file set from the my-dev-kit index contract, each with SHA-256 content identity, size, and modified-time metadata, plus the my-dev-kit version and index-command evidence — and assesses freshness immediately before each task's warm retrieval as `fresh`, `stale`, `partially-stale`, or `unknown`. The evidence appears in `warm-index-execution.json` and in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: it covers only files the snapshot lists (not the whole repository), never changes retrieval, execution or provider status, and adds no CLI flag, metric, plot, gallery item, or reindex recommendation in v0.6.0. See [docs/METRICS.md](docs/METRICS.md#index-freshness-evidence-v060) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
 - **Compare no-refresh, partial-refresh, and full-refresh retrieval after a controlled change (v0.6.3, previous release):** the `incremental-change-staleness` experiment plugin runs six frozen controlled-change scenario families (unrelated-file, local-implementation, exported-symbol, public-API, import-graph, and test-only changes) against four disposable target copies — `stale-index` (no refresh), `changed-files-refresh` and `affected-neighborhood-refresh` (real partial refreshes performed by my-dev-kit 1.12.5 through `index --incremental --refresh-scope`, selected internally with no Lab-level flag), and `full-refresh` (a complete new post-change index, used as a comparison reference rather than a preferred treatment). Each partial treatment truthfully records whether upstream applied the requested partial scope or fell back to a full rebuild (`APPLIED_PARTIAL` or `FALLBACK_FULL`); a fallback is not counted as partial-refresh evidence. Three comparisons against `full-refresh` are persisted with retrieval, deterministic fake-agent correctness, and required-file evidence in `incremental-change-staleness-execution.json` (schema `my-dev-kit-lab-incremental-change-staleness-execution-v2`) and the existing `report.json`/`report.txt`/`report.html`; historical v0.6.2 (V1) artifacts stay readable. There is no winner, ranking, or safety claim, and the plugin is self-only (`--target` is not accepted). `--kit-command` also accepts `incremental-change-staleness` and now defaults to my-dev-kit 1.12.5. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md) and [docs/METRICS.md](docs/METRICS.md).
 - **Measure context-window scaling (v0.7.0, released):** the self-only `context-window-scaling` plugin compares `raw-full-file` and `my-dev-kit-guided` over 8k, 16k, 32k, 64k, or custom positive-integer estimated-token budgets. It records fit, utilization, deterministic fake-agent correctness/success, and omitted expected relevant files, with JSON/text/HTML reports and three SVG plots. `plots generate` reads `context-window-scaling-execution.json`. Token counts are a character-based estimate, not provider-tokenizer or billing data; fake-agent correctness is context-independent in the current harness. See [docs/WORKFLOWS.md](docs/WORKFLOWS.md), [docs/METRICS.md](docs/METRICS.md), and [docs/COMMANDS.md](docs/COMMANDS.md).
+- **Run context-window scaling over synthetic repositories (v0.7.1, current release):** `context-window-scaling` uses the bundled fixed corpus by default, or `--synthetic-config <path>` (a `SyntheticRepositoryConfigV1` JSON file) to generate disposable deterministic TypeScript/Python repositories beneath the run output. The option is a command input selector, not plugin scientific config; it is mutually exclusive with `--case`, and the scientific semantics, execution artifact, reports, and plots are unchanged. See [docs/COMMANDS.md](docs/COMMANDS.md) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md).
 - **Assess affected-neighborhood evidence (v0.6.1, released; retained in the current package):** using the baseline my-dev-kit graph of the same prepared warm index (loaded once per session), each task's confirmed modified or missing indexed files, and the baseline symbols those files contain, are mapped to graph nodes. The affected neighborhood is those seed nodes plus their direct one-hop neighbors (edges of any kind, either direction). The task's `expectedFiles` and file-scoped `expectedSymbols` are mapped to graph nodes and compared with that neighborhood, which yields six numeric warm-side metrics (`changedFileCount`, `changedSymbolCount`, `affectedNodeCount`, `affectedEdgeCount`, `taskOverlapCount`, `taskOverlapPercent`) plus categorical evidence: a `related`, `unrelated`, or `unknown` relationship and a `recommended`, `not-indicated`, or `unknown` `reindexRecommendation`. Incomplete evidence is reported as `unknown`, never as unrelated. The assessment is recorded additively in `warm-index-execution.json` and shown in the existing `report.json`/`report.txt`/`report.html` warm-index reports. It is observational only: retrieval, execution status, and provider status are unchanged, nothing is reindexed automatically, and no CLI flag, experiment plugin, plot, or gallery item was added. See [docs/METRICS.md](docs/METRICS.md#affected-neighborhood-metrics-v061) and [docs/WORKFLOWS.md](docs/WORKFLOWS.md#warm-index-reuse-experiment).
 - **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
 - **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
@@ -237,7 +238,7 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Claude and Codex token totals are used only when the CLI/run exposes them; missing provider token evidence stays explicitly unavailable (never zero, and never substituted with an estimated context-token count)
 - Codex and Claude campaign runs can produce timeouts, invalid-output, agent-unavailable, or agent-limit-reached outcomes; these are structured provider states, not warm-index infrastructure failures
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
-- The generic experiment-plugin framework has four registered plugins: `context-strategy-comparison`, `warm-index-reuse`, `incremental-change-staleness`, and `context-window-scaling`. The v0.7.0 context-window-scaling plugin is released; retrieval precision/recall and agent-success plugins remain planned.
+- The generic experiment-plugin framework has four registered plugins: `context-strategy-comparison`, `warm-index-reuse`, `incremental-change-staleness`, and `context-window-scaling`. The v0.7.0 context-window-scaling plugin is released; v0.7.1 lets it also run over caller-supplied synthetic repositories; external/local repository experiments (v0.7.2), retrieval precision/recall, and agent-success plugins remain planned.
 - `warm-index-reuse` without `--campaign` still uses the deterministic fake agent only. With `--campaign`, exactly one real provider (Codex or Claude) runs per campaign preset — there is no multi-agent matrix, retry, provider switching, or scheduler. The v0.5.1 expanded corpus is deterministic fake-agent benchmark evidence when run without `--campaign`. No warm-index mode calculates a token-savings percentage, break-even task, winner, or ranking, and component-duration sums are not wall-clock latency
 - The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Context-window scaling uses estimated tokens (`ceil(characters / 4)`) and experiment budgets, not actual provider tokenizer or model-window limits. Its deterministic fake-agent correctness check does not measure semantic sensitivity to the supplied context.

package/benchmarks/projects/README.md

@@ -2,7 +2,7 @@
 
 This directory contains deterministic benchmark projects used by my-dev-kit-lab.
 
-The current suite mixes baseline Todo fixtures with more complex workflow and analytics projects:
+The shared-profile suite mixes baseline Todo fixtures with more complex workflow and analytics projects:
 
 - `todo-ts`
 - `todo-js`
@@ -11,6 +11,8 @@ The current suite mixes baseline Todo fixtures with more complex workflow and an
 - `task-workflow-medium-ts`
 - `task-analytics-large-mixed`
 
+The v0.7.0 release also includes `context-window-scaling-fixed-ts` as an experiment-specific, static scaling corpus. It is intentionally not part of the shared `benchmark-project-profiles.json` contract: the `context-window-scaling` plugin derives its project profile in memory. The fixed corpus is frozen v0.7.0 evidence and is not a generator.
+
 The Todo fixtures stay intentionally small so validation and smoke experiments remain cheap. The newer medium and large projects are intentionally more connected so raw full-file context and guided retrieval diverge more meaningfully.
 
 Benchmark project profiles live in `benchmarks/contracts/benchmark-project-profiles.json`.

package/dist/src/cli/help.js

@@ -167,8 +167,14 @@ export function renderExperimentRunHelp() {
         "                                                      64k (8192/16384/32768/65536), or positive integers such as",
         "                                                      12000 (default: 8k,16k,32k,64k). Duplicates after",
         "                                                      normalization are rejected. Only --out, --case,",
-        "                                                      --context-budgets, and --kit-command are accepted; --case",
-        "                                                      selects from the bundled four-case catalog. Self target only.",
+        "                                                      --synthetic-config, --context-budgets, and --kit-command are",
+        "                                                      accepted; --case selects from the bundled four-case catalog.",
+        "                                                      Self target only.",
+        "  --synthetic-config <path>                           Path to a SyntheticRepositoryConfigV1 JSON file; relative paths",
+        "                                                      resolve against the invocation directory. Generated",
+        "                                                      repositories are written beneath the selected experiment",
+        "                                                      output directory (synthetic-repositories/). Mutually exclusive",
+        "                                                      with --case. Without it, the bundled four-case catalog is used.",
         "                                                      Generate plots with \"plots generate --experiment <run dir>\".",
         "",
         "warm-index-reuse only:",

package/dist/src/commands/runExperimentDescribeCommand.js

@@ -133,6 +133,9 @@ function describeTargetBehavior(plugin) {
     if (supportsExternal) {
         return "Runs against the current lab repository when --target is omitted, or against an explicit local target project when --target <path> is provided. Outputs stay under lab-controlled output directories by default.";
     }
+    if (plugin.metadata.id === "context-window-scaling") {
+        return "Self-targeted. Uses the bundled fixed scaling corpus by default, or caller-supplied deterministic synthetic repositories when --synthetic-config <path> (a SyntheticRepositoryConfigV1 JSON file, mutually exclusive with --case) is provided; generated repositories are written beneath the experiment output directory.";
+    }
     return "Runs against the current lab repository.";
 }
 function describeExpectedReports(plugin) {
@@ -157,6 +160,9 @@ function describeExamples(plugin) {
     if (hasConfigField(plugin, "campaignPreset")) {
         examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign codex-full --include-real-agents --out <dir>`, `my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --campaign claude-full --include-real-agents --case warm-medium-complete-idempotent --out <dir>`);
     }
+    if (plugin.metadata.id === "context-window-scaling") {
+        examples.push(`my-dev-kit-lab experiment run --experiment ${plugin.metadata.id} --synthetic-config <path-to-config.json> --out <run-dir>`);
+    }
     return examples;
 }
 function hasConfigField(plugin, name) {

package/dist/src/commands/runExperimentRunCommand.js

@@ -1,8 +1,10 @@
+import { readFile } from "node:fs/promises";
 import path from "node:path";
 import { parseAgentCommandTemplate } from "../agents/index.js";
 import { parseAgentId } from "../agents/agentRegistry.js";
 import { readBenchmarkProjectProfiles, readEvaluationCases } from "../evaluation/index.js";
 import { contextStrategyComparisonPlugin, contextWindowScalingPlugin, createDefaultExperimentPluginRegistry, getWarmIndexCampaignPreset, incrementalChangeStalenessPlugin, parseContextBudgetsCliValue, parseWarmIndexCampaignPresetId, resolveExperimentTarget, runExperiment, warmIndexReusePlugin } from "../experiments/index.js";
+import { prepareSyntheticContextWindowScalingInputs } from "../experiments/plugins/contextWindowScaling/index.js";
 import { buildDefaultExperimentOutputRoot } from "../experiments/outputPaths.js";
 import { parsePromptComplexityLevel, parsePromptStrategy } from "../prompts/index.js";
 import { writePluginExperimentReports } from "../report/index.js";
@@ -36,22 +38,32 @@ const DEFAULT_PROJECT_PROFILES_RESOURCE = "benchmarks/contracts/benchmark-projec
 // catalog's package-relative targetRoot, so both resolve from the package root, never the cwd.
 const CONTEXT_WINDOW_SCALING_CASES_RESOURCE = "benchmarks/contracts/context-window-scaling-cases.json";
 // Options the context-window-scaling plugin accepts; every other flag is rejected, not ignored.
-const CONTEXT_WINDOW_SCALING_ALLOWED_FLAGS = ["--experiment", "--out", "--case", "--context-budgets", "--kit-command"];
+const CONTEXT_WINDOW_SCALING_ALLOWED_FLAGS = ["--experiment", "--out", "--case", "--synthetic-config", "--context-budgets", "--kit-command"];
 export async function runExperimentRunCommandFromArgs(argv, options = {}) {
     try {
         const args = parseRunExperimentArgs(argv);
         const context = options.context ?? createLabExecutionContext();
         const toolRoot = context.packageRoot;
         const registry = createDefaultExperimentPluginRegistry();
-        const inputs = await loadPluginInputs(args, toolRoot, context);
         const runId = generateExperimentRunId(args.experimentId);
-        const outputRoot = resolveOutputRoot(args, {
+        let outputRoot = resolveOutputRoot(args, {
             toolRoot,
             invocationCwd: context.invocationCwd,
             installedDefaultOutputRoot: options.installedDefaultOutputRoot,
             experimentId: args.experimentId,
             runId
         });
+        if (args.syntheticConfigPath !== undefined && outputRoot === undefined) {
+            // Synthetic generation needs the concrete root before the runner runs; use the exact default the
+            // runner itself would compute for this self-targeted invocation.
+            outputRoot = buildDefaultExperimentOutputRoot({
+                toolRoot,
+                pluginId: args.experimentId,
+                target: resolveExperimentTarget(args.targetPath, toolRoot),
+                runId
+            });
+        }
+        const inputs = await loadPluginInputs(args, toolRoot, context, outputRoot);
         const result = await runExperiment({
             pluginId: args.experimentId,
             registry,
@@ -148,6 +160,7 @@ export function parseRunExperimentArgs(argv) {
     let kitCommand;
     let campaignPreset;
     let contextBudgets;
+    let syntheticConfigPath;
     const seenFlags = [];
     const commandTemplates = {};
     for (let index = 0; index < argv.length; index += 1) {
@@ -169,6 +182,12 @@ export function parseRunExperimentArgs(argv) {
         else if (arg === "--project-profiles") {
             projectProfilesPath = readRequiredValue(argv, ++index, "--project-profiles");
         }
+        else if (arg === "--synthetic-config") {
+            if (syntheticConfigPath !== undefined) {
+                throw new Error("--synthetic-config may be supplied only once.");
+            }
+            syntheticConfigPath = readRequiredValue(argv, ++index, "--synthetic-config");
+        }
         else if (arg === "--case") {
             caseIds.push(...splitList(readRequiredValue(argv, ++index, "--case")));
         }
@@ -236,6 +255,9 @@ export function parseRunExperimentArgs(argv) {
     if (contextBudgets !== undefined && experimentId !== contextWindowScalingPlugin.metadata.id) {
         throw new Error(`--context-budgets is only supported for --experiment ${contextWindowScalingPlugin.metadata.id}.`);
     }
+    if (syntheticConfigPath !== undefined && experimentId !== contextWindowScalingPlugin.metadata.id) {
+        throw new Error(`--synthetic-config is only supported for --experiment ${contextWindowScalingPlugin.metadata.id}.`);
+    }
     if (experimentId === contextWindowScalingPlugin.metadata.id) {
         const unsupported = [...new Set(seenFlags.filter((flag) => !CONTEXT_WINDOW_SCALING_ALLOWED_FLAGS.includes(flag)))];
         if (unsupported.length > 0) {
@@ -244,6 +266,9 @@ export function parseRunExperimentArgs(argv) {
         if (seenFlags.includes("--case") && caseIds.length === 0) {
             throw new Error("--case must list at least one case id.");
         }
+        if (syntheticConfigPath !== undefined && seenFlags.includes("--case")) {
+            throw new Error("--case and --synthetic-config are mutually exclusive; the synthetic config owns the generated case set.");
+        }
     }
     if (kitCommand !== undefined && !KIT_COMMAND_PLUGIN_IDS.includes(experimentId)) {
         throw new Error(`--kit-command is only supported for --experiment ${KIT_COMMAND_PLUGIN_IDS.join(" or ")}.`);
@@ -266,6 +291,7 @@ export function parseRunExperimentArgs(argv) {
         experimentId,
         targetPath,
         outDir,
+        syntheticConfigPath,
         config: withoutUndefined({
             casesPath,
             projectProfilesPath,
@@ -308,7 +334,7 @@ function generateExperimentRunId(pluginId) {
     const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
     return `${pluginId}-${timestamp}`;
 }
-async function loadPluginInputs(args, toolRoot, context) {
+async function loadPluginInputs(args, toolRoot, context, outputRoot) {
     if (args.experimentId === contextStrategyComparisonPlugin.metadata.id) {
         const validation = contextStrategyComparisonPlugin.validateConfig(args.config);
         if (!validation.valid || !validation.config) {
@@ -327,7 +353,7 @@ async function loadPluginInputs(args, toolRoot, context) {
         return loadCasesAndProjectProfiles(args, toolRoot, context);
     }
     if (args.experimentId === contextWindowScalingPlugin.metadata.id) {
-        return loadContextWindowScalingInputs(args, toolRoot, context);
+        return loadContextWindowScalingInputs(args, toolRoot, context, outputRoot);
     }
     return undefined;
 }
@@ -339,11 +365,20 @@ function pluginConfigFor(args) {
     const { caseIds: _caseIds, ...config } = args.config;
     return config;
 }
-async function loadContextWindowScalingInputs(args, toolRoot, context) {
+async function loadContextWindowScalingInputs(args, toolRoot, context, outputRoot) {
     const validation = contextWindowScalingPlugin.validateConfig(pluginConfigFor(args));
     if (!validation.valid || !validation.config) {
         throw new Error(`Invalid context window scaling config: ${validation.errors.join("; ")}`);
     }
+    if (args.syntheticConfigPath !== undefined) {
+        if (outputRoot === undefined) {
+            throw new Error("Internal error: synthetic generation requires a resolved experiment output root.");
+        }
+        const configPath = path.resolve(context.invocationCwd, args.syntheticConfigPath);
+        const syntheticRepositoryConfig = await readSyntheticConfigJson(configPath, args.syntheticConfigPath);
+        const prepared = prepareSyntheticContextWindowScalingInputs({ syntheticRepositoryConfig, outputRoot });
+        return { cases: prepared.cases, projectProfiles: prepared.projectProfiles, env: process.env };
+    }
     const catalog = await readEvaluationCases(resolvePackageResource(context, CONTEXT_WINDOW_SCALING_CASES_RESOURCE), toolRoot);
     const requested = args.config.caseIds;
     if (requested === undefined)
@@ -359,6 +394,22 @@ async function loadContextWindowScalingInputs(args, toolRoot, context) {
     }
     return { cases: catalog.filter((evaluationCase) => requested.includes(evaluationCase.id)), env: process.env };
 }
+// The command only reads and parses the caller-supplied file; schema validation belongs to the synthetic planner.
+async function readSyntheticConfigJson(resolvedPath, displayPath) {
+    let text;
+    try {
+        text = await readFile(resolvedPath, "utf8");
+    }
+    catch (error) {
+        throw new Error(`Cannot read --synthetic-config ${displayPath}: ${error instanceof Error ? error.message : String(error)}`);
+    }
+    try {
+        return JSON.parse(text);
+    }
+    catch (error) {
+        throw new Error(`--synthetic-config ${displayPath} is not valid JSON: ${error instanceof Error ? error.message : String(error)}`);
+    }
+}
 // Shared by both registered plugins: explicit paths resolve against toolRoot, defaults come from
 // bundled package resources so they do not depend on the invocation cwd.
 async function loadCasesAndProjectProfiles(args, toolRoot, context) {

package/dist/src/evaluation/syntheticRepository/config.js

@@ -0,0 +1,304 @@
+import { TASK_LOCALITIES } from "../types.js";
+import { SYNTHETIC_REPOSITORY_CONFIG_SCHEMA_VERSION, SYNTHETIC_REPOSITORY_LANGUAGES, SyntheticRepositoryConfigError, } from "./types.js";
+export const SYNTHETIC_REPOSITORY_LIMITS = {
+    maxCases: 64,
+    caseIdMaxLength: 64,
+    seedMaxLength: 256,
+    sourceFileCount: { min: 1, max: 10000 },
+    moduleDepth: { min: 1, max: 64 },
+    internalImportCount: { min: 0, max: 100000 },
+    symbolCount: { min: 1, max: 100000 },
+    testFileCount: { min: 0, max: 10000 },
+    repeatedPatternCount: { min: 0, max: 100000 },
+    broadChangeMinSourceFiles: 4,
+    crossModuleMinSourceFiles: 2,
+};
+/** Fixed field order of one case; also the canonical serialization order. */
+export const SYNTHETIC_REPOSITORY_CASE_FIELDS = [
+    "id",
+    "language",
+    "seed",
+    "sourceFileCount",
+    "moduleDepth",
+    "internalImportCount",
+    "symbolCount",
+    "testFileCount",
+    "taskLocality",
+    "repeatedPatternCount",
+];
+const COUNT_FIELDS = [
+    "sourceFileCount",
+    "moduleDepth",
+    "internalImportCount",
+    "symbolCount",
+    "testFileCount",
+    "repeatedPatternCount",
+];
+const CASE_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]*$/;
+const WINDOWS_DEVICE_NAME_PATTERN = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i;
+/** Locale-independent ordering by UTF-16 code unit. */
+export function compareCodeUnits(left, right) {
+    if (left < right)
+        return -1;
+    if (left > right)
+        return 1;
+    return 0;
+}
+/**
+ * Maximum number of unique directed acyclic import edges among `sourceFileCount` modules whose longest
+ * dependency chain has exactly `moduleDepth` modules: the edge count of the balanced complete
+ * multipartite arrangement, (N^2 - sum of squared layer sizes) / 2.
+ */
+export function maxImportEdges(sourceFileCount, moduleDepth) {
+    const base = Math.floor(sourceFileCount / moduleDepth);
+    const larger = sourceFileCount % moduleDepth;
+    const sumOfSquares = larger * (base + 1) * (base + 1) + (moduleDepth - larger) * base * base;
+    return (sourceFileCount * sourceFileCount - sumOfSquares) / 2;
+}
+/** Returns the case in canonical field order. Assumes the spec is valid. */
+export function orderCaseSpec(spec) {
+    return {
+        id: spec.id,
+        language: spec.language,
+        seed: spec.seed,
+        sourceFileCount: spec.sourceFileCount,
+        moduleDepth: spec.moduleDepth,
+        internalImportCount: spec.internalImportCount,
+        symbolCount: spec.symbolCount,
+        testFileCount: spec.testFileCount,
+        taskLocality: spec.taskLocality,
+        repeatedPatternCount: spec.repeatedPatternCount,
+    };
+}
+function isPlainObject(value) {
+    if (typeof value !== "object" || value === null || Array.isArray(value))
+        return false;
+    const prototype = Object.getPrototypeOf(value);
+    return prototype === Object.prototype || prototype === null;
+}
+function hasOwn(value, key) {
+    return Object.prototype.hasOwnProperty.call(value, key);
+}
+function describeValue(value) {
+    if (typeof value === "string") {
+        return JSON.stringify(value.length > 40 ? `${value.slice(0, 40)}...` : value);
+    }
+    if (typeof value === "number" || typeof value === "boolean")
+        return String(value);
+    if (value === null)
+        return "null";
+    if (Array.isArray(value))
+        return "array";
+    return typeof value;
+}
+function hasLoneSurrogate(value) {
+    for (let index = 0; index < value.length; index += 1) {
+        const unit = value.charCodeAt(index);
+        if (unit >= 0xd800 && unit <= 0xdbff) {
+            const next = value.charCodeAt(index + 1);
+            if (next >= 0xdc00 && next <= 0xdfff) {
+                index += 1;
+                continue;
+            }
+            return true;
+        }
+        if (unit >= 0xdc00 && unit <= 0xdfff)
+            return true;
+    }
+    return false;
+}
+function validateCaseId(value, label, errors) {
+    if (typeof value !== "string") {
+        errors.push(`${label}.id: must be a string (received ${describeValue(value)}).`);
+        return false;
+    }
+    const before = errors.length;
+    if (value.length < 1 || value.length > SYNTHETIC_REPOSITORY_LIMITS.caseIdMaxLength) {
+        errors.push(`${label}.id: length must be between 1 and ${SYNTHETIC_REPOSITORY_LIMITS.caseIdMaxLength}.`);
+    }
+    else if (!CASE_ID_PATTERN.test(value)) {
+        errors.push(`${label}.id: must begin with an ASCII letter or digit and contain only ASCII letters, digits, ".", "_" and "-" (received ${describeValue(value)}).`);
+    }
+    else {
+        if (value.endsWith(".")) {
+            errors.push(`${label}.id: must not end with ".".`);
+        }
+        if (WINDOWS_DEVICE_NAME_PATTERN.test(value.split(".")[0])) {
+            errors.push(`${label}.id: must not use a reserved device name (received ${describeValue(value)}).`);
+        }
+    }
+    return errors.length === before;
+}
+function validateSeed(value, label, errors) {
+    if (typeof value !== "string") {
+        errors.push(`${label}.seed: must be a string (received ${describeValue(value)}).`);
+        return false;
+    }
+    const before = errors.length;
+    if (!/\S/.test(value)) {
+        errors.push(`${label}.seed: must contain at least one non-whitespace character.`);
+    }
+    if (value.length > SYNTHETIC_REPOSITORY_LIMITS.seedMaxLength) {
+        errors.push(`${label}.seed: must be at most ${SYNTHETIC_REPOSITORY_LIMITS.seedMaxLength} UTF-16 code units.`);
+    }
+    if (hasLoneSurrogate(value)) {
+        errors.push(`${label}.seed: must be well-formed Unicode (lone surrogate found).`);
+    }
+    return errors.length === before;
+}
+function validateCount(value, field, label, errors) {
+    const { min, max } = SYNTHETIC_REPOSITORY_LIMITS[field];
+    if (typeof value !== "number" || !Number.isSafeInteger(value) || value < min || value > max) {
+        errors.push(`${label}.${field}: must be an integer between ${min} and ${max} (received ${describeValue(value)}).`);
+        return false;
+    }
+    return true;
+}
+function validateCase(value, label, errors) {
+    if (!isPlainObject(value)) {
+        errors.push(`${label}: must be an object (received ${describeValue(value)}).`);
+        return undefined;
+    }
+    const before = errors.length;
+    const allowed = new Set(SYNTHETIC_REPOSITORY_CASE_FIELDS);
+    for (const key of Object.keys(value).sort(compareCodeUnits)) {
+        if (!allowed.has(key))
+            errors.push(`${label}: unknown field ${JSON.stringify(key)}.`);
+    }
+    for (const field of SYNTHETIC_REPOSITORY_CASE_FIELDS) {
+        if (!hasOwn(value, field))
+            errors.push(`${label}: missing required field ${field}.`);
+    }
+    const idOk = hasOwn(value, "id") && validateCaseId(value.id, label, errors);
+    const languageOk = hasOwn(value, "language") &&
+        SYNTHETIC_REPOSITORY_LANGUAGES.includes(value.language);
+    if (hasOwn(value, "language") && !languageOk) {
+        errors.push(`${label}.language: must be one of ${SYNTHETIC_REPOSITORY_LANGUAGES.join(", ")} (received ${describeValue(value.language)}).`);
+    }
+    const seedOk = hasOwn(value, "seed") && validateSeed(value.seed, label, errors);
+    const countOk = new Map();
+    for (const field of COUNT_FIELDS) {
+        countOk.set(field, hasOwn(value, field) && validateCount(value[field], field, label, errors));
+    }
+    const localityOk = hasOwn(value, "taskLocality") && TASK_LOCALITIES.includes(value.taskLocality);
+    if (hasOwn(value, "taskLocality") && !localityOk) {
+        errors.push(`${label}.taskLocality: must be one of ${TASK_LOCALITIES.join(", ")} (received ${describeValue(value.taskLocality)}).`);
+    }
+    const sourceFileCount = value.sourceFileCount;
+    const moduleDepth = value.moduleDepth;
+    const internalImportCount = value.internalImportCount;
+    const symbolCount = value.symbolCount;
+    const taskLocality = value.taskLocality;
+    const sourceOk = countOk.get("sourceFileCount") === true;
+    const depthOk = countOk.get("moduleDepth") === true;
+    const importsOk = countOk.get("internalImportCount") === true;
+    const symbolsOk = countOk.get("symbolCount") === true;
+    if (sourceOk && depthOk) {
+        if (moduleDepth > sourceFileCount) {
+            errors.push(`${label}.moduleDepth: must not exceed sourceFileCount (${moduleDepth} > ${sourceFileCount}).`);
+        }
+        else if (importsOk) {
+            if (moduleDepth > 1 && internalImportCount < moduleDepth - 1) {
+                errors.push(`${label}.internalImportCount: must be at least moduleDepth - 1 (${moduleDepth - 1}) to realize moduleDepth ${moduleDepth}.`);
+            }
+            const capacity = maxImportEdges(sourceFileCount, moduleDepth);
+            if (internalImportCount > capacity) {
+                errors.push(`${label}.internalImportCount: exceeds the ${capacity} unique acyclic edges possible for sourceFileCount ${sourceFileCount} with exact moduleDepth ${moduleDepth}.`);
+            }
+        }
+    }
+    if (sourceOk && symbolsOk && symbolCount < sourceFileCount) {
+        errors.push(`${label}.symbolCount: must be at least sourceFileCount (${symbolCount} < ${sourceFileCount}).`);
+    }
+    if (sourceOk && localityOk) {
+        if (taskLocality === "cross-module") {
+            if (sourceFileCount < SYNTHETIC_REPOSITORY_LIMITS.crossModuleMinSourceFiles) {
+                errors.push(`${label}.taskLocality: cross-module requires sourceFileCount >= 2.`);
+            }
+            if (importsOk && internalImportCount < 1) {
+                errors.push(`${label}.taskLocality: cross-module requires internalImportCount >= 1.`);
+            }
+        }
+        else if (taskLocality === "broad-change" && sourceFileCount < SYNTHETIC_REPOSITORY_LIMITS.broadChangeMinSourceFiles) {
+            errors.push(`${label}.taskLocality: broad-change requires sourceFileCount >= 4.`);
+        }
+    }
+    if (errors.length !== before || !idOk || !languageOk || !seedOk || !localityOk)
+        return undefined;
+    return orderCaseSpec(value);
+}
+/** Validates one case spec and returns it in canonical field order. Never throws, never repairs. */
+export function validateSyntheticRepositoryCaseSpec(input) {
+    const errors = [];
+    const spec = validateCase(input, "case", errors);
+    if (errors.length > 0 || spec === undefined)
+        return { ok: false, errors };
+    return { ok: true, errors: [], spec };
+}
+/**
+ * Strictly validates an unknown value as SyntheticRepositoryConfigV1. On success the returned config is
+ * canonical: cases sorted ascending by id (code-unit order), fixed field order, values untouched.
+ */
+export function validateSyntheticRepositoryConfig(input) {
+    const errors = [];
+    if (!isPlainObject(input)) {
+        return { ok: false, errors: [`config: must be an object (received ${describeValue(input)}).`] };
+    }
+    for (const key of Object.keys(input).sort(compareCodeUnits)) {
+        if (key !== "schemaVersion" && key !== "cases")
+            errors.push(`config: unknown field ${JSON.stringify(key)}.`);
+    }
+    if (!hasOwn(input, "schemaVersion")) {
+        errors.push("config: missing required field schemaVersion.");
+    }
+    else if (input.schemaVersion !== SYNTHETIC_REPOSITORY_CONFIG_SCHEMA_VERSION) {
+        errors.push(`config.schemaVersion: must be ${JSON.stringify(SYNTHETIC_REPOSITORY_CONFIG_SCHEMA_VERSION)} (received ${describeValue(input.schemaVersion)}).`);
+    }
+    if (!hasOwn(input, "cases")) {
+        errors.push("config: missing required field cases.");
+        return { ok: false, errors };
+    }
+    const cases = input.cases;
+    if (!Array.isArray(cases)) {
+        errors.push(`config.cases: must be an array (received ${describeValue(cases)}).`);
+        return { ok: false, errors };
+    }
+    if (cases.length < 1 || cases.length > SYNTHETIC_REPOSITORY_LIMITS.maxCases) {
+        errors.push(`config.cases: must contain between 1 and ${SYNTHETIC_REPOSITORY_LIMITS.maxCases} cases (received ${cases.length}).`);
+        return { ok: false, errors };
+    }
+    const valid = [];
+    const firstIndexById = new Map();
+    cases.forEach((candidate, index) => {
+        const label = `cases[${index}]`;
+        const spec = validateCase(candidate, label, errors);
+        if (spec)
+            valid.push(spec);
+        const id = isPlainObject(candidate) ? candidate.id : undefined;
+        if (typeof id === "string") {
+            const key = id.toLowerCase();
+            const first = firstIndexById.get(key);
+            if (first === undefined) {
+                firstIndexById.set(key, index);
+            }
+            else {
+                errors.push(`${label}.id: duplicates cases[${first}].id (ids are compared without regard to letter case).`);
+            }
+        }
+    });
+    if (errors.length > 0)
+        return { ok: false, errors };
+    const sorted = [...valid].sort((left, right) => compareCodeUnits(left.id, right.id));
+    return {
+        ok: true,
+        errors: [],
+        config: { schemaVersion: SYNTHETIC_REPOSITORY_CONFIG_SCHEMA_VERSION, cases: sorted },
+    };
+}
+/** Validates then returns the canonical configuration; throws SyntheticRepositoryConfigError when invalid. */
+export function normalizeSyntheticRepositoryConfig(input) {
+    const result = validateSyntheticRepositoryConfig(input);
+    if (!result.ok)
+        throw new SyntheticRepositoryConfigError(result.errors);
+    return result.config;
+}

package/dist/src/evaluation/syntheticRepository/evaluationCase.js

@@ -0,0 +1,43 @@
+import { describeSyntheticTask } from "./layout.js";
+/** The plan's answer key expressed in the existing benchmark answer-key vocabulary (no second correctness contract). */
+export function buildSyntheticAnswerKey(plan) {
+    const { factTextById } = describeSyntheticTask(plan);
+    return {
+        expectedFiles: [...plan.answerKey.expectedFiles],
+        expectedSymbols: [...plan.answerKey.expectedSymbols],
+        expectedFacts: plan.answerKey.facts.map((fact) => ({
+            id: fact.factId,
+            text: factTextById.get(fact.factId),
+            weight: fact.weight,
+            required: fact.required,
+        })),
+        expectedContextTargets: plan.answerKey.expectedContextTargets.map((target) => ({
+            file: target.file,
+            symbols: [...target.symbols],
+            required: target.required,
+        })),
+        minimumCorrectFacts: plan.answerKey.minimumCorrectFacts,
+    };
+}
+/**
+ * Builds the existing EvaluationCase directly in memory (never through readEvaluationCases, which owns
+ * repository-contained JSON cases). `targetRoot` is the stable logical root from the manifest; only
+ * `absoluteTargetRoot` carries the physical generated repository directory.
+ */
+export function buildSyntheticEvaluationCase(plan, manifest, repositoryRoot) {
+    const text = describeSyntheticTask(plan);
+    return {
+        id: plan.task.taskId,
+        title: text.title,
+        benchmarkProject: plan.logicalProjectId,
+        targetRoot: manifest.logicalTargetRoot,
+        sourceRoots: [...manifest.sourceRoots],
+        query: text.query,
+        expectedFiles: [...plan.answerKey.expectedFiles],
+        expectedSymbols: [...plan.answerKey.expectedSymbols],
+        rawIncludeGlobs: [...manifest.rawIncludeGlobs],
+        answerKey: buildSyntheticAnswerKey(plan),
+        taskLocality: plan.task.locality,
+        absoluteTargetRoot: repositoryRoot,
+    };
+}