@dailephd/my-dev-kit-lab 0.4.2 → 0.4.3

package/README.md

@@ -1,66 +1,19 @@
 # my-dev-kit-lab
 
-> Current release state: v0.4.1 is the latest published baseline. v0.4.0 delivered the Android validation MVP; v0.4.1 delivered advanced Android security. v0.4.2 is implemented and release-prepared (package metadata `0.4.2`) but not yet published. Manual pentest is post-v1 / version TBD.
+my-dev-kit-lab is the experiment, audit, and evidence companion for [my-dev-kit](https://www.npmjs.com/package/@dailephd/my-dev-kit). It helps users compare repository-context strategies, audit project health, validate CLI/package and Android security boundaries, and turn each run into reviewable reports and visual artifacts.
 
-my-dev-kit-lab is the experiment, evidence, reporting, security-validation, and audit companion for my-dev-kit. It runs reproducible experiments that test whether my-dev-kit's graph-guided retrieval helps coding-agent workflows, collects metrics, renders reports, generates plots, captures screenshots, builds gallery outputs, performs automated CLI/package security validation, and runs the generic audit framework.
+my-dev-kit provides local repository indexing and graph-guided retrieval. my-dev-kit-lab supplies the controlled benchmarks, agent adapters, metrics, security checks, and reports needed to evaluate when that retrieval is useful. Results are evidence for a specific target and configuration; they do not guarantee token savings or security.
 
-The generic experiment-plugin framework was introduced in `v0.2.0`. Its first plugin, `context-strategy-comparison`, preserves the raw-full-file vs my-dev-kit-guided comparison through the plugin runner. Later releases added the generic audit framework, language-aware code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, a security-validation audit adapter, and Android validation. See [CHANGELOG.md](CHANGELOG.md) for the complete release history.
-
-my-dev-kit is most useful when the repository is larger than the task. It helps coding agents work with large codebases through reusable structural indexing, graph-guided retrieval, targeted source slices, and auditable context selection. Results are scoped evidence; the project does not claim that my-dev-kit always saves tokens.
-
-**my-dev-kit** is the repo indexing and graph-guided retrieval engine.
-**my-dev-kit-lab** is the separate lab layer that feeds it benchmark inputs and records evaluation outputs.
-
----
+The latest published release is v0.4.3 (stage-specific bounded-context and workflow-instruction evaluation). See [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md).
 
 ## Current capabilities
 
-- Benchmark projects at small, medium, and large complexity levels
-- Project complexity metrics and benchmark case metadata with answer keys
-- Prompt variant generation at `short`, `medium`, `long`, and `multi-step` complexity levels
-- Fake-agent adapter for deterministic smoke and demo validation
-- Codex and Claude adapters for real-agent campaigns
-- Controlled experiment runner comparing `raw-full-file` vs `my-dev-kit-guided` strategies
-- Deterministic correctness scoring from answer keys
-- Token usage, duration, and status comparisons between matched strategy pairs
-- HTML experiment report rendering
-- Static SVG plot generation
-- Optional PNG screenshot capture
-- Gallery manifest and static gallery index output
-- Visualization demos using my-dev-kit commands against benchmark projects
-- Final demo workflow combining all pipeline stages
-- Generic experiment-plugin command surface: `experiment:list`, `experiment:describe`, and `experiment:run`
-- First experiment plugin: `context-strategy-comparison`
-- Target-aware experiment execution for local projects via `experiment:run -- --target <path>`
-- Plugin-aware JSON and HTML reports with plugin, target, variant, metric, artifact, warning, skip, and failure metadata
-- Security validation framework: dependency audit, package tarball inspection, CLI adversarial tests, static scans (CodeQL/Semgrep), bounded fuzz smoke, attack-scenario checks, and structured verdict/report output — runnable against any local project via `security:validate --target <path>`. `security:validate` remains the standalone, focused security-validation command; it is not replaced by the audit adapter below.
-- Generic audit framework: `npm run audit` runs conservative code-rot and security detectors and writes text/JSON reports under `reports/audits/` by default. `code-rot` and `security` are the currently implemented audit types (`--types code-rot`, `--types security`, or combined `--types code-rot,security`); `quality`, `project`, and `all` audit types are planned and fail cleanly instead of running. Audit does not modify target files and does not auto-fix issues. See [docs/COMMANDS.md](docs/COMMANDS.md) for full flags.
-- Language-aware code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, built on a shared source-facts model and normalized language/file-role inventory. Java/Kotlin support is conservative static analysis only: no compiler parsing, type/classpath resolution, or Gradle/Maven execution.
-- Security-validation audit adapter: `npm run audit -- --types security` runs the same `security:validate` internals through the shared audit/report surface, mapping findings into audit issues and linking back to the original `reports/security/` report. `security:validate`'s own checks and reports are unaffected.
-- Android validation: `security:validate --profile android` runs nineteen static checks against an Android project (manifest, permissions, exported components, network security config, backup/release configuration, secrets, signing, WebView/FileProvider, sensitive storage/logging, and Firebase/Google services), with optional opt-in Gradle operations and external tools (Semgrep, OSV-Scanner, Android Lint, Dependency-Check). Default execution starts zero Gradle processes, zero external tools, and zero network operations.
-- Android-aware generic audit: `npm run audit -- --types security --android` runs the same static Android validation through the audit adapter, mapping confirmed findings into audit issues while keeping review-only `CandidateEvidence` separate and never treated as a confirmed issue.
-
----
-
-## Architecture overview
-
-```mermaid
-flowchart TD
-  A[scripts/experiments] --> B[Experiment plugin registry + runner]
-  B --> C[context-strategy-comparison]
-  C --> D[Controlled experiment foundations]
-  D --> E[Plugin + legacy artifacts]
-  E --> F[Plugin-aware reports]
-  E --> G[Plots / screenshots / gallery]
-  H[scripts/security] --> I[Automated security validation]
-  I --> J[Security reports + verdict]
-  I --> K[Android validation]
-  L[scripts/audits] --> M[Generic audit framework]
-  M --> N[Code-rot detectors]
-  M --> I
-```
-
----
+- **Run context-strategy experiments:** compare `raw-full-file` with `my-dev-kit-guided` using deterministic fixtures or locally configured Codex and Claude CLIs.
+- **Audit repository health:** run conservative code-rot detectors for TypeScript/JavaScript, Python, Java, and Kotlin, or adapt security findings into the common audit report.
+- **Validate CLI/package security:** inspect dependencies, package contents, path and subprocess boundaries, malformed inputs, optional static scanners, and bounded fuzz targets.
+- **Validate Android projects:** run nineteen static checks by default, with Gradle operations, external tools, and network access available only through explicit opt-in flags.
+- **Review evidence:** generate JSON and HTML reports, SVG plots, optional screenshots, visualization demos, and a static gallery.
+- **Evaluate stage-context strategies:** compare the two legacy strategies against six additional bounded stage-context strategies — `architecture-context-only`, `architecture-plus-implementation-refresh`, `architecture-plus-implementation-and-test-refresh`, `full-workflow-library`, `bounded-workflow-instruction-packet`, and `combined-bounded-stage-context` — selected through programmatic configuration, not CLI flags. Each strategy's evidence is reported through bounded `report.json`, `report.html`, and `report.txt` output with an explicit `available`/`unavailable`/`not-applicable` metric-availability model and no composite score, grade, ranking, or winning strategy.
 
 ## Quickstart
 
@@ -170,10 +123,13 @@ When `--target` is omitted, the experiment runs in self mode against my-dev-kit-
 | Report screenshot | `lab-output/<report>/experiment-report.png` |
 | Plugin experiment report JSON | `lab-output/experiments/<plugin>/<target>/<run>/report.json` |
 | Plugin experiment report HTML | `lab-output/experiments/<plugin>/<target>/<run>/report.html` |
+| Plugin experiment report text | `lab-output/experiments/<plugin>/<target>/<run>/report.txt` |
 | Plot data | `lab-output/<plots>/plot-data.json` |
 | SVG charts | `lab-output/<plots>/charts/*.svg` |
 | Gallery manifest | `lab-output/<gallery>/gallery-manifest.json` |
 | Gallery index | `lab-output/<gallery>/gallery-index.html` |
+| Audit reports | `reports/audits/<type>/code-rot-audit.txt` and `.json` |
+| Security reports | `reports/security/<prefix>-security-validation.txt` and `.json` |
 
 ---
 
@@ -202,70 +158,22 @@ See [docs/METRICS.md](docs/METRICS.md) for full metric definitions.
 - Codex may expose token totals but can produce timeouts or invalid-output runs
 - Small projects may make raw-full-file cheaper than my-dev-kit-guided; larger localized tasks are where my-dev-kit is expected to become more useful
 - The generic experiment-plugin framework currently ships one plugin, `context-strategy-comparison`; future plugins such as warm-index reuse, incremental-change, and context-window scaling are not implemented yet
-- The current baseline does not prove token savings are guaranteed; it produces auditable evidence for specific cases, targets, agents, and strategies
+- The current release does not guarantee token savings; it produces auditable evidence for specific cases, targets, agents, and strategies
 - Provider telemetry dashboards, semantic LLM judging, and cloud API billing integration are not yet implemented
-
----
-
-## Roadmap direction
-
-my-dev-kit-lab is at a working baseline. The raw-vs-indexed experiment pipeline is fully implemented and produces reproducible artifacts, and real-agent campaign support exists for Codex and Claude. Recent releases added language-aware code-rot detectors (TypeScript/JavaScript, Python, Java, Kotlin), a security-validation audit adapter, and Android validation with an Android-aware extension of that same adapter.
-
-- `v0.4.0` and `v0.4.1`: Android automated security validation (published)
-- `v0.4.2`: Android-aware extension of the existing security audit adapter (see [CHANGELOG.md](CHANGELOG.md) for release status)
-- manual pentest: post-v1 / version TBD
-
-See [docs/ROADMAP.md](docs/ROADMAP.md) for the complete, versioned plan.
+- The six new stage-context strategies have no CLI flags yet, are configured programmatically, and do not yet include plots, screenshots, or gallery integration
+- The published upstream artifacts the stage-context strategies read do not expose considered-but-unselected reads or unnecessary-read evidence; those metrics report `unavailable` rather than zero
 
 ---
 
 ## Security validation
 
-my-dev-kit-lab owns a release-security validation track for **my-dev-kit**. This work is separate from the experiment pipeline and does not replace the generic experiment-plugin roadmap. Its purpose is to generate release-validation evidence for the local CLI/package before release preparation.
-
-`security:validate` remains the standalone, focused command for this validation and is unaffected by the current checked-out state. `npm run audit -- --types security` additionally adapts the same `security:validate` internals into the shared audit/report surface (mapped issues, a `securitySummary` field, and links to the original `reports/security/` output) — it does not replace or duplicate `security:validate`'s own checks or reports.
-
-This is not a web application pentest framework. **my-dev-kit** is a local CLI/package, so the validation model is CLI/package adversarial testing focused on whether it remains:
-
-- local-first
-- deterministic
-- read-only with respect to user source files
-- network-free during normal CLI operation
-- LLM-free
-- database-free
-- safe to run on local repositories
-
-The automated validation gate is implemented. It combines static scans, dependency/package checks, adversarial CLI tests, bounded fuzz smoke tests, and attack-scenario checks with structured text/JSON reports and a four-category verdict.
-
-### Security commands
-
-| Command | Description |
-|---|---|
-| `npm run security:deps` | npm audit, OSV-Scanner (if available), outdated packages |
-| `npm run security:package` | npm pack --dry-run, forbidden content detection |
-| `npm run security:codeql` | CodeQL CLI availability check; skipped gracefully when absent |
-| `npm run security:semgrep` | Semgrep scan via local binary or npx; skipped gracefully when both absent |
-| `npm run test:security` | Automated security and adversarial CLI tests (path traversal, read-only boundaries, malformed artifacts, JSON safety, and related checks) |
-| `npm run test:fuzz:smoke` | 9 bounded fuzz targets, seeded PRNG, completes in under 1 second |
-| `npm run security:validate` | Runs selected security checks, applies profile-aware defaults when requested, and writes text/JSON reports according to `--format` |
-
-CodeQL, Semgrep, and OSV-Scanner are optional. When unavailable locally, they are recorded as `skipped` in the report, not as passed. That can lead to `ready except optional manual checks`; it does not silently turn missing tooling into a clean pass.
-
-Each security command can validate my-dev-kit-lab itself or another local project via `--target <path>`. When `--target` is omitted, the framework performs self-validation. Target projects are inspected in place: their source files are not modified by default, generated artifacts stay under `reports/security/` unless `--out` is used, and external-target reports identify the tool root, target root, target package metadata, target git metadata, command cwd, exit code, and stdout/stderr summaries.
-
-`security:validate` supports `--checks`, `--profile`, `--format`, `--fail-on`, `--out`, and `--report-prefix`. The no-flag path remains backward compatible and still runs the classic implemented check groups: `deps`, `package`, `static`, `cli-adversarial`, and `fuzz`. Supplying `--profile` without `--checks` swaps in that profile's default checks; supplying explicit `--checks` always wins over profile defaults.
-
-The current attack-scenario checks cover boundary, subprocess, secrets, and network assumptions. They are automated adversarial checks, not a manual pentest. Manual pentest is deferred until after `v1.0.0`. `verdictImpact` metadata from each registered scenario drives blocker categorization in report reasoning, and narrowed `--checks` runs are labeled as scoped rather than described as a full release gate.
-
-JSON report poisoning/config-injection checks use a baseline-diff schema guard so legitimate additive JSON fields do not fail the guard while payload-created trusted top-level fields still do. Text reports are sanitized to strip ANSI/control-byte payloads before rendering.
-
-Known limits remain intentionally explicit: secret scanning is bounded rather than exhaustive, the network/local-first check is a bounded static assumption check rather than proof of runtime isolation, package-boundary severity is still result-level rather than per-evidence-item, and profile-specific scenario behavior is limited to profile-based scenario selection/default checks.
+`npm run security:validate` is the standalone security-validation command. It checks local CLI/package boundaries and can inspect another local project with `--target <path>`. The generic audit command can reuse those results through `--types security`, but it does not replace the standalone validator or its reports.
 
-For external-target validation, `security:validate` reads the target `package.json`, detects `scripts.test:security`, and runs `npm run test:security` in the target project root when that script exists. This behavior is validated both from the source checkout and from an installed packed tarball because published-package execution differs from local source execution.
+Android validation uses `--profile android`. Its default path is static and non-destructive: it starts zero Gradle processes, zero external tools, and zero network operations. Only confirmed `SecurityFinding` records can become audit issues; Android `CandidateEvidence` remains review-only evidence.
 
-Generated security reports under `reports/security/` are excluded from git by default. They are produced locally or in CI as release-gate evidence and are not committed to the repository.
+Optional scanners are reported as `skipped` when unavailable, never as passed. The framework does not provide runtime isolation proof, device or APK/AAB analysis, signing verification, Play Console validation, automatic fixes, or manual pentesting. Manual pentest remains deferred until post-v1/version TBD.
 
-See [docs/COMMANDS.md](docs/COMMANDS.md) for full command options and [docs/security-validation-framework.md](docs/security-validation-framework.md) for the security model, implemented modules, and release verdicts.
+See [COMMANDS.md](docs/COMMANDS.md) for exact syntax and [Security Validation Framework](docs/security-validation-framework.md) for checks, evidence semantics, verdicts, and limitations.
 
 ---
 
@@ -291,10 +199,12 @@ Support is optional and does not affect access to the project.
 ## Documentation
 
 - [docs/PROJECT_OVERVIEW.md](docs/PROJECT_OVERVIEW.md) — product purpose and target users
-- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — current and future architecture
+- [docs/CURRENT_STATE.md](docs/CURRENT_STATE.md) — implemented, planned, validated, blocked, and next state
+- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — current components, ownership, flows, and invariants
 - [docs/WORKFLOWS.md](docs/WORKFLOWS.md) — step-by-step workflows with diagrams
 - [docs/COMMANDS.md](docs/COMMANDS.md) — all commands with options and examples
 - [docs/TUTORIAL.md](docs/TUTORIAL.md) — first-run walkthrough
 - [docs/METRICS.md](docs/METRICS.md) — metric definitions and interpretation
-- [docs/ROADMAP.md](docs/ROADMAP.md) — current baseline and future phases
+- [docs/ROADMAP.md](docs/ROADMAP.md) — versioned plans, dependencies, exclusions, and acceptance criteria
 - [docs/GALLERY.md](docs/GALLERY.md) — gallery output explained
+- [docs/security-validation-framework.md](docs/security-validation-framework.md) — security evidence, verdicts, and safety boundaries

package/benchmarks/contracts/todo-behavior.md

@@ -66,5 +66,5 @@ Answer key requirements:
 - `answerKey.expectedFiles` lists files a correct answer may need to cite
 - `answerKey.expectedSymbols` lists relevant symbols
 - `answerKey.expectedFacts` lists weighted facts with unique `id`, `text`, `weight`, and `required`
-- `answerKey.minimumCorrectFacts` defines the future scoring threshold, but scoring is not implemented in this prompt
+- `answerKey.minimumCorrectFacts` defines the required-fact threshold used by deterministic correctness scoring
 - optional `forbiddenWrongClaims` records claims that should be penalized by a later correctness scorer

package/benchmarks/projects/task-analytics-large-mixed/README.md

@@ -1 +1,3 @@
+# task-analytics-large-mixed
+
 Mixed TypeScript and Python benchmark project for deterministic task analytics, quality labeling, and report generation.

package/benchmarks/projects/task-workflow-medium-ts/README.md

@@ -1 +1,3 @@
-Task workflow benchmark project with multiple services, validation rules, filtering, summarization, and deterministic imports.
+# task-workflow-medium-ts
+
+TypeScript task-workflow benchmark with multiple services, validation rules, filtering, summarization, and deterministic imports.

package/dist/src/evaluation/stageContextDeterminism/calculateStageContextDeterminism.js

@@ -0,0 +1,52 @@
+import { createHash } from "node:crypto";
+import { canonicalizeStageContextRun } from "./canonicalizeStageContextRun.js";
+function sha256Hex(text) {
+    return createHash("sha256").update(Buffer.from(text, "utf8")).digest("hex");
+}
+export function calculateStageContextDeterminism(candidates) {
+    if (candidates.length === 0) {
+        throw new Error("calculateStageContextDeterminism requires at least one candidate.");
+    }
+    if (candidates.length === 1) {
+        return {
+            availability: "not-applicable",
+            repeatCount: 1,
+            deterministic: null,
+            baselineSha256: null,
+            runDigests: [],
+            mismatchRunNumbers: [],
+            reason: "Repeated-run determinism requires at least two runs."
+        };
+    }
+    let runDigests;
+    try {
+        runDigests = candidates.map((candidate) => ({
+            runNumber: candidate.runNumber,
+            sha256: sha256Hex(canonicalizeStageContextRun(candidate.value))
+        }));
+    }
+    catch {
+        return {
+            availability: "unavailable",
+            repeatCount: candidates.length,
+            deterministic: null,
+            baselineSha256: null,
+            runDigests: [],
+            mismatchRunNumbers: [],
+            reason: "Stage-context run values could not be canonicalized for determinism comparison."
+        };
+    }
+    const baselineSha256 = runDigests[0].sha256;
+    const mismatchRunNumbers = runDigests
+        .filter((digest) => digest.sha256 !== baselineSha256)
+        .map((digest) => digest.runNumber);
+    return {
+        availability: "available",
+        repeatCount: candidates.length,
+        deterministic: mismatchRunNumbers.length === 0,
+        baselineSha256,
+        runDigests,
+        mismatchRunNumbers,
+        reason: null
+    };
+}

package/dist/src/evaluation/stageContextDeterminism/canonicalizeStageContextRun.js

@@ -0,0 +1,53 @@
+export function canonicalizeStageContextRun(value) {
+    const seen = new WeakSet();
+    function serialize(input) {
+        if (input === null)
+            return "null";
+        const kind = typeof input;
+        if (kind === "bigint") {
+            throw new TypeError("Stage-context run values must not contain BigInt.");
+        }
+        if (kind === "function") {
+            throw new TypeError("Stage-context run values must not contain functions.");
+        }
+        if (kind === "symbol") {
+            throw new TypeError("Stage-context run values must not contain symbols.");
+        }
+        if (kind === "undefined") {
+            return "null";
+        }
+        if (kind === "number") {
+            if (!Number.isFinite(input)) {
+                throw new RangeError("Stage-context run values must not contain non-finite numbers.");
+            }
+            return JSON.stringify(input);
+        }
+        if (kind === "boolean" || kind === "string") {
+            return JSON.stringify(input);
+        }
+        if (Array.isArray(input)) {
+            if (seen.has(input)) {
+                throw new TypeError("Stage-context run values must not contain circular references.");
+            }
+            seen.add(input);
+            const items = input.map((item) => (item === undefined ? "null" : serialize(item)));
+            seen.delete(input);
+            return `[${items.join(",")}]`;
+        }
+        if (kind === "object") {
+            const obj = input;
+            if (seen.has(obj)) {
+                throw new TypeError("Stage-context run values must not contain circular references.");
+            }
+            seen.add(obj);
+            const keys = Object.keys(obj)
+                .filter((key) => obj[key] !== undefined)
+                .sort();
+            const parts = keys.map((key) => `${JSON.stringify(key)}:${serialize(obj[key])}`);
+            seen.delete(obj);
+            return `{${parts.join(",")}}`;
+        }
+        throw new TypeError("Stage-context run values contain an unsupported type.");
+    }
+    return serialize(value);
+}

package/dist/src/evaluation/stageContextDeterminism/index.js

@@ -0,0 +1,2 @@
+export { canonicalizeStageContextRun } from "./canonicalizeStageContextRun.js";
+export { calculateStageContextDeterminism } from "./calculateStageContextDeterminism.js";

package/dist/src/evaluation/stageContextDeterminism/types.js

@@ -0,0 +1 @@
+export {};

package/dist/src/evaluation/stageContextExpectations/index.js

@@ -0,0 +1,2 @@
+export { validateStageContextExpectationFixtureV1 } from "./validation.js";
+export { readStageContextExpectationFixtureV1 } from "./readStageContextExpectationFixtureV1.js";

package/dist/src/evaluation/stageContextExpectations/readStageContextExpectationFixtureV1.js

@@ -0,0 +1,50 @@
+import { readFile } from "node:fs/promises";
+import path from "node:path";
+import { validateStageContextExpectationFixtureV1 } from "./validation.js";
+export async function readStageContextExpectationFixtureV1(sourcePathInput) {
+    const sourcePath = path.resolve(sourcePathInput);
+    let raw;
+    try {
+        raw = await readFile(sourcePath, "utf8");
+    }
+    catch (error) {
+        const code = error.code;
+        if (code === "ENOENT") {
+            return {
+                ok: false,
+                sourcePath,
+                code: "FILE_NOT_FOUND",
+                message: `Stage context expectation fixture was not found at "${sourcePath}".`
+            };
+        }
+        return {
+            ok: false,
+            sourcePath,
+            code: "UNREADABLE_FILE",
+            message: `Stage context expectation fixture at "${sourcePath}" could not be read: ${error.message}`
+        };
+    }
+    let parsed;
+    try {
+        parsed = JSON.parse(raw);
+    }
+    catch (error) {
+        return {
+            ok: false,
+            sourcePath,
+            code: "MALFORMED_JSON",
+            message: `Stage context expectation fixture at "${sourcePath}" is not valid JSON: ${error.message}`
+        };
+    }
+    if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) {
+        return {
+            ok: false,
+            sourcePath,
+            code: "NON_OBJECT_ROOT",
+            message: `Stage context expectation fixture at "${sourcePath}" must parse to a JSON object at the root.`,
+            expected: "object",
+            actual: parsed
+        };
+    }
+    return validateStageContextExpectationFixtureV1(parsed, sourcePath);
+}

package/dist/src/evaluation/stageContextExpectations/readTypes.js

@@ -0,0 +1 @@
+export {};

package/dist/src/evaluation/stageContextExpectations/types.js

@@ -0,0 +1 @@
+export {};