@prismatic-io/lux 0.0.2-preview.3 → 0.0.2-preview.32

package/skills/lux/SKILL.md

@@ -1,85 +0,0 @@
----
-name: lux
-description: Author, validate, run, inspect, compare, and improve coding-agent evaluations with the Lux CLI. Use when creating Lux cases or campaigns, choosing assertions/personas/drivers, diagnosing eval failures, comparing runs, or running Lux experiments.
----
-
-# Lux
-
-Use Lux as an eval test runner and evidence system for coding agents. Prefer
-small, discriminating cases with deterministic assertions; add an LLM rubric
-only for qualities that artifacts and commands cannot establish.
-
-## Start here
-
-1. Inspect the repository and its existing `lux.config.ts`, cases, fixtures,
-   prompts, and experiments before adding files.
-2. If Lux is not initialized, run `npm exec -- lux init` for ordinary evals or
-   `npm exec -- lux init --experiment` for an improvement campaign.
-3. Read [references/eval-authoring.md](references/eval-authoring.md) before
-   authoring or reviewing a case.
-4. Read [references/cli.md](references/cli.md) before running, diagnosing,
-   comparing, or promoting results.
-5. Run `npm exec -- lux doctor` before model calls. Treat its errors as
-   authoring problems, not as permission to bypass validation.
-
-## Workflow
-
-### Author an eval
-
-- Define the behavior and likely failure mode first.
-- Make the prompt realistic and self-contained.
-- Give the persona only facts and preferences the simulated user should know.
-- Assert observable outcomes, not implementation details.
-- Use stable assertion IDs so grading evidence remains intelligible.
-- Use the cheapest deterministic assertion that proves each requirement.
-- Include a rubric assertion only when semantic judgment is unavoidable, and
-  write one narrow, falsifiable criterion per assertion.
-- Pin the subject driver model explicitly when comparability matters.
-
-### Run and diagnose
-
-```bash
-npm exec -- lux doctor
-npm exec -- lux run <case-or-filter> --loop 3
-npm exec -- lux view <run-or-suite>
-```
-
-Inspect `case.json`, `run.json`, `events.jsonl`, `grading.json`, and `artifacts/`
-before changing the case. Separate subject failure from harness failure:
-
-- Subject failure: the agent misunderstood, asked poorly, used the wrong tools,
-  or left incorrect artifacts.
-- Harness failure: the prompt was underspecified, the persona lacked a needed
-  fact, an assertion measured a proxy, or the judge criterion was ambiguous.
-
-Do not weaken a valid assertion merely to make a run pass.
-
-### Improve with evidence
-
-Use repeated runs and held-out splits for noisy agent behavior. For skill,
-prompt, or agent-source changes, use `subjectPath()` in cases and an experiment
-campaign with immutable candidates. Plan first, run the experiment, inspect the
-report, and apply a promoted candidate as a separate action:
-
-```bash
-npm exec -- lux experiment experiments/example.experiment.ts --plan
-npm exec -- lux experiment experiments/example.experiment.ts
-npm exec -- lux report <experiment-dir>
-npm exec -- lux apply <experiment-dir>
-```
-
-Never edit a materialized candidate or managed experiment record by hand.
-
-## Guardrails
-
-- Lux eval projects execute project-authored commands and agent-generated work.
-  Run untrusted evals only in a disposable environment.
-- Do not infer success from agent prose. Grade the final artifacts and recorded
-  behavior.
-- Do not use only `run-succeeded` for a case that claims to verify a concrete
-  result.
-- Do not let train examples leak into validation or test splits.
-- Do not combine evidence across changed models, harnesses, cases, or subjects;
-  Lux identity checks exist to prevent that.
-- Prefer `--plan`, `doctor`, `view`, and `report` before expensive or mutating
-  commands.

package/skills/lux/references/cli.md

@@ -1,102 +0,0 @@
-# Lux CLI runbook
-
-Invoke the project-local CLI with `npm exec -- lux ...` unless the repository
-uses another package-manager convention.
-
-## Author and preflight
-
-```bash
-lux init
-lux init --experiment
-lux doctor
-lux doctor experiments/campaign.experiment.ts --json
-lux run --list
-lux run --list-tags
-```
-
-`init` is conservative around existing files. `doctor` validates configuration,
-case discovery, driver executables, campaign splits, confidence feasibility,
-and projected budgets without model calls.
-
-## Run
-
-```bash
-lux run
-lux run retry-flag
-lux run cases/retry-flag.ts
-lux run --tag smoke
-lux run retry --loop 5 --concurrency 2
-lux run one-case --interactive
-```
-
-Filters match case IDs and paths. Tags come from `meta.tags`. A run invocation
-also creates a suite record, which is the right unit for repeated or multi-case
-comparison.
-
-## Inspect and compare
-
-```bash
-lux view
-lux view <run-or-suite>
-lux view --experiments
-lux compare <run-or-suite-a> <run-or-suite-b>
-```
-
-Use `view` before opening raw files. When diagnosing, inspect:
-
-- `case.json`: frozen authored inputs;
-- `run.json`: exit reason, timing, usage, source identity;
-- `events.jsonl`: trajectory, tools, and interrupts;
-- `grading.json`: assertion evidence;
-- `artifacts/`: the subject workspace output.
-
-`compare` exits nonzero on a gated regression, making it suitable for CI.
-
-## Regrade and annotate
-
-```bash
-lux grade <run>
-lux grade <run> --case cases/changed-grader.ts
-lux annotate <run> \
-  --out evals/annotations.json \
-  --rubric evals/rubrics/case.json \
-  --label pass
-```
-
-Regrading preserves subject evidence and changes only grading. Human annotations
-support grader-alignment experiments; keep assertion IDs aligned with the
-rubric and record reviewer context where available.
-
-## Experiment lifecycle
-
-```bash
-lux experiment <campaign> --plan
-lux experiment <campaign>
-lux optimize <campaign> --plan
-lux optimize <campaign>
-lux optimize <campaign> --resume <experiment-dir>
-lux view <experiment-dir>
-lux report <experiment-dir> --out .lux-runs/reports/decision.md
-lux apply <experiment-dir>
-```
-
-`experiment` compares authored variants. `optimize` searches proposed changes.
-Both create durable, resumable, integrity-checked records. `report` re-verifies
-the ledger and promotion decision. `apply` is deliberately separate and
-hash-guards the source it changes.
-
-Use the exact resume, report, and apply commands printed by Lux. Do not edit
-`.lux-runs/.experiments/` manually.
-
-## Failure triage
-
-- Discovery/config error: run `doctor`, verify paths and exported definitions.
-- Driver unavailable: verify the selected CLI is installed and the model is
-  explicitly configured where required.
-- Agent exit/error: inspect `events.jsonl` and the driver section of `run.json`.
-- Assertion failure: inspect the assertion evidence and final artifact; decide
-  whether the subject or measurement is wrong.
-- Judge disagreement: narrow the rubric, regrade stored evidence, and collect
-  human annotations before changing the subject.
-- Interrupted experiment: resume the printed experiment directory; do not start
-  a nominally identical campaign and combine results yourself.

package/skills/lux/references/eval-authoring.md

@@ -1,105 +0,0 @@
-# Authoring useful Lux evals
-
-## Case shape
-
-A case default-exports `defineEvalCase`:
-
-```ts
-import { defineEvalCase } from "@prismatic-io/lux";
-
-export default defineEvalCase({
-  id: "retry-flag",
-  prompt: "Add a --retry <n> flag to the fetch command.",
-  persona: "You are the maintainer. If asked, retries default to 3.",
-  driver: { name: "codex", config: { model: "YOUR_CODEX_MODEL" } },
-  assertions: [
-    { id: "flag-present", type: "file-contains", path: "src/cli.ts", text: "--retry" },
-    { id: "tests-pass", type: "command-exits-zero", command: "npm test" },
-    { id: "test-coverage", type: "rubric", criteria: "A test covers the new retry flag." },
-  ],
-  meta: { tags: ["cli", "validation"] },
-});
-```
-
-Use the project-level default driver when all cases share it. Case-level
-drivers are useful for intentional cross-agent or cross-model comparisons.
-
-## Design checklist
-
-Before writing:
-
-1. State the capability or regression in one sentence.
-2. Identify the artifact, command result, or interaction that proves it.
-3. List plausible false positives. Adjust assertions to reject them.
-4. Decide whether the case needs a fixture and whether each run must begin from
-   an isolated copy.
-5. Identify any question a competent agent must ask. Put only the answer and
-   relevant preferences in the persona.
-
-After writing:
-
-1. Run `lux doctor`.
-2. List discovery with `lux run <filter> --list`.
-3. Run the case more than once if agent behavior can vary.
-4. Inspect events and artifacts, including failed runs.
-5. Confirm each assertion fails when its requirement is deliberately absent.
-
-## Assertion selection
-
-Prefer deterministic assertions:
-
-- `run-succeeded`: the driver completed; necessary but rarely sufficient.
-- `file-exists`: a required artifact exists.
-- `file-contains`: a stable literal is present.
-- `json-path-equals`: structured output contains the expected value.
-- `command-exits-zero`: project-authored validation succeeds.
-
-Use `rubric` for semantic properties such as design quality, completeness, or
-whether a test meaningfully covers behavior. Keep criteria narrow. Avoid words
-like “good,” “proper,” or “best practice” without observable conditions.
-
-Assertions should test user-visible behavior and durable contracts. Avoid
-requiring a specific function name, file layout, or algorithm unless that is
-the contract under evaluation.
-
-## Human-in-the-loop behavior
-
-The persona is simulated user state, not another system prompt. Include:
-
-- facts the user would know;
-- preferences needed to resolve legitimate ambiguity;
-- approval boundaries relevant to the task.
-
-Do not tell the persona how to help the agent solve the task. If no interaction
-is part of the capability being evaluated, omit the persona or keep it minimal.
-
-Use `interactionMode: "defer-resume"` for automated Claude Code question-tool
-handling. Use `textQuestionFallback: true` only when intentionally evaluating
-models that ask in prose instead of using the supported tool.
-
-## Experiments for prompts, skills, and agent source
-
-Bind mutable source through `subjectPath()`:
-
-```ts
-import { defineEvalCase, subjectPath } from "@prismatic-io/lux";
-
-export default defineEvalCase({
-  id: "skill-routing",
-  prompt: `Use the skill at ${subjectPath("plugin/skills/routing/SKILL.md")} to complete the task.`,
-  assertions: [{ id: "completed", type: "run-succeeded" }],
-  meta: { tags: ["validation"] },
-});
-```
-
-Campaigns should:
-
-- restrict `subject.mutable` to the intended source;
-- define named, explicit driver/model profiles;
-- keep train, validation, and test selectors disjoint;
-- budget calls, tokens, cost, and wall time;
-- use repetitions and confidence gates appropriate to stochastic outcomes;
-- reserve held-out cases for promotion and final confirmation.
-
-Use `experiment` for authored variants and `optimize` for model-proposed source
-changes. Always run the corresponding `--plan` command first.

package/src/answerers/scripted/index.ts

@@ -1,186 +0,0 @@
-import { readFile } from "node:fs/promises";
-import { resolve } from "node:path";
-import { load as loadYaml } from "js-yaml";
-import { z } from "zod";
-import {
-  type Answer,
-  type Answerer,
-  type AnswererPlugin,
-  formatZodIssues,
-  type Interrupt,
-  InterruptKindSchema,
-} from "../../core/index.js";
-import { approveAnswer } from "../shared.js";
-
-const MatchSchema = z
-  .object({
-    kind: InterruptKindSchema.optional(),
-    id: z.string().optional(),
-    question_contains: z.string().optional(),
-    question_matches: z.string().optional(),
-    request_contains: z.string().optional(),
-  })
-  .strict();
-
-const AnsweredAskSchema = z.object({ answer: z.string() }).strict();
-const AnsweredApproveSchema = z
-  .object({ approved: z.boolean(), reason: z.string().optional() })
-  .strict();
-const AnsweredCustomSchema = z.object({ payload: z.unknown() }).strict();
-
-const RuleSchema = z.union([
-  z.object({ match: MatchSchema, answer: z.string() }).strict(),
-  z.object({ match: MatchSchema, approved: z.boolean(), reason: z.string().optional() }).strict(),
-  z.object({ match: MatchSchema, payload: z.unknown() }).strict(),
-  z.object({ default: AnsweredAskSchema }).strict(),
-  z.object({ default: AnsweredApproveSchema }).strict(),
-  z.object({ default: AnsweredCustomSchema }).strict(),
-]);
-
-const FixtureSchema = z.array(RuleSchema);
-
-export const ScriptedAnswererConfigSchema = z.object({ fixturePath: z.string() }).strict();
-
-export type ScriptedAnswererConfig = z.infer<typeof ScriptedAnswererConfigSchema>;
-
-type Rule = z.infer<typeof RuleSchema>;
-
-/** A rule vetted at load time: kind/body cross-checked, regex pre-compiled. */
-type PreparedRule = { rule: Rule; questionPattern: RegExp | undefined };
-
-const ruleLabel = (path: string, index: number): string => `rule ${index + 1} of ${path}`;
-
-const bodyKind = (rule: Rule): Interrupt["kind"] => {
-  const body = "default" in rule ? rule.default : rule;
-  if ("answer" in body) return "ask";
-  if ("approved" in body) return "approve";
-  return "custom";
-};
-
-const matchableKind = (m: z.infer<typeof MatchSchema>): Interrupt["kind"] | undefined => {
-  if (m.kind) return m.kind;
-  if (m.question_contains !== undefined || m.question_matches !== undefined) return "ask";
-  if (m.request_contains !== undefined) return "approve";
-  return undefined;
-};
-
-const compileQuestionPattern = (rule: Rule, path: string, index: number): RegExp | undefined => {
-  if (!("match" in rule) || rule.match.question_matches === undefined) return undefined;
-  try {
-    return new RegExp(rule.match.question_matches);
-  } catch (err) {
-    throw new Error(
-      `scripted: invalid question_matches in ${ruleLabel(path, index)}: ${
-        err instanceof Error ? err.message : String(err)
-      }`,
-    );
-  }
-};
-
-// The rule's match fields and answer body are independently valid shapes, so
-// their interrupt kinds still need a semantic cross-check.
-const prepareRule = (rule: Rule, path: string, index: number): PreparedRule => {
-  if ("match" in rule) {
-    const wanted = matchableKind(rule.match);
-    const answered = bodyKind(rule);
-    if (wanted !== undefined && wanted !== answered) {
-      throw new Error(
-        `scripted: ${ruleLabel(path, index)} matches ${wanted} interrupts but its body answers ${answered}`,
-      );
-    }
-  }
-  return { rule, questionPattern: compileQuestionPattern(rule, path, index) };
-};
-
-const matches = ({ rule, questionPattern }: PreparedRule, interrupt: Interrupt): boolean => {
-  if (!("match" in rule)) return false;
-  const m = rule.match;
-  if (m.kind && m.kind !== interrupt.kind) return false;
-  if (m.id && m.id !== interrupt.id) return false;
-  if (m.question_contains) {
-    if (interrupt.kind !== "ask") return false;
-    if (!interrupt.question.includes(m.question_contains)) return false;
-  }
-  if (questionPattern) {
-    if (interrupt.kind !== "ask") return false;
-    if (!questionPattern.test(interrupt.question)) return false;
-  }
-  if (m.request_contains) {
-    if (interrupt.kind !== "approve") return false;
-    if (!interrupt.request.includes(m.request_contains)) return false;
-  }
-  return true;
-};
-
-// A fixture may carry one default per interrupt kind; the shape of the
-// default's body is what says which kind it can answer.
-const defaultAnswersKind = (rule: Rule, kind: Interrupt["kind"]): boolean => {
-  if (!("default" in rule)) return false;
-  const body = rule.default;
-  if (kind === "ask") return "answer" in body;
-  if (kind === "approve") return "approved" in body;
-  return "payload" in body;
-};
-
-const buildAnswer = (rule: Rule, interrupt: Interrupt): Answer => {
-  const body = "default" in rule ? rule.default : rule;
-  if (interrupt.kind === "ask") {
-    if (!("answer" in body)) {
-      throw new Error(`scripted: rule for ask interrupt missing 'answer' (id=${interrupt.id})`);
-    }
-    return { kind: "ask", id: interrupt.id, text: body.answer };
-  }
-  if (interrupt.kind === "approve") {
-    if (!("approved" in body)) {
-      throw new Error(
-        `scripted: rule for approve interrupt missing 'approved' (id=${interrupt.id})`,
-      );
-    }
-    return approveAnswer(interrupt.id, body.approved, body.reason);
-  }
-  if (!("payload" in body)) {
-    throw new Error(`scripted: rule for custom interrupt missing 'payload' (id=${interrupt.id})`);
-  }
-  return { kind: "custom", id: interrupt.id, payload: body.payload };
-};
-
-class ScriptedAnswererImpl implements Answerer {
-  private rules: PreparedRule[] = [];
-  private readonly config: ScriptedAnswererConfig;
-
-  constructor(config: ScriptedAnswererConfig) {
-    this.config = config;
-  }
-
-  async prepare(): Promise<void> {
-    const path = resolve(this.config.fixturePath);
-    const raw = loadYaml(await readFile(path, "utf-8"));
-    const parsed = FixtureSchema.safeParse(raw);
-    if (!parsed.success) {
-      throw new Error(`scripted: invalid fixture at ${path}: ${formatZodIssues(parsed.error)}`);
-    }
-    this.rules = parsed.data.map((rule, index) => prepareRule(rule, path, index));
-  }
-
-  async answer(interrupt: Interrupt): Promise<Answer> {
-    const matched = this.rules.find((r) => matches(r, interrupt));
-    if (matched) {
-      return buildAnswer(matched.rule, interrupt);
-    }
-    const fallback = this.rules.find((r) => defaultAnswersKind(r.rule, interrupt.kind));
-    if (fallback) {
-      return buildAnswer(fallback.rule, interrupt);
-    }
-    throw new Error(
-      `scripted: no rule matched interrupt (kind=${interrupt.kind}, id=${interrupt.id}) and no default provided`,
-    );
-  }
-
-  async close(): Promise<void> {}
-}
-
-export const scriptedAnswerer: AnswererPlugin<ScriptedAnswererConfig> = {
-  name: "scripted",
-  configSchema: ScriptedAnswererConfigSchema,
-  create: (config) => new ScriptedAnswererImpl(config),
-};

package/src/cli/skills.ts

@@ -1,129 +0,0 @@
-import { lstat, mkdir, readdir, readFile, writeFile } from "node:fs/promises";
-import { homedir } from "node:os";
-import { dirname, join, relative, resolve, sep } from "node:path";
-import { fileURLToPath } from "node:url";
-import { z } from "zod";
-
-const PlatformSchema = z.enum(["claude", "claude-code", "codex", "all"]);
-export type SkillsPlatform = z.infer<typeof PlatformSchema>;
-const SkillNameSchema = z.enum(["lux", "lux-answerer"]);
-export type BundledSkillName = z.infer<typeof SkillNameSchema>;
-
-const InstallSkillsOptionsSchema = z
-  .object({
-    cwd: z.string().min(1),
-    platform: PlatformSchema,
-    project: z.boolean().default(false),
-    dir: z.string().min(1).optional(),
-    name: SkillNameSchema.optional(),
-  })
-  .strict();
-
-export interface InstalledSkill {
-  name: BundledSkillName;
-  platform: "claude-code" | "codex";
-  path: string;
-}
-
-const BUNDLED_SKILLS = [
-  {
-    name: "lux",
-    description: "Author and drive coding-agent evals with Lux",
-    platforms: ["claude-code", "codex"],
-  },
-  {
-    name: "lux-answerer",
-    description: "Play a simulated user for a running Lux eval",
-    platforms: ["claude-code"],
-  },
-] as const satisfies readonly {
-  name: BundledSkillName;
-  description: string;
-  platforms: readonly ("claude-code" | "codex")[];
-}[];
-
-export const formatSkillsList = (): string =>
-  BUNDLED_SKILLS.map(
-    ({ name, description, platforms }) => `${name}\t${description}\t${platforms.join(",")}`,
-  ).join("\n");
-
-const targetsFor = (
-  platform: SkillsPlatform,
-  cwd: string,
-  project: boolean,
-  dir?: string,
-  name?: BundledSkillName,
-): (InstalledSkill & { source: string })[] => {
-  const platforms =
-    platform === "all"
-      ? (["claude-code", "codex"] as const)
-      : ([platform === "claude" ? "claude-code" : platform] as const);
-  const scopeRoot = project ? resolve(cwd) : homedir();
-  const skills = name ? BUNDLED_SKILLS.filter((skill) => skill.name === name) : BUNDLED_SKILLS;
-  return platforms.flatMap((platformName) =>
-    skills
-      .filter((skill) =>
-        (skill.platforms as readonly ("claude-code" | "codex")[]).includes(platformName),
-      )
-      .map((skill) => ({
-        name: skill.name,
-        platform: platformName,
-        source: fileURLToPath(new URL(`../../skills/${skill.name}`, import.meta.url)),
-        path: dir
-          ? join(resolve(dir), skill.name)
-          : join(
-              scopeRoot,
-              platformName === "claude-code" ? ".claude" : ".codex",
-              "skills",
-              skill.name,
-            ),
-      })),
-  );
-};
-
-const copySkillDirectory = async (source: string, target: string): Promise<void> => {
-  const sourceInfo = await lstat(source);
-  if (!sourceInfo.isDirectory() || sourceInfo.isSymbolicLink()) {
-    throw new Error(`lux skills: bundled skill source is not a real directory: ${source}`);
-  }
-  await mkdir(target, { recursive: true });
-  for (const entry of await readdir(source, { withFileTypes: true })) {
-    const sourcePath = join(source, entry.name);
-    const targetPath = join(target, entry.name);
-    if (entry.isSymbolicLink()) {
-      throw new Error(`lux skills: bundled skill contains a symbolic link: ${sourcePath}`);
-    }
-    if (entry.isDirectory()) {
-      await copySkillDirectory(sourcePath, targetPath);
-    } else if (entry.isFile()) {
-      await mkdir(dirname(targetPath), { recursive: true });
-      await writeFile(targetPath, await readFile(sourcePath));
-    } else {
-      throw new Error(`lux skills: bundled skill contains an unsupported entry: ${sourcePath}`);
-    }
-  }
-};
-
-export const installSkills = async (
-  options: z.input<typeof InstallSkillsOptionsSchema>,
-): Promise<InstalledSkill[]> => {
-  const parsed = InstallSkillsOptionsSchema.parse(options);
-  const targets = targetsFor(parsed.platform, parsed.cwd, parsed.project, parsed.dir, parsed.name);
-  if (targets.length === 0) {
-    throw new Error(
-      `lux skills: '${parsed.name}' is not available for platform '${parsed.platform}'`,
-    );
-  }
-  for (const target of targets) await copySkillDirectory(target.source, target.path);
-  return targets.map(({ source: _, ...installed }) => installed);
-};
-
-export const formatSkillsInstall = (installed: InstalledSkill[], cwd: string): string =>
-  installed
-    .map(
-      ({ name, platform, path }) =>
-        `installed ${name} skill for ${platform} at ${
-          relative(cwd, path).split(sep).join("/") || "."
-        }`,
-    )
-    .join("\n");