@prismatic-io/lux 0.0.2-preview.4 → 0.0.2-preview.40

package/skills/lux/references/eval-authoring.md

@@ -1,257 +0,0 @@
-# Authoring useful Lux evals
-
-## Case shape
-
-A case default-exports `defineEvalCase`:
-
-```ts
-import { defineEvalCase } from "@prismatic-io/lux";
-
-export default defineEvalCase({
-  id: "retry-flag",
-  prompt: "Add a --retry <n> flag to the fetch command.",
-  persona: "You are the maintainer. If asked, retries default to 3.",
-  driver: { name: "codex", config: { model: "YOUR_CODEX_MODEL" } },
-  assertions: [
-    { id: "flag-present", type: "file-contains", path: "src/cli.ts", text: "--retry" },
-    { id: "tests-pass", type: "command-exits-zero", command: "npm test" },
-    { id: "test-coverage", type: "rubric", criteria: "A test covers the new retry flag." },
-  ],
-  meta: { tags: ["cli", "validation"] },
-});
-```
-
-Use the project-level default driver when all cases share it. Case-level
-drivers are useful for intentional cross-agent or cross-model comparisons.
-
-## Design checklist
-
-Before writing:
-
-1. State the capability or regression in one sentence.
-2. Identify the artifact, command result, or interaction that proves it.
-3. List plausible false positives. Adjust assertions to reject them.
-4. Decide whether the case needs a fixture and whether each run must begin from
-   an isolated copy.
-5. Identify any question a competent agent must ask. Put only the answer and
-   relevant preferences in the persona.
-
-After writing:
-
-1. Run `lux doctor`.
-2. List discovery with `lux run <filter> --list`.
-3. Run the case more than once if agent behavior can vary.
-4. Inspect events and artifacts, including failed runs.
-5. Confirm each assertion fails when its requirement is deliberately absent.
-
-## Assertion selection
-
-Prefer deterministic assertions:
-
-- `run-succeeded`: the driver completed; necessary but rarely sufficient.
-- `file-exists`: a required artifact exists.
-- `file-contains`: a stable literal is present.
-- `json-pointer-equals`: structured output contains the expected value.
-- `command-exits-zero`: project-authored validation succeeds.
-
-Use the more specific bundled evidence checks when they match the contract:
-
-- Files: `file-matches`, `file-regex-set`, `file-regex-count`, and `glob-count`.
-- Output/data: `contains`, `not-contains`, `regex`, and `json-pointer-equals`.
-- Interaction: `interrupt-count`, `tool-called`, `tool-called-with`, and
-  `tool-call-count`; use `by: "main"` or `by: "subagent"` when ownership matters.
-- Paired tool evidence: `tool-invocation`, `tool-result-contains`,
-  `tool-result-json-pointer-equals`, `tool-result-is-error`, and
-  `no-error-results`. Prefer these when result status or payload matters; a tool
-  call without a paired result is unresolved, not successful.
-- Budgets: `latency-under`, `tokens-under`, `cost-under`, and
-  `artifact-bytes-under`.
-- MCP discovery: `server-info-equals`, `tool-listed`,
-  `tools-listed-count`, and `capability-absent`.
-
-Most file assertions read captured `artifacts`; set `where: "cwd"` only when
-the durable contract intentionally includes the staged run workspace. Use
-stable assertion `id` values for repetitions, comparisons, annotations, and
-experiments. `weight` affects aggregate score and `threshold` sets an optional
-per-assertion score floor.
-
-Use `rubric` for semantic properties such as design quality, completeness, or
-whether a test meaningfully covers behavior. Keep criteria narrow. Avoid words
-like “good,” “proper,” or “best practice” without observable conditions.
-
-Bound rubric evidence to what the criterion needs. Include `tool-result` event
-evidence when status matters, and request tool inputs/results only when their
-payload is necessary; this controls judge context and cost.
-
-Persona answering and rubric judging have different difficulty profiles. A
-persona usually chooses from facts already supplied in the case, so prefer a
-cheap model and low effort. Semantic grading should use a stronger model and
-enough effort to distinguish real evidence from plausible prose. Configure
-them independently when that tradeoff matters:
-
-```ts
-// lux.config.ts
-defaultAnswerer: {
-  name: "persona",
-  config: {
-    provider: "codex",
-    model: "YOUR_SMALL_PERSONA_MODEL",
-    reasoningEffort: "low",
-  },
-},
-
-// case assertion
-{
-  id: "behavior-quality",
-  type: "rubric",
-  criteria: "The tests exercise retry exhaustion and recovery.",
-  provider: "codex",
-  model: "YOUR_STRONG_JUDGE_MODEL",
-  reasoningEffort: "high",
-}
-```
-
-The project `harness` supplies shared defaults to persona and rubric calls;
-answerer config and rubric fields override it independently. Always pin model
-and effort when comparing runs, and inspect answerer versus judge usage in the
-run summary before increasing either budget.
-
-Assertions should test user-visible behavior and durable contracts. Avoid
-requiring a specific function name, file layout, or algorithm unless that is
-the contract under evaluation.
-
-## Human-in-the-loop behavior
-
-The persona is simulated user state, not another system prompt. Include:
-
-- facts the user would know;
-- preferences needed to resolve legitimate ambiguity;
-- approval boundaries relevant to the task.
-
-Do not tell the persona how to help the agent solve the task. If no interaction
-is part of the capability being evaluated, omit the persona or keep it minimal.
-
-For Claude Code, use `interactionMode: "defer-resume"` for automated
-question-tool handling. Use `textQuestionFallback: true` only when
-intentionally evaluating models that ask in prose instead of using the
-supported tool. For Codex, keep the default `interactionMode: "app-server"`
-when the case can interrupt; compatibility mode `"exec"` cannot accept
-mid-run responses.
-
-Use `--interactive` for a real terminal user or `--claude-answerer` for the
-Claude plugin protocol. Both force sequential execution.
-
-## Project extensions
-
-For a reusable project-specific assertion, define it with `defineAssertion`
-and a direct Zod object whose `type` is a matching literal. Import `z` from Lux,
-return an ordinary failed result for missing/invalid product evidence, and
-reserve throws or `errorMessage` for evaluator faults. Increment
-`pluginVersion` whenever schema or evaluation behavior changes. Prefer the
-bounded, traversal-safe `readRunArtifactText` helper over direct filesystem
-reads.
-
-Build a typed case authoring function once instead of importing plugins in
-every case:
-
-```ts
-const assertions = defineAssertionRegistry(projectAssertion);
-export const defineEvalCase = defineLuxCase.extend({ assertions });
-```
-
-Case-bound plugins remain plain serialized plans but require the current case
-path for regrading: `lux grade <run> --case <case.ts>`.
-
-Use `defineFixtures` for Vitest-style suite/case setup around `await use()` or
-`defineLifecycleHooks` when cleanup-returning callbacks fit better. Setup runs
-in declaration order and teardown in reverse, including failures. Call
-`await use()` exactly once. Fixture `value`/`identity` must be public JSON;
-keep secrets in the prepared resource or environment. Increment
-`pluginVersion` when lifecycle behavior changes. Register globally in
-`lux.config.ts`, or attach selectively with
-`defineEvalCase.extend({ fixtures: [...] })`.
-
-Custom drivers that own an in-memory filesystem should persist it with
-`writeArtifactSnapshot` from `@prismatic-io/lux/artifact-snapshot`. Snapshots
-are bounded, traversal-safe, immutable per name, and can use `baseSnapshot`
-for copy-on-write reuse while remaining independently complete. Ordinary file
-assertions then grade the indexed files without a snapshot-specific API.
-
-## Experiments for prompts, skills, and agent source
-
-Bind mutable source through `subjectPath()`:
-
-```ts
-import { defineEvalCase, subjectPath } from "@prismatic-io/lux";
-
-export default defineEvalCase({
-  id: "skill-routing",
-  prompt: `Use the skill at ${subjectPath("plugin/skills/routing/SKILL.md")} to complete the task.`,
-  assertions: [{ id: "completed", type: "run-succeeded" }],
-  meta: { tags: ["validation"] },
-});
-```
-
-Campaigns should:
-
-- restrict `subject.mutable` to the intended source;
-- define named, explicit driver/model profiles;
-- keep train, validation, and test selectors disjoint;
-- budget calls, tokens, cost, and wall time;
-- use repetitions and confidence gates appropriate to stochastic outcomes;
-- reserve held-out cases for promotion and final confirmation.
-- acknowledge every candidate validation command with
-  `unsafeHostExecution: true`; validators must be observational and must not
-  mutate the candidate tree.
-
-A minimal current agent campaign has this shape (selectors are objects, mutable
-paths are candidate-relative globs, and budgets use call/token/cost limits):
-
-```ts
-export default defineExperiment({
-  id: "skill-improvement",
-  subject: { root: "..", mutable: ["skills/example/**"] },
-  profiles: ["claude-low", "codex-low"],
-  splits: {
-    train: { tags: ["train"] },
-    validation: { tags: ["validation"] },
-    test: { tags: ["test"] },
-  },
-  repetitions: { search: 1, promotion: 3 },
-  objectives: {
-    constraints: [
-      { metric: "quality.caseRegressions", max: 0 },
-      { metric: "quality.passRate", min: 0.8, split: "test" },
-    ],
-    minimize: [{ metric: "agent.total" }, { metric: "subject.bytes" }],
-  },
-  optimizer: {
-    algorithm: "reflective",
-    candidateSelection: "pareto",
-    promotionTopK: 2,
-    config: {
-      provider: "codex",
-      model: "YOUR_OPTIMIZER_MODEL",
-      reasoningEffort: "low",
-    },
-  },
-  budget: {
-    maxCandidates: 4,
-    patience: 1,
-    maxMetricCalls: 60,
-    maxJudgeCalls: 120,
-    maxProposalCalls: 3,
-    maxTokens: 2_000_000,
-    maxCostUsd: 30,
-    maxUnpricedModelCalls: 0,
-    maxWallTimeMs: 3_600_000,
-  },
-});
-```
-
-Use the experiment directory printed by the run for `view` and `report`.
-Review its held-out evidence, then pass that same verified directory to
-`apply`; do not guess a directory from the campaign ID.
-
-Use `experiment` for authored variants and `optimize` for model-proposed source
-changes. Always run the corresponding `--plan` command first.

package/src/cli/skills.ts

@@ -1,129 +0,0 @@
-import { lstat, mkdir, readdir, readFile, writeFile } from "node:fs/promises";
-import { homedir } from "node:os";
-import { dirname, join, relative, resolve, sep } from "node:path";
-import { fileURLToPath } from "node:url";
-import { z } from "zod";
-
-const PlatformSchema = z.enum(["claude", "claude-code", "codex", "all"]);
-export type SkillsPlatform = z.infer<typeof PlatformSchema>;
-const SkillNameSchema = z.enum(["lux", "lux-answerer"]);
-export type BundledSkillName = z.infer<typeof SkillNameSchema>;
-
-const InstallSkillsOptionsSchema = z
-  .object({
-    cwd: z.string().min(1),
-    platform: PlatformSchema,
-    project: z.boolean().default(false),
-    dir: z.string().min(1).optional(),
-    name: SkillNameSchema.optional(),
-  })
-  .strict();
-
-export interface InstalledSkill {
-  name: BundledSkillName;
-  platform: "claude-code" | "codex";
-  path: string;
-}
-
-const BUNDLED_SKILLS = [
-  {
-    name: "lux",
-    description: "Author and drive coding-agent evals with Lux",
-    platforms: ["claude-code", "codex"],
-  },
-  {
-    name: "lux-answerer",
-    description: "Play a simulated user for a running Lux eval",
-    platforms: ["claude-code"],
-  },
-] as const satisfies readonly {
-  name: BundledSkillName;
-  description: string;
-  platforms: readonly ("claude-code" | "codex")[];
-}[];
-
-export const formatSkillsList = (): string =>
-  BUNDLED_SKILLS.map(
-    ({ name, description, platforms }) => `${name}\t${description}\t${platforms.join(",")}`,
-  ).join("\n");
-
-const targetsFor = (
-  platform: SkillsPlatform,
-  cwd: string,
-  project: boolean,
-  dir?: string,
-  name?: BundledSkillName,
-): (InstalledSkill & { source: string })[] => {
-  const platforms =
-    platform === "all"
-      ? (["claude-code", "codex"] as const)
-      : ([platform === "claude" ? "claude-code" : platform] as const);
-  const scopeRoot = project ? resolve(cwd) : homedir();
-  const skills = name ? BUNDLED_SKILLS.filter((skill) => skill.name === name) : BUNDLED_SKILLS;
-  return platforms.flatMap((platformName) =>
-    skills
-      .filter((skill) =>
-        (skill.platforms as readonly ("claude-code" | "codex")[]).includes(platformName),
-      )
-      .map((skill) => ({
-        name: skill.name,
-        platform: platformName,
-        source: fileURLToPath(new URL(`../../skills/${skill.name}`, import.meta.url)),
-        path: dir
-          ? join(resolve(dir), skill.name)
-          : join(
-              scopeRoot,
-              platformName === "claude-code" ? ".claude" : ".codex",
-              "skills",
-              skill.name,
-            ),
-      })),
-  );
-};
-
-const copySkillDirectory = async (source: string, target: string): Promise<void> => {
-  const sourceInfo = await lstat(source);
-  if (!sourceInfo.isDirectory() || sourceInfo.isSymbolicLink()) {
-    throw new Error(`lux skills: bundled skill source is not a real directory: ${source}`);
-  }
-  await mkdir(target, { recursive: true });
-  for (const entry of await readdir(source, { withFileTypes: true })) {
-    const sourcePath = join(source, entry.name);
-    const targetPath = join(target, entry.name);
-    if (entry.isSymbolicLink()) {
-      throw new Error(`lux skills: bundled skill contains a symbolic link: ${sourcePath}`);
-    }
-    if (entry.isDirectory()) {
-      await copySkillDirectory(sourcePath, targetPath);
-    } else if (entry.isFile()) {
-      await mkdir(dirname(targetPath), { recursive: true });
-      await writeFile(targetPath, await readFile(sourcePath));
-    } else {
-      throw new Error(`lux skills: bundled skill contains an unsupported entry: ${sourcePath}`);
-    }
-  }
-};
-
-export const installSkills = async (
-  options: z.input<typeof InstallSkillsOptionsSchema>,
-): Promise<InstalledSkill[]> => {
-  const parsed = InstallSkillsOptionsSchema.parse(options);
-  const targets = targetsFor(parsed.platform, parsed.cwd, parsed.project, parsed.dir, parsed.name);
-  if (targets.length === 0) {
-    throw new Error(
-      `lux skills: '${parsed.name}' is not available for platform '${parsed.platform}'`,
-    );
-  }
-  for (const target of targets) await copySkillDirectory(target.source, target.path);
-  return targets.map(({ source: _, ...installed }) => installed);
-};
-
-export const formatSkillsInstall = (installed: InstalledSkill[], cwd: string): string =>
-  installed
-    .map(
-      ({ name, platform, path }) =>
-        `installed ${name} skill for ${platform} at ${
-          relative(cwd, path).split(sep).join("/") || "."
-        }`,
-    )
-    .join("\n");