@prismatic-io/lux 0.0.2-preview.18 → 0.0.2-preview.21

package/skills/lux/references/eval-authoring.md

@@ -1,266 +0,0 @@
-# Authoring useful Lux evals
-
-## Case shape
-
-A case default-exports `defineEvalCase`:
-
-```ts
-import { defineEvalCase } from "@prismatic-io/lux";
-
-export default defineEvalCase({
-  id: "retry-flag",
-  prompt: "Add a --retry <n> flag to the fetch command.",
-  persona: "You are the maintainer. If asked, retries default to 3.",
-  driver: { name: "codex", config: { model: "YOUR_CODEX_MODEL" } },
-  assertions: [
-    { id: "flag-present", type: "file-contains", path: "src/cli.ts", text: "--retry" },
-    { id: "tests-pass", type: "command-exits-zero", command: "npm test" },
-    { id: "test-coverage", type: "rubric", criteria: "A test covers the new retry flag." },
-  ],
-  meta: { tags: ["cli", "validation"] },
-});
-```
-
-Use the project-level default driver when all cases share it. Case-level
-drivers are useful for intentional cross-agent or cross-model comparisons.
-
-## Design checklist
-
-Before writing:
-
-1. State the capability or regression in one sentence.
-2. Identify the artifact, command result, or interaction that proves it.
-3. List plausible false positives. Adjust assertions to reject them.
-4. Decide whether the case needs a fixture and whether each run must begin from
-   an isolated copy.
-5. Identify any question a competent agent must ask. Put only the answer and
-   relevant preferences in the persona.
-
-After writing:
-
-1. Run `lux doctor`.
-2. List discovery with `lux run <filter> --list`.
-3. Run the case more than once if agent behavior can vary.
-4. Inspect events and artifacts, including failed runs.
-5. Confirm each assertion fails when its requirement is deliberately absent.
-
-## Assertion selection
-
-Prefer deterministic assertions:
-
-- `run-succeeded`: the driver completed; necessary but rarely sufficient.
-- `file-exists`: a required artifact exists.
-- `file-contains`: a stable literal is present.
-- `json-pointer-equals`: structured output contains the expected value.
-- `command-exits-zero`: project-authored validation succeeds.
-
-Use the more specific bundled evidence checks when they match the contract:
-
-- Files: `file-matches`, `file-regex-set`, `file-regex-count`, and `glob-count`.
-- Output/data: `contains`, `not-contains`, `regex`, and `json-pointer-equals`.
-- Interaction: `interrupt-count`, `tool-called`, `tool-called-with`, and
-  `tool-call-count`; use `by: "main"` or `by: "subagent"` when ownership matters.
-- Paired tool evidence: `tool-invocation`, `tool-result-contains`,
-  `tool-result-json-pointer-equals`, `tool-result-is-error`, and
-  `no-error-results`. Prefer these when result status or payload matters; a tool
-  call without a paired result is unresolved, not successful.
-- Budgets: `latency-under`, `time-to-first-action-under`, `tokens-under`,
-  `cost-under`, and `artifact-bytes-under`. Time to first action requires the
-  driver to emit semantic `action` progress events; use `category: "mutation"`
-  to exclude queries and outer code-mode executor calls.
-- MCP discovery: `server-info-equals`, `tool-listed`,
-  `tools-listed-count`, and `capability-absent`.
-
-Most file assertions read captured `artifacts`; set `where: "cwd"` only when
-the durable contract intentionally includes the staged run workspace. Use
-stable assertion `id` values for repetitions, comparisons, annotations, and
-experiments. `weight` affects aggregate score and `threshold` sets an optional
-per-assertion score floor.
-
-Use `rubric` for semantic properties such as design quality, completeness, or
-whether a test meaningfully covers behavior. Keep criteria narrow. Avoid words
-like “good,” “proper,” or “best practice” without observable conditions.
-
-Bound rubric evidence to what the criterion needs. Include `tool-result` event
-evidence when status matters, and request tool inputs/results only when their
-payload is necessary; this controls judge context and cost.
-
-Persona answering and rubric judging have different difficulty profiles. A
-persona usually chooses from facts already supplied in the case, so prefer a
-cheap model and low effort. Semantic grading should use a stronger model and
-enough effort to distinguish real evidence from plausible prose. Configure
-them independently when that tradeoff matters:
-
-```ts
-// lux.config.ts
-defaultAnswerer: {
-  name: "persona",
-  config: {
-    provider: "codex",
-    model: "YOUR_SMALL_PERSONA_MODEL",
-    reasoningEffort: "low",
-  },
-},
-
-// case assertion
-{
-  id: "behavior-quality",
-  type: "rubric",
-  criteria: "The tests exercise retry exhaustion and recovery.",
-  provider: "codex",
-  model: "YOUR_STRONG_JUDGE_MODEL",
-  reasoningEffort: "high",
-}
-```
-
-The project `harness` supplies shared defaults to persona and rubric calls;
-answerer config and rubric fields override it independently. Always pin model
-and effort when comparing runs, and inspect answerer versus judge usage in the
-run summary before increasing either budget.
-
-Assertions should test user-visible behavior and durable contracts. Avoid
-requiring a specific function name, file layout, or algorithm unless that is
-the contract under evaluation.
-
-## Human-in-the-loop behavior
-
-The persona is simulated user state, not another system prompt. Include:
-
-- facts the user would know;
-- preferences needed to resolve legitimate ambiguity;
-- approval boundaries relevant to the task.
-
-Do not tell the persona how to help the agent solve the task. If no interaction
-is part of the capability being evaluated, omit the persona or keep it minimal.
-
-For Claude Code, use `interactionMode: "defer-resume"` for automated
-question-tool handling. Use `textQuestionFallback: true` only when
-intentionally evaluating models that ask in prose instead of using the
-supported tool. For Codex, keep the default `interactionMode: "app-server"`
-when the case can interrupt; compatibility mode `"exec"` cannot accept
-mid-run responses.
-
-Use `--interactive` for a real terminal user or `--claude-answerer` for the
-Claude plugin protocol. Both force sequential execution.
-
-To count only questions about specific concepts, set `kind: "ask"` with
-`questionIncludesAll`, `questionIncludesAny`, or `questionExcludesAny` on
-`interrupt-count`. Matching is case-insensitive and ignores separators, so
-terms such as `"first name"` also match `"firstName"`. Filters compose: the
-question must include every `questionIncludesAll` term, at least one
-`questionIncludesAny` term, and no `questionExcludesAny` terms.
-
-## Project extensions
-
-For a reusable project-specific assertion, define it with `defineAssertion`
-and a direct Zod object whose `type` is a matching literal. Import `z` from Lux,
-return an ordinary failed result for missing/invalid product evidence, and
-reserve throws or `errorMessage` for evaluator faults. Increment
-`pluginVersion` whenever schema or evaluation behavior changes. Prefer the
-bounded, traversal-safe `readRunArtifactText` helper over direct filesystem
-reads.
-
-Build a typed case authoring function once instead of importing plugins in
-every case:
-
-```ts
-const assertions = defineAssertionRegistry(projectAssertion);
-export const defineEvalCase = defineLuxCase.extend({ assertions });
-```
-
-Case-bound plugins remain plain serialized plans but require the current case
-path for regrading: `lux grade <run> --case <case.ts>`.
-
-Use `defineFixtures` for Vitest-style suite/case setup around `await use()` or
-`defineLifecycleHooks` when cleanup-returning callbacks fit better. Setup runs
-in declaration order and teardown in reverse, including failures. Call
-`await use()` exactly once. Fixture `value`/`identity` must be public JSON;
-keep secrets in the prepared resource or environment. Increment
-`pluginVersion` when lifecycle behavior changes. Register globally in
-`lux.config.ts`, or attach selectively with
-`defineEvalCase.extend({ fixtures: [...] })`.
-
-Custom drivers that own an in-memory filesystem should persist it with
-`writeArtifactSnapshot` from `@prismatic-io/lux/artifact-snapshot`. Snapshots
-are bounded, traversal-safe, immutable per name, and can use `baseSnapshot`
-for copy-on-write reuse while remaining independently complete. Ordinary file
-assertions then grade the indexed files without a snapshot-specific API.
-
-## Experiments for prompts, skills, and agent source
-
-Bind mutable source through `subjectPath()`:
-
-```ts
-import { defineEvalCase, subjectPath } from "@prismatic-io/lux";
-
-export default defineEvalCase({
-  id: "skill-routing",
-  prompt: `Use the skill at ${subjectPath("plugin/skills/routing/SKILL.md")} to complete the task.`,
-  assertions: [{ id: "completed", type: "run-succeeded" }],
-  meta: { tags: ["validation"] },
-});
-```
-
-Campaigns should:
-
-- restrict `subject.mutable` to the intended source;
-- define named, explicit driver/model profiles;
-- keep train, validation, and test selectors disjoint;
-- budget calls, tokens, cost, and wall time;
-- use repetitions and confidence gates appropriate to stochastic outcomes;
-- reserve held-out cases for promotion and final confirmation.
-- acknowledge every candidate validation command with
-  `unsafeHostExecution: true`; validators must be observational and must not
-  mutate the candidate tree.
-
-A minimal current agent campaign has this shape (selectors are objects, mutable
-paths are candidate-relative globs, and budgets use call/token/cost limits):
-
-```ts
-export default defineExperiment({
-  id: "skill-improvement",
-  subject: { root: "..", mutable: ["skills/example/**"] },
-  profiles: ["claude-low", "codex-low"],
-  splits: {
-    train: { tags: ["train"] },
-    validation: { tags: ["validation"] },
-    test: { tags: ["test"] },
-  },
-  repetitions: { search: 1, promotion: 3 },
-  objectives: {
-    constraints: [
-      { metric: "quality.caseRegressions", max: 0 },
-      { metric: "quality.passRate", min: 0.8, split: "test" },
-    ],
-    minimize: [{ metric: "agent.total" }, { metric: "subject.bytes" }],
-  },
-  optimizer: {
-    algorithm: "reflective",
-    candidateSelection: "pareto",
-    promotionTopK: 2,
-    config: {
-      provider: "codex",
-      model: "YOUR_OPTIMIZER_MODEL",
-      reasoningEffort: "low",
-    },
-  },
-  budget: {
-    maxCandidates: 4,
-    patience: 1,
-    maxMetricCalls: 60,
-    maxJudgeCalls: 120,
-    maxProposalCalls: 3,
-    maxTokens: 2_000_000,
-    maxCostUsd: 30,
-    maxUnpricedModelCalls: 0,
-    maxWallTimeMs: 3_600_000,
-  },
-});
-```
-
-Use the experiment directory printed by the run for `view` and `report`.
-Review its held-out evidence, then pass that same verified directory to
-`apply`; do not guess a directory from the campaign ID.
-
-Use `experiment` for authored variants and `optimize` for model-proposed source
-changes. Always run the corresponding `--plan` command first.