@prismatic-io/lux 0.0.1

package/package.json

@@ -0,0 +1,84 @@
+{
+  "name": "@prismatic-io/lux",
+  "version": "0.0.1",
+  "description": "Coding-agent evaluation and improvement with deterministic assertions, optional LLM judges, and first-class human-in-the-loop runs.",
+  "license": "MIT",
+  "homepage": "https://github.com/prismatic-io/lux",
+  "bugs": {
+    "url": "https://github.com/prismatic-io/lux/issues"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/prismatic-io/lux.git",
+    "directory": "packages/lux"
+  },
+  "type": "module",
+  "keywords": [
+    "evals",
+    "evaluation",
+    "agent",
+    "coding-agent",
+    "human-in-the-loop",
+    "claude",
+    "claude-code",
+    "codex",
+    "openai",
+    "llm-judge",
+    "testing"
+  ],
+  "sideEffects": false,
+  "engines": {
+    "node": ">=22.18.0"
+  },
+  "main": "./lib/index.js",
+  "types": "./lib/index.d.ts",
+  "bin": {
+    "lux": "./lib/cli/bin.js"
+  },
+  "exports": {
+    ".": {
+      "types": "./lib/index.d.ts",
+      "default": "./lib/index.js"
+    },
+    "./artifact-snapshot": {
+      "types": "./lib/core/artifact-snapshot.d.ts",
+      "default": "./lib/core/artifact-snapshot.js"
+    },
+    "./authoring": {
+      "types": "./lib/authoring.d.ts",
+      "default": "./lib/authoring.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "files": [
+    "lib",
+    "skills",
+    "src",
+    "!**/*.test.ts",
+    "!**/test-helpers.ts",
+    "!src/**/fixtures/**"
+  ],
+  "scripts": {
+    "build": "tsc --build",
+    "clean": "node ../../scripts/clean.mjs"
+  },
+  "devDependencies": {
+    "@arethetypeswrong/cli": "0.18.4",
+    "publint": "0.3.21"
+  },
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.30.0",
+    "@vscode/windows-process-tree": "0.8.0",
+    "commander": "^15.0.0",
+    "js-yaml": "^5.2.2",
+    "oxc-parser": "^0.142.0",
+    "tempy": "^3.2.0",
+    "tinyexec": "1.2.4",
+    "tinyglobby": "^0.2.17",
+    "xstate": "5.32.0",
+    "zod": "^4.4.3"
+  }
+}

package/skills/lux/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: lux
+description: Author, validate, run, inspect, compare, and improve coding-agent evaluations with the Lux CLI. Use when creating Lux cases or campaigns, choosing assertions/personas/drivers, diagnosing eval failures, comparing runs, or running Lux experiments.
+---
+
+# Lux
+
+Use Lux as an eval test runner and evidence system for coding agents. Prefer
+small, discriminating cases with deterministic assertions; add an LLM rubric
+only for qualities that artifacts and commands cannot establish.
+
+## Start here
+
+1. Inspect the repository and its existing `lux.config.ts`, cases, fixtures,
+   prompts, and experiments before adding files.
+2. If Lux is not initialized, run `npm exec -- lux init` for ordinary evals or
+   `npm exec -- lux init --experiment` for an improvement campaign.
+3. Read [references/eval-authoring.md](references/eval-authoring.md) before
+   authoring or reviewing a case.
+4. Read [references/cli.md](references/cli.md) before running, diagnosing,
+   comparing, or promoting results.
+5. Run `npm exec -- lux doctor` before model calls. Treat its errors as
+   authoring problems, not as permission to bypass validation.
+
+## Workflow
+
+### Author an eval
+
+- Define the behavior and likely failure mode first.
+- Make the prompt realistic and self-contained.
+- Give the persona only facts and preferences the simulated user should know.
+- Assert observable outcomes, not implementation details.
+- Use stable assertion IDs so grading evidence remains intelligible.
+- Use the cheapest deterministic assertion that proves each requirement.
+- Include a rubric assertion only when semantic judgment is unavoidable, and
+  write one narrow, falsifiable criterion per assertion.
+- Pin the subject driver model explicitly when comparability matters.
+
+### Run and diagnose
+
+```bash
+npm exec -- lux doctor
+npm exec -- lux run <case-or-filter> --loop 3
+npm exec -- lux view <run-or-suite>
+```
+
+Inspect `case.json`, `run.json`, `events.jsonl`, `grading.json`, and `artifacts/`
+before changing the case. Separate subject failure from harness failure:
+
+- Subject failure: the agent misunderstood, asked poorly, used the wrong tools,
+  or left incorrect artifacts.
+- Harness failure: the prompt was underspecified, the persona lacked a needed
+  fact, an assertion measured a proxy, or the judge criterion was ambiguous.
+
+Do not weaken a valid assertion merely to make a run pass.
+
+### Improve with evidence
+
+Use repeated runs and held-out splits for noisy agent behavior. For skill,
+prompt, or agent-source changes, use `subjectPath()` in cases and an experiment
+campaign with immutable candidates. Plan first, run the experiment, inspect the
+report, and apply a promoted candidate as a separate action:
+
+```bash
+npm exec -- lux experiment experiments/example.experiment.ts --plan
+npm exec -- lux experiment experiments/example.experiment.ts
+npm exec -- lux report <experiment-dir>
+npm exec -- lux apply <experiment-dir>
+```
+
+Never edit a materialized candidate or managed experiment record by hand.
+
+## Guardrails
+
+- Lux eval projects execute project-authored commands and agent-generated work.
+  Run untrusted evals only in a disposable environment.
+- Do not infer success from agent prose. Grade the final artifacts and recorded
+  behavior.
+- Do not use only `run-succeeded` for a case that claims to verify a concrete
+  result.
+- Do not let train examples leak into validation or test splits.
+- Do not combine evidence across changed models, harnesses, cases, or subjects;
+  Lux identity checks exist to prevent that.
+- Prefer `--plan`, `doctor`, `view`, and `report` before expensive or mutating
+  commands.

package/skills/lux/references/cli.md

@@ -0,0 +1,102 @@
+# Lux CLI runbook
+
+Invoke the project-local CLI with `npm exec -- lux ...` unless the repository
+uses another package-manager convention.
+
+## Author and preflight
+
+```bash
+lux init
+lux init --experiment
+lux doctor
+lux doctor experiments/campaign.experiment.ts --json
+lux run --list
+lux run --list-tags
+```
+
+`init` is conservative around existing files. `doctor` validates configuration,
+case discovery, driver executables, campaign splits, confidence feasibility,
+and projected budgets without model calls.
+
+## Run
+
+```bash
+lux run
+lux run retry-flag
+lux run cases/retry-flag.ts
+lux run --tag smoke
+lux run retry --loop 5 --concurrency 2
+lux run one-case --interactive
+```
+
+Filters match case IDs and paths. Tags come from `meta.tags`. A run invocation
+also creates a suite record, which is the right unit for repeated or multi-case
+comparison.
+
+## Inspect and compare
+
+```bash
+lux view
+lux view <run-or-suite>
+lux view --experiments
+lux compare <run-or-suite-a> <run-or-suite-b>
+```
+
+Use `view` before opening raw files. When diagnosing, inspect:
+
+- `case.json`: frozen authored inputs;
+- `run.json`: exit reason, timing, usage, source identity;
+- `events.jsonl`: trajectory, tools, and interrupts;
+- `grading.json`: assertion evidence;
+- `artifacts/`: the subject workspace output.
+
+`compare` exits nonzero on a gated regression, making it suitable for CI.
+
+## Regrade and annotate
+
+```bash
+lux grade <run>
+lux grade <run> --case cases/changed-grader.ts
+lux annotate <run> \
+  --out evals/annotations.json \
+  --rubric evals/rubrics/case.json \
+  --label pass
+```
+
+Regrading preserves subject evidence and changes only grading. Human annotations
+support grader-alignment experiments; keep assertion IDs aligned with the
+rubric and record reviewer context where available.
+
+## Experiment lifecycle
+
+```bash
+lux experiment <campaign> --plan
+lux experiment <campaign>
+lux optimize <campaign> --plan
+lux optimize <campaign>
+lux optimize <campaign> --resume <experiment-dir>
+lux view <experiment-dir>
+lux report <experiment-dir> --out .lux-runs/reports/decision.md
+lux apply <experiment-dir>
+```
+
+`experiment` compares authored variants. `optimize` searches proposed changes.
+Both create durable, resumable, integrity-checked records. `report` re-verifies
+the ledger and promotion decision. `apply` is deliberately separate and
+hash-guards the source it changes.
+
+Use the exact resume, report, and apply commands printed by Lux. Do not edit
+`.lux-runs/.experiments/` manually.
+
+## Failure triage
+
+- Discovery/config error: run `doctor`, verify paths and exported definitions.
+- Driver unavailable: verify the selected CLI is installed and the model is
+  explicitly configured where required.
+- Agent exit/error: inspect `events.jsonl` and the driver section of `run.json`.
+- Assertion failure: inspect the assertion evidence and final artifact; decide
+  whether the subject or measurement is wrong.
+- Judge disagreement: narrow the rubric, regrade stored evidence, and collect
+  human annotations before changing the subject.
+- Interrupted experiment: resume the printed experiment directory; do not start
+  a nominally identical campaign and combine results yourself.

package/skills/lux/references/eval-authoring.md

@@ -0,0 +1,105 @@
+# Authoring useful Lux evals
+
+## Case shape
+
+A case default-exports `defineEvalCase`:
+
+```ts
+import { defineEvalCase } from "@prismatic-io/lux";
+
+export default defineEvalCase({
+  id: "retry-flag",
+  prompt: "Add a --retry <n> flag to the fetch command.",
+  persona: "You are the maintainer. If asked, retries default to 3.",
+  driver: { name: "codex", config: { model: "YOUR_CODEX_MODEL" } },
+  assertions: [
+    { id: "flag-present", type: "file-contains", path: "src/cli.ts", text: "--retry" },
+    { id: "tests-pass", type: "command-exits-zero", command: "npm test" },
+    { id: "test-coverage", type: "rubric", criteria: "A test covers the new retry flag." },
+  ],
+  meta: { tags: ["cli", "validation"] },
+});
+```
+
+Use the project-level default driver when all cases share it. Case-level
+drivers are useful for intentional cross-agent or cross-model comparisons.
+
+## Design checklist
+
+Before writing:
+
+1. State the capability or regression in one sentence.
+2. Identify the artifact, command result, or interaction that proves it.
+3. List plausible false positives. Adjust assertions to reject them.
+4. Decide whether the case needs a fixture and whether each run must begin from
+   an isolated copy.
+5. Identify any question a competent agent must ask. Put only the answer and
+   relevant preferences in the persona.
+
+After writing:
+
+1. Run `lux doctor`.
+2. List discovery with `lux run <filter> --list`.
+3. Run the case more than once if agent behavior can vary.
+4. Inspect events and artifacts, including failed runs.
+5. Confirm each assertion fails when its requirement is deliberately absent.
+
+## Assertion selection
+
+Prefer deterministic assertions:
+
+- `run-succeeded`: the driver completed; necessary but rarely sufficient.
+- `file-exists`: a required artifact exists.
+- `file-contains`: a stable literal is present.
+- `json-path-equals`: structured output contains the expected value.
+- `command-exits-zero`: project-authored validation succeeds.
+
+Use `rubric` for semantic properties such as design quality, completeness, or
+whether a test meaningfully covers behavior. Keep criteria narrow. Avoid words
+like “good,” “proper,” or “best practice” without observable conditions.
+
+Assertions should test user-visible behavior and durable contracts. Avoid
+requiring a specific function name, file layout, or algorithm unless that is
+the contract under evaluation.
+
+## Human-in-the-loop behavior
+
+The persona is simulated user state, not another system prompt. Include:
+
+- facts the user would know;
+- preferences needed to resolve legitimate ambiguity;
+- approval boundaries relevant to the task.
+
+Do not tell the persona how to help the agent solve the task. If no interaction
+is part of the capability being evaluated, omit the persona or keep it minimal.
+
+Use `interactionMode: "defer-resume"` for automated Claude Code question-tool
+handling. Use `textQuestionFallback: true` only when intentionally evaluating
+models that ask in prose instead of using the supported tool.
+
+## Experiments for prompts, skills, and agent source
+
+Bind mutable source through `subjectPath()`:
+
+```ts
+import { defineEvalCase, subjectPath } from "@prismatic-io/lux";
+
+export default defineEvalCase({
+  id: "skill-routing",
+  prompt: `Use the skill at ${subjectPath("plugin/skills/routing/SKILL.md")} to complete the task.`,
+  assertions: [{ id: "completed", type: "run-succeeded" }],
+  meta: { tags: ["validation"] },
+});
+```
+
+Campaigns should:
+
+- restrict `subject.mutable` to the intended source;
+- define named, explicit driver/model profiles;
+- keep train, validation, and test selectors disjoint;
+- budget calls, tokens, cost, and wall time;
+- use repetitions and confidence gates appropriate to stochastic outcomes;
+- reserve held-out cases for promotion and final confirmation.
+
+Use `experiment` for authored variants and `optimize` for model-proposed source
+changes. Always run the corresponding `--plan` command first.

package/skills/lux-answerer/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: lux-answerer
+version: 0.0.1
+description: Play the persona for a running Lux orchestrator that uses the claude-code answerer. Read structured events on stdout, decide answers from persona + context, write JSON to its answer channel.
+user-invocable: false
+allowed-tools: Bash, Read
+---
+
+# Lux answerer
+
+This skill is the runbook for driving a Lux orchestrator from a
+Claude Code session. The orchestrator's claude-code answerer emits
+structured events on its stdout; you read them, decide an answer in
+character, and write the answer to the channel it provides. Lux uses a FIFO on
+Linux and macOS and a regular answer file on Windows.
+
+## TL;DR
+
+1. Find the orchestrator's stdout — usually it's a Bash process you
+   started, so its output is right there in the conversation.
+2. The first interesting event is `lux.answerer.ready`. **Capture
+   `fifoPath`, `transport`, `persona`, and `prompt` from it.** Commit to the
+   persona for the rest of the run.
+3. For each `lux.answerer.question` event:
+   - Read the embedded `interrupt` (kind, id, question, context).
+   - Decide the answer **strictly from persona + question + context.**
+   - Write one newline-terminated JSON answer to `<fifoPath>` using the
+     event's `transport`.
+4. The orchestrator finishes and exits when the agent reports done.
+
+## 1. The wire protocol
+
+### Events you receive (orchestrator → you, on stdout)
+
+| Type | Payload | Action |
+|------|---------|--------|
+| `lux.answerer.ready` | `{ fifoPath, transport, persona, prompt, runDir }` | Capture the answer path, transport, and persona. |
+| `lux.answerer.question` | `{ interrupt, historyDepth }` | Decide an answer; write to `fifoPath`. |
+
+The `interrupt` field is the standard Lux Interrupt:
+
+```ts
+type Interrupt =
+  | { kind: "ask";     id: string; question: string; context?: unknown }
+  | { kind: "approve"; id: string; request: string;   context?: unknown }
+  | { kind: "custom";  id: string; payload: unknown;  context?: unknown };
+```
+
+### Commands you send (you → orchestrator, written to the answer channel)
+
+One JSON object per line. Newline required. Always echo the
+interrupt's `id` so the answerer can verify the reply pairs with the
+question it is waiting on.
+
+For `ask` interrupts:
+```
+{"id": "<interrupt id>", "text": "<your answer>"}
+```
+
+For `approve` interrupts:
+```
+{"id": "<interrupt id>", "approved": true, "reason": "<optional reason>"}
+```
+
+For `custom` interrupts:
+```
+{"id": "<interrupt id>", "payload": <whatever shape the interrupt expects>}
+```
+
+### Examples
+
+**Answer on a FIFO** (`transport: "fifo"`, from Bash):
+
+```
+printf '%s\n' '{"id":"q-1","text":"Use API key authentication."}' > "$LUX_FIFO"
+```
+
+**Answer in a regular file** (`transport: "file"`, portable Node command):
+
+```
+node -e "require('node:fs').writeFileSync(process.argv[1], process.argv[2] + '\n')" "$LUX_FIFO" '{"id":"q-1","text":"Use API key authentication."}'
+```
+
+The same JSON shapes apply to approvals:
+
+```
+{"id":"a-1","approved":true,"reason":"plan looks correct"}
+{"id":"a-1","approved":false,"reason":"step 3 should use Bash, not Code"}
+```
+
+## 2. Decision-making during a run
+
+The persona is the single source of truth. When deciding an answer:
+
+1. **Read the persona first.** It defines who you are and what you
+   want out of this run. Stay in character.
+2. **Consult the embedded `context`.** Drivers attach what they
+   think you need — recent assistant text, recent tool calls. This
+   usually disambiguates ambiguous questions.
+3. **If the persona supplies an explicit value** for the question
+   (a URL, an auth method, a config var name), use it verbatim.
+4. **If the persona is silent** on a detail, choose a value
+   consistent with the persona's role and the run's overall goal.
+   Don't fabricate contradictory specifics; don't ask the agent for
+   help (you ARE the human in this loop).
+5. **Keep answers concise.** One line where possible. Long answers
+   slow the run down and add noise.
+
+## 3. Critical rules
+
+- **Always terminate the JSON with a newline.** For a FIFO, use `printf`,
+  never `echo`. For a regular file, overwrite the file with one complete
+  newline-terminated answer.
+- **Single-quote the JSON** in shell, double-quote inside the JSON.
+  If the answer text contains a literal single quote, escape via
+  `'\''` or rephrase.
+- **Echo the interrupt id in your reply.** A reply whose `id` does
+  not match the interrupt currently awaiting an answer is silently
+  discarded — this protects the run from a late reply to an already
+  timed-out question being misread as the answer to the next one.
+  A reply without an `id` is paired with the most-recent outstanding
+  interrupt, so omitting it forfeits that protection.
+- **Don't pre-commit answers.** Wait for the structured event;
+  don't guess what's coming next.
+
+## 4. Failure modes and recovery
+
+### Answer wait timed out (no event is emitted)
+
+If no reply lands on the answer channel within the answerer's wait budget
+(`maxAnswerWaitMs`, default 10 minutes), the pending answer is
+rejected and the run fails — there is no further stdout event to
+watch for. Separately, the driver can hit its own idle window if the
+agent stalls mid-LLM-call. Either way, inspect the run dir
+(`<runDir>/events.jsonl`) to see how far the run got.
+
+### Wrong answer shape
+
+If the answerer rejects your answer (e.g. you sent `{"approved":...}`
+to an `ask` interrupt), it logs an error and the run fails. Re-read
+the interrupt kind before composing your reply.
+
+### Answer write hangs
+
+A FIFO write can block when the orchestrator has already exited. A regular-file
+answer may be ignored after the question has timed out. Check whether the
+orchestrator process is still running.
+
+## 5. After the run
+
+The run dir contains everything captured:
+
+- `case.json` — frozen prompt + persona used for this run.
+- `run.json` — exit reason, duration, interrupt counts, tokens.
+- `events.jsonl` — every event the orchestrator processed.
+- `grading.json` — per-assertion results.
+- `artifacts/` — files the agent wrote.
+
+Use `/lux:view <runDir>` for a formatted summary.
+
+## 6. What you cannot do from this skill
+
+- **Pause the agent.** Once an interrupt is resolved, the agent
+  runs until the next interrupt or `done`. There is no
+  pause/resume primitive.
+- **Revise a resolved interrupt.** Once a command is consumed, the
+  interrupt is gone. If your answer was wrong, the only recovery is
+  to wait for a follow-up question (if any) and course-correct
+  there.