@dungle-scrubs/skillval 0.2.0 → 0.3.0

package/README.md

@@ -6,6 +6,19 @@ agent behavior instead of merely checking whether the final answer looks accepta
 baseline also passes, the rule is flagged as a no-op and a possible prune candidate. You can only
 trust what you test.
 
+## Capability and preference rules
+
+A skill carries two kinds of rule. **Capability** rules teach a model something it does not yet
+reliably do. **Preference** rules express a choice - style, convention, house taste - a model would
+not reach on its own. Most skills mix both. The distinction has teeth because capabilities expire:
+as models are trained on the same information, a capability rule stops changing behavior and turns
+into dead weight. skillval finds those.
+
+Each case runs with the skill and again without it (the baseline arm). Skill-pass with
+baseline-fail means the rule is load-bearing. Skill-pass with baseline-pass means the model already
+does this on its own - the rule is a prune candidate. Preferences stay; stale capabilities go.
+Cases can record which kind they exercise with the `type` field (`capability` or `preference`).
+
 ## Install
 
 ```sh
@@ -64,6 +77,14 @@ to select one case, `--no-cache` to ignore cached arm results, `--skip-baseline`
 arms, and `--json` for the complete report. The command exits with status 1 when any selected
 case's skill arm fails.
 
+Use `--model <model>` and `--effort <level>` to pin the executor's model and effort for the run,
+so you can evaluate one skill under, for example, `--model sonnet --effort medium`. Both pass
+through to the configured executor and are recorded in the report and the cache identity, so runs
+at different levels are cached and compared separately. Effort levels are executor-specific and
+validated before the run: `codex` accepts `none, minimal, low, medium, high, xhigh, max`; `claude`
+accepts `low, medium, high, xhigh, max`; `pi` accepts `off, minimal, low, medium, high, xhigh`.
+Model support for a given effort is a subset of these, enforced by the harness itself.
+
 ## Configuration
 
 The configuration follows the [configuration JSON Schema](schemas/config.schema.json):
@@ -76,7 +97,7 @@ executor: codex
 ```
 
 `roots` contains directories whose immediate children have the form `<skill>/SKILL.md`. Both `~`
-and `$HOME` are expanded. Missing roots are skipped during `run`; `list` returns them in
+and `$HOME` are expanded. `executor` selects the trial adapter: `codex`, `claude`, or `pi`. Missing roots are skipped during `run`; `list` returns them in
 `missingRoots` with JSON output and prints each as `missing root: <path>` in human output.
 
 Configuration path precedence is:
@@ -92,12 +113,31 @@ There is no legacy `~/.skillval` lookup. State uses `$XDG_STATE_HOME/skillval`,
 - `cache/` stores arm results.
 - `reports/` stores run reports named by a hash of the participating skills and their content
   hashes. Each report also includes every participating skill's content hash and the executor's
-  name, version, and model identity.
+  name, version, model, thinking-level identity, and invocation-detection method.
 
 `skillval list` returns the skill name, configured root, class, case count, whether `skillval.yml`
 exists, and a `missing`, `invalid`, or `ready` status in JSON output. Invalid case files include a
 validation error. Discovery only requires `SKILL.md`; evaluation requires a valid `skillval.yml`.
 
+## Trust model
+
+A `skillval.yml` is executable input, not passive configuration. Two fields run case-authored
+shell commands directly on the machine that grades the suite:
+
+- fixture `setup` commands, before the trial's agent runs;
+- `assert.command_exit`, at grading time.
+
+Both run with a minimal environment - only `PATH` is inherited, and `HOME` points at a throwaway
+trial directory - and are killed on timeout, but that is scoping, not a sandbox: nothing prevents
+a command from reading or writing anything your user account can reach. Evaluating a skill
+therefore means trusting its `skillval.yml` exactly as you would trust running its Makefile or
+npm scripts. Review the case file before running a suite from a repository you do not control.
+
+The agent trials themselves are a separate boundary, sandboxed per executor (see
+[Executors](#executors)): codex trials get an OS sandbox, claude trials get permission modes, and
+pi generation trials have no sandbox at all and must be acknowledged with
+`--allow-unsandboxed-pi`.
+
 ## Case files
 
 Only a file named `skillval.yml` next to `SKILL.md` is recognized. There is no `evals.yml`
@@ -127,8 +167,21 @@ Case fields:
 - `prompt`: the complete trial prompt.
 - `assert.must_match`: JavaScript regular expressions that must match, with the `m` flag.
 - `assert.must_not_match`: JavaScript regular expressions that must not match, with the `m` flag.
-- `assert.graders`: deterministic graders. `tsc` is supported for generation cases. Unknown
-  graders and graders used with an unsupported mode are validation errors.
+- `assert.graders`: parameterless deterministic graders. `tsc` is supported for generation cases.
+  Unknown graders and graders used with an unsupported mode are validation errors.
+- `assert.json_schema`: validates a produced file against a JSON Schema (draft 2020-12), for
+  generation cases. Takes `file` (relative to the workspace) and `schema` (the JSON Schema, an
+  object or boolean). The file must exist inside the workspace, be a regular file, and parse as
+  JSON; a schema mismatch reports the failing instance path. Omit `$schema` or set it to 2020-12;
+  other declared dialects, an escaping `file` path, or a schema that does not compile are validation
+  errors.
+- `assert.command_exit`: runs a shell command in the workspace and passes when it exits with the
+  expected code, for generation cases. Takes `command` and optional `expect` (default `0`). The
+  command is case-authored arbitrary shell, the same trust level as fixture `setup` (see
+  [Trust model](#trust-model)); it runs with a minimal environment and is killed after
+  120 seconds. This is the language-agnostic grader: run a
+  compiler, test runner, or validator over produced files in any language. Used in a non-generation
+  case it is a validation error.
 - `trials`: an integer from 1 through 5. Results use a strict majority. If configured trials
   disagree, the arm escalates to 5 trials.
 - `fixture`: optional workspace fixture for this case. It replaces the suite-level `fixture`
@@ -150,8 +203,9 @@ two fields, and at least one is required:
   directory, and it may not contain symbolic links (create links with `setup` commands instead);
   anything else is a validation error at load time.
 - `setup`: shell commands run sequentially inside the workspace after the copy, with a minimal
-  environment (`PATH` plus a throwaway `HOME`). A non-zero exit fails the trial with a
-  `fixture-setup` error before the agent runs; it is never a grading failure. Each command's
+  environment (`PATH` plus a throwaway `HOME`). These are case-authored arbitrary shell commands
+  executed on the grading machine (see [Trust model](#trust-model)). A non-zero exit fails the
+  trial with a `fixture-setup` error before the agent runs; it is never a grading failure. Each command's
   stdout and stderr are captured into the trial record.
 
 A suite-level `fixture` applies to every case; a case-level `fixture` replaces it entirely.
@@ -197,7 +251,7 @@ the trial as a fixture-setup error - here the conflict is the point.
 Executors are adapters with three responsibilities: report stable metadata for cache keys, prepare
 provider-specific skill and environment state, and run one trial request to return a normalized
 `Trace`. The runner owns temporary workspace lifecycle, grading, caching, majority voting, and
-reports. `codex` is the only adapter today.
+reports. Three adapters exist: `codex`, `claude`, and `pi`.
 
 The Codex adapter runs:
 
@@ -212,16 +266,70 @@ command contains `<skill>/SKILL.md`. The adapter also gives the skill arm a work
 
 Baseline arms are not seeded. Their `HOME` points to an empty temporary directory so globally
 installed skills are invisible, while `CODEX_HOME` still points to the user's real `~/.codex` for
-authentication and model configuration. A Claude executor is planned.
+authentication and model configuration.
+
+The Claude adapter runs Claude Code headlessly:
+
+```text
+claude -p <prompt> --output-format stream-json --verbose --no-session-persistence <permissions>
+```
+
+with the workspace as the working directory. Trigger cases run
+`--permission-mode dontAsk --allowedTools "Read,Glob,Grep,Skill"` - read-only, but the Skill tool
+must be allowed or invocation would be blocked before it can be observed. Generation cases run
+`--permission-mode acceptEdits`. Invocation is detected from `Skill` tool_use blocks in the
+stream-json trace that name the evaluated skill. The skill arm seeds a workspace-local
+`.claude/skills/<name>` symlink; the baseline arm points `CLAUDE_CONFIG_DIR` at an empty
+temporary directory so user-level skills are invisible (on macOS credentials live in the
+Keychain, so authentication survives; elsewhere the credentials file is copied across). The
+reported model comes from the real configuration's `settings.json`, or `default`.
+
+The pi adapter runs [pi](https://github.com/badlogic/pi-mono) headlessly:
+
+```text
+pi -p --mode json --no-session <arm flags> <tool flags> <prompt>
+```
+
+with the workspace as the working directory. pi has first-class arm switches: the skill arm
+passes `--skill <directory>` so the evaluated skill is discoverable alongside the user's normal
+library, and the baseline arm passes `--no-skills` - no HOME or config redirection is involved.
+Trigger cases restrict tools with `-t read` (read also loads SKILL.md, so invocation stays
+observable); generation cases keep pi's default tool set. pi implements the Agent Skills
+progressive-disclosure standard by having the model `read` a listed skill's SKILL.md, so
+invocation is detected from `read` toolCalls targeting `<skill>/SKILL.md` in the transcript.
+The reported model is `defaultProvider/defaultModel` from `~/.pi/settings.json`. pi resolves
+provider API keys from its auth file or environment variables (e.g. `ZAI_API_KEY`) - the key
+must be available in the environment running skillval.
+
+Unlike codex (which gets a read-only or `workspace-write` sandbox) and claude (permission modes),
+pi has no OS sandbox: generation trials rely on the temporary-workspace convention alone, with no
+enforced isolation, so an agent's writes are only conventionally scoped to the workspace. Because
+of this, skillval refuses to run pi generation cases unless you pass `--allow-unsandboxed-pi` to
+acknowledge the missing sandbox. Trigger cases are read-only and unaffected. Prefer codex or claude
+for untrusted generation cases.
+
+Each adapter reports its detection method as `invocationDetection` in report metadata. The
+`invoked` signal has asymmetric confidence: claude detects invocation from a structured `Skill`
+tool_use block, while codex and pi string-match trace text for `<skill>/SKILL.md`. Trigger rates
+should not be compared across executors as if they measured the same thing.
+
+By default, executors do not set a model or thinking/effort level; trials inherit the harness
+defaults the user has configured, and each adapter captures both into its identity so results are
+always associated with what actually ran: codex reads `model` and `model_reasoning_effort` from
+`~/.codex/config.toml`, claude reads `model` and `effort` from `settings.json`, and pi reads
+`defaultProvider/defaultModel` and `defaultThinkingLevel` from `~/.pi/settings.json`. A missing
+value is recorded as `default` (the provider's own default applies). Passing `--model`/`--effort`
+overrides the default for the run, and the override is what gets captured. Changing any of these -
+in the provider configuration or via the flags - therefore keys distinct cached results.
 
 Cached arm results are keyed by runner version, skill content hash, serialized case, arm, executor
-name, executor version, and configured model. A trial has a 15-minute timeout and a 64 MB output
-buffer.
+name, executor version, configured model, and configured thinking level. A trial has a 15-minute
+timeout and a 64 MB output buffer.
 
 ## Roadmap
 
-- Add a Claude executor, then support multi-executor runs through the same normalized trace
-  interface.
+- Support multi-executor runs through the same normalized trace interface, now that `codex` and
+  `claude` adapters share it.
 - Run multiple models and emit per-model reports. A passing binding or trigger result on a weaker
   tier is a conservative bound for stronger tiers. Baseline no-op results remain model-specific,
   and a rule is a prune candidate only when every model in normal use passes at baseline.