skill-lens 0.2.0__tar.gz

skill_lens-0.2.0/.claude/skills/writing-skill-evals/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: writing-skill-evals
+description: Use when writing, running, or auditing skill-lens eval suites for an Agent Skill — deciding which cases a skill needs, choosing between assertions, judge rubrics and trajectory checks, and reading a failing case correctly
+---
+
+# Writing skill evals
+
+An eval suite is worth exactly what its cases check. The two failure modes are a suite
+that is green because it checks nothing, and a red case that gets the skill edited when
+the eval was wrong. Everything here exists to prevent one of those.
+
+## Workflow
+
+1. **Orient.** Read the target `SKILL.md`. Look for existing evals in `evals/` or
+   `*.eval.yaml` beside it, and run `skill-lens list <path>` to see what the tool already
+   discovers.
+2. **Scaffold, or extend.** If step 1 found no suite, run `skill-lens init <skill-dir>` —
+   do not hand-roll the file structure, the generated file already carries the triggering
+   pair and the placeholders that stop an unfinished suite from running. If a suite
+   already exists, do not run `init`: it exits 2 rather than touch an existing file. Read
+   `references/auditing.md` and extend the suite you found instead. (`--force` overwrites
+   the file outright, which is not what you want when a suite is already there to build
+   on.)
+3. **Mine the skill for claims.** Every "always", "never" and "must" in the instructions
+   is a candidate case. The frontmatter `description` is a claim too, and it is precisely
+   what the `mode: offered` cases test.
+4. **Propose the case list and confirm it.** Show the user the cases you intend to write,
+   one line each, before writing any YAML. Ask only for what `SKILL.md` cannot tell you:
+   which tools exist and what they return, which policy edges are real, what a good answer
+   sounds like. Do not ask for what the file already says.
+5. **Write the cases.** Replace every `TODO(skill-lens)`. Read
+   `references/eval-file-syntax.md` for the fields, and `references/case-design.md` for
+   patterns to draw from.
+6. **Validate, then run.** `skill-lens list <path>` is the structural check — it parses
+   every case at zero cost and catches a malformed file before anything else runs; it is
+   what this repo's own CI self-check uses. `skill-lens run <path>` is a different thing:
+   it needs a configured runner, and under the defaults (`FakeRunner`, `FakeJudge`) a
+   `mode: offered` case and any case with a `judge:` block come back `errored`, not
+   `failed` — the fake runner cannot report a triggering decision and the fake judge does
+   not grade. Neither is a statement about the skill. Configure a real runner (and a judge,
+   if any case needs one) before reading those results as anything.
+7. **Triage every red case** by the rule below before editing anything.
+
+## Choosing the check
+
+| Use | When |
+| --- | --- |
+| `assertions` | The check is mechanical and stable: an id appears, a traceback does not. |
+| `judge` | The claim is about quality, tone or reasoning. "Explains it plainly" is not a substring. |
+| `trajectory` | The failure is invisible in the output — deciding without looking the order up, calling the tool that was forbidden. |
+| `budget` | Guarding against a regression into a tool-call loop or a runaway answer. |
+
+## Triage: the eval, the skill, or the harness?
+
+A red case means one of three different things, and naming which comes before any edit.
+Start with whether the case is `failed` or `errored` — that alone rules one branch out or
+in before you look at anything else.
+
+**The eval is wrong** when the case `failed` and the output was actually fine: a regex
+tight enough to feel rigorous but that rejects phrasing a model may legitimately vary, an
+assertion on wording the skill never promised, a budget below what the task honestly
+costs. `examples/greeting/greeting.eval.yaml` in the skill-lens repo documents a real
+instance — a single-sentence regex relaxed after real model output failed it for no good
+reason.
+
+**The skill is wrong** when the case `failed` and the output genuinely was not what the
+skill claims: the instruction is missing, or is present but too weak to survive a
+plausible prompt.
+
+**The harness is wrong** when the case `errored`. An errored case never ran to a real
+verdict — the runner or the judge broke, or was never configured for what the case asks
+of it. A `mode: offered` case under the default fake runner, or a `judge:` case under the
+default fake judge, errors this way by design: the fake runner cannot report a triggering
+decision, and the fake judge does not grade. An errored case says nothing about the
+skill — fix the configuration (a real runner, a real judge), and change neither the eval
+nor the SKILL.md.
+
+Fixing the eval is yours to do. **Changing the target `SKILL.md` is proposed and
+confirmed, never silent** — the user is the author of their skill, and an eval that gets
+its own subject rewritten to match it has stopped measuring anything.
+
+## Rules that are not negotiable
+
+- **Ship the negative control.** A suite of triggering positives scores a skill that fires
+  on everything at 100%. `mode: offered` cases come in pairs.
+- **Never leave a case with no check.** At least one of `assertions`, `trajectory` or
+  `judge` — a case with none of the three passes without checking anything. `mode:
+  offered` cases legitimately carry `trajectory.skill_triggered` and no `assertions`;
+  that still counts as a check.
+- **Every rubric entry must be independently checkable and evidenced.** skill-lens records
+  a check that passes without citing evidence as a failure, so a vague entry costs a case
+  rather than buying coverage.
+- **Never leave a `TODO(skill-lens)` behind.** The run will refuse it, which is the point,
+  but a suite that cannot run is not a suite.
+
+## Auditing an existing suite
+
+Read `references/auditing.md` and work its checklist. Report findings; do not rewrite the
+user's suite unasked.

skill_lens-0.2.0/.claude/skills/writing-skill-evals/evals/writing-skill-evals.eval.yaml

@@ -0,0 +1,93 @@
+# Evals for the skill that writes evals. Four cases, three concerns:
+#   - triggering: does it fire on an eval-writing request, and stay quiet
+#     otherwise (the negative control this skill insists on for everyone else).
+#   - authoring: does it read the skill before writing the file, and is what it
+#     writes a suite rather than a shape -- judged, because "includes a negative
+#     control" is a property of the YAML, not a substring of the reply.
+#   - auditing: handed a deliberately weak suite, does it name what is missing.
+cases:
+  - name: reaches for the skill on an eval-writing request
+    mode: offered
+    task: Write evals for my order-support skill in ./skills/order-support
+    tags: [triggering]
+    trajectory:
+      skill_triggered: true
+
+  - name: leaves unrelated work alone
+    mode: offered
+    task: Rename the `parse` function in utils.py to `parse_input`
+    tags: [triggering]
+    trajectory:
+      skill_triggered: false
+
+  - name: reads the skill before writing the suite
+    task: >-
+      Write an eval suite for the skill at ./skills/order-support. Read the
+      skill first, then write the file.
+    tags: [authoring, judged]
+    tools:
+      - name: read_file
+        description: Read a file from disk
+        parameters:
+          path: string
+        returns: |
+          ---
+          name: order-support
+          description: Handle customer refund requests against the 30-day return policy
+          ---
+          Always call lookup_order before saying anything about an order's state.
+          Never issue a refund for an order delivered more than 30 days ago.
+      - name: write_file
+        description: Write text to a file on disk
+        parameters:
+          path: string
+          content: string
+        returns: '{"ok": true}'
+      - name: run_command
+        description: Run a shell command and return its output
+        parameters:
+          command: string
+        returns: 'order-support  0 case(s)  skills/order-support'
+    trajectory:
+      order: [read_file, write_file]
+      max_calls: 8
+    judge:
+      expected: >-
+        An eval suite covering both sides of the 30-day policy, with a
+        mode: offered pair including the negative control, and no placeholders
+        left in it.
+      rubric:
+        - The suite it wrote includes a case where a refund is refused
+        - The suite it wrote includes a case where a refund is allowed
+        # Quoted: a plain YAML scalar cannot contain ": ".
+        - "The suite it wrote includes a mode: offered case expecting skill_triggered true"
+        - "The suite it wrote includes a mode: offered case expecting skill_triggered false"
+        - No case in the suite it wrote still contains the text TODO
+        - Every non-triggering case it wrote has at least one assertion, judge or trajectory check
+
+  - name: names what a weak suite is missing
+    task: Review the eval file at ./skills/greeting/evals/greeting.eval.yaml and tell me what is wrong with it
+    tags: [auditing, judged]
+    tools:
+      - name: read_file
+        description: Read a file from disk
+        parameters:
+          path: string
+        returns: |
+          cases:
+            - name: greets someone
+              task: greet Ada
+            - name: fires on a greeting request
+              mode: offered
+              task: say hello to Ada
+              trajectory:
+                skill_triggered: true
+    trajectory:
+      called: [read_file]
+    judge:
+      expected: >-
+        A review naming the assertion-free first case and the missing negative
+        control for the triggering case.
+      rubric:
+        - The review says the first case checks nothing because it has no assertions
+        - The review says the triggering case has no negative control

skill_lens-0.2.0/.claude/skills/writing-skill-evals/references/auditing.md

@@ -0,0 +1,38 @@
+# Auditing an existing suite
+
+Work the checklist, then report findings with the file and case named. Do not rewrite the
+user's suite unasked.
+
+## Checklist
+
+- [ ] **Vacuous cases.** Any `loaded` case with no `assertions`, no `trajectory`, no
+      `judge`? It passes without checking anything.
+- [ ] **Missing negative control.** Any `mode: offered` positive with no negative
+      counterpart? The suite cannot distinguish a well-targeted skill from one that fires
+      on everything.
+- [ ] **Claims with no case.** List the skill's "always/never/must" statements and find
+      the case for each. Name the ones with none.
+- [ ] **One-sided edges.** A policy case that only proves the refusal, never the approval.
+- [ ] **Over-tight assertions.** `equals` or a `regex` pinning phrasing the skill never
+      promised; an assertion that would fail on a legitimately different good answer.
+- [ ] **Fixture assertions.** An assertion whose value comes from a mock tool's `returns`
+      rather than from the skill's behavior.
+- [ ] **Unevidenceable rubric entries.** Anything you could not prove by quoting the
+      output ("is helpful", "is well structured"), or compound entries hiding which half
+      failed.
+- [ ] **Budgets that never bind, or bind too tightly.** A ceiling far above any plausible
+      run checks nothing; one at the current spend turns every prompt change red.
+- [ ] **A cost limit as the only budget check** on a model with no pricing entry — the
+      check is skipped, so the case fails for having verified nothing.
+- [ ] **Leftover placeholders.** `TODO(skill-lens)` anywhere.
+- [ ] **Tags.** Is there a `smoke` subset a fast CI job could run?
+- [ ] **Cases that error under the configured runner or judge.** An `errored` case never
+      ran to a verdict; it says nothing about the skill. Check whether the harness is
+      simply unconfigured (default fake runner, default fake judge) or the case declares
+      something this runner cannot report (`mode: offered` under a runner with no
+      triggering support, for instance).
+
+## Reporting
+
+For each finding: the file and case, what is wrong, and the smallest change that fixes it.
+Rank by what would let a broken skill through, not by what is easiest to fix.

skill_lens-0.2.0/.claude/skills/writing-skill-evals/references/case-design.md

@@ -0,0 +1,51 @@
+# Designing the cases
+
+## Deriving cases from the skill
+
+Read the instructions and list every claim. Claims look like: "always call X first",
+"never do Y after 30 days", "answer in one short sentence", "ask before writing files".
+Each claim gets at least one case; the interesting ones get two, one either side of the
+line.
+
+The frontmatter `description` is the claim that triggering tests: it is the only text an
+agent sees when deciding whether to reach for the skill.
+
+## A minimum suite
+
+1. **The common case** — the prompt this skill exists for, with an assertion on something
+   every good answer contains.
+2. **The edge** — the prompt that reaches the policy line, with `trajectory` if getting
+   there requires a tool call.
+3. **The other side of the edge** — the near-identical prompt where the answer flips. A
+   skill that refuses everything passes case 2 alone.
+4. **The triggering pair** — `mode: offered`, positive and negative.
+
+## Patterns by skill archetype
+
+| Archetype | What to check |
+| --- | --- |
+| Policy skill (refunds, approvals) | `trajectory` proving it looked before deciding; `forbidden` on the destructive tool; both sides of the policy line |
+| Tool-using skill | `order` for a required sequence; `max_calls` against loops; a case where the tool returns an error string |
+| Formatting skill | `regex` — but only on structure the skill actually promised; a judge for "reads plainly" |
+| Knowledge skill | `contains` on the fact; `not_contains` on the plausible wrong answer; a judge for reasoning |
+
+## Writing rubric entries
+
+Each entry is checked independently and must be evidenced by quoting the output. That
+makes the test for a good entry mechanical: **could you point at the sentence that proves
+it?**
+
+- Good: "The reply names order 1234." "The reply states the return window has closed."
+- Bad: "The reply is helpful." "The response is well structured." Nothing can be quoted
+  as proof, and an unsupported pass is an LLM judge's characteristic failure mode.
+
+Split compound entries. "Names the order and explains the policy" hides which half failed.
+
+## Assertions that age badly
+
+- A regex pinning phrasing the skill never promised. Check the structure the skill
+  committed to, nothing more.
+- `equals` on anything a model generates freely.
+- Asserting on a number the mock tool returns — that tests the fixture, not the skill.
+- A budget set at the current spend. Leave headroom, or every prompt improvement is a
+  red case.

skill_lens-0.2.0/.claude/skills/writing-skill-evals/references/eval-file-syntax.md

@@ -0,0 +1,100 @@
+# Eval file syntax
+
+A file has one top-level `cases:` list. Unknown keys inside a case or an assertion are
+rejected — without that, a typo like `assertion:` would yield a case that passes
+vacuously.
+
+## Case fields
+
+| Field | Required | Meaning |
+| --- | --- | --- |
+| `name` | yes | Case name, shown in reports |
+| `task` | yes | The prompt handed to the runner |
+| `assertions` | no | Output checks; a case with none passes vacuously |
+| `tools` | no | Mock tools the agent may call |
+| `trajectory` | no | Which tools must and must not have been called, and in what order |
+| `budget` | no | Ceilings on tokens, cost, and latency |
+| `judge` | no | A rubric for an LLM judge |
+| `mode` | no | `loaded` (default) or `offered` |
+| `tags` | no | Labels for `--tag` filtering |
+
+## Assertion kinds
+
+| `kind` | Passes when |
+| --- | --- |
+| `contains` | `value` appears in the output |
+| `not_contains` | `value` does not appear in the output |
+| `regex` | `value` matches anywhere in the output (`re.search`) |
+| `equals` | the stripped output equals `value` exactly |
+
+Every assertion must hold for the case to pass. An unknown kind or a malformed regex
+aborts the run as an authoring error rather than being reported as a skill failure.
+
+## Tools
+
+Nothing executes. Calling a mock tool records the call and returns `returns` verbatim, so
+the trajectory is genuinely the model's choice and a run has no side effects. Mock tools
+accept any arguments — a hallucinated argument must not surface as an infra error.
+
+```yaml
+    tools:
+      - name: lookup_order          # must be a valid identifier
+        description: Look up an order by its id
+        parameters:
+          order_id: string          # string | integer | number | boolean
+        returns: '{"id": "1234", "status": "delivered"}'
+```
+
+## Trajectory
+
+```yaml
+    trajectory:
+      called: [lookup_order]        # must have been called
+      forbidden: [issue_refund]     # must not have been
+      order: [lookup_order, issue_refund]   # relative order, not exhaustive
+      max_calls: 3
+      skill_triggered: true         # mode: offered only
+```
+
+Every name in `called`, `forbidden` and `order` must be a tool the case itself declares.
+
+## Budget
+
+```yaml
+    budget:
+      max_tokens: 2000
+      max_cost_usd: 0.01
+      max_latency_ms: 20000
+```
+
+An unpriced model makes a cost limit unverifiable, so that check is skipped rather than
+counted as passed — a cost limit as the only budget check then fails the case, because
+nothing was verified.
+
+## Judge
+
+```yaml
+    judge:
+      expected: A short, plain-language refusal that names the order id.
+      rubric:
+        - The reply names order 1234
+        - The reply explains that the return window has closed
+```
+
+One verdict per rubric entry, each with its evidence; skill-lens derives pass and score
+from those. A check that passes without evidence is recorded as a failure. An empty
+rubric, or a blank entry, is an authoring error. Judging costs money and is opted into
+with `judge = "pydantic-ai"` in `skill-lens.toml`; the default `judge = "fake"` reports a
+judged case as **errored** rather than passing a rubric nobody checked.
+
+## Triggering (`mode: offered`)
+
+The skill is not force-loaded; it is registered as a tool named after the skill
+(`order-support` becomes `order_support`) and described by its frontmatter description.
+Calling it delivers the instructions. Check it with `skill_triggered`, not by naming the
+tool in `called:`. Setting `skill_triggered` on a `loaded` case is an authoring error.
+
+## Placeholders
+
+`skill-lens init` writes `TODO(skill-lens)` into every field you must supply. A case still
+containing one aborts the run with exit 2, naming the field.

skill_lens-0.2.0/.github/copilot-instructions.md

@@ -0,0 +1,60 @@
+# Copilot instructions for skill-lens
+
+`skill-lens` is a standalone CLI and library that runs evaluations on Agent Skills
+(`SKILL.md` files). Skills under test and their eval cases are **inputs** — nothing about a
+skill under test is vendored here. The tool runs as a CI gate where the exit code is the
+contract. Full design: [`ARCHITECTURE.md`](../ARCHITECTURE.md).
+
+## The two seams
+
+- `Runner.run(skill, case) -> RunResult` — where every agent framework plugs in.
+- `Evaluator.evaluate(case, result) -> EvalScore` — where every scoring strategy plugs in.
+
+`models.py` holds every Pydantic model; no other module defines a data shape.
+
+## Invariants — flag any change that breaks one
+
+These are decided behaviors with tests asserting them, not accidents.
+
+1. **`errored` is not `failed`.** `failed` = the case ran and scored below the bar (an eval
+   signal). `errored` = the runner itself blew up (an infra signal). Runners must **never
+   raise** for provider failures — they set `RunResult.error`. Errored cases fail the gate.
+2. **A run executing zero cases fails the gate.** "Nothing ran" is a broken run.
+3. **Authoring errors abort the run; they never score as failures.** An unknown assertion
+   kind, a bad regex, or an unknown YAML key is a mistake in the user's files and says
+   nothing about the skill. They propagate out of the orchestrator and `cli.py` exits 2.
+4. **Exit codes are the CI contract:** gate passed `0`, gate failed `1`, user or authoring
+   error `2`. A JSON-write failure escalates to 2 only when the gate already passed.
+5. **`extra="forbid"`** on every user-authored model. Without it a typo like `assertion:`
+   yields a case that passes vacuously.
+6. **All file IO pins `encoding="utf-8"`** and re-raises as a typed parse error naming the
+   file and field.
+7. **YAML goes through `skill_lens.yaml_loading.safe_load`**, never `yaml.safe_load`.
+8. **Secrets come from environment variables only** — never from `skill-lens.toml`.
+9. **No agent-framework type appears outside `runners/pydantic_ai.py`.**
+10. **Cost lookup degrades, never raises.** An unpriced model yields `cost_usd = 0.0` and a
+    `cost_note`. An unpriceable budget check is *skipped*, not passed.
+11. **`skill_lens` (underscore) never appears in user-facing output.** The user-facing name
+    is `skill-lens` everywhere.
+
+## Do not suggest
+
+- Importing an agent framework anywhere outside `runners/pydantic_ai.py`.
+- Replacing `yaml_loading.safe_load` with `yaml.safe_load`.
+- Raising from a runner when a provider call fails.
+- Removing `extra="forbid"` to make a model more permissive.
+- Making a mock tool reject unexpected arguments — a hallucinated argument is an eval
+  signal, and raising would surface it as an infra error.
+- Treating a check that could not be performed as passed.
+
+## Conventions
+
+- **Test-driven.** The failing test comes first. The default test tier must stay zero-cost,
+  offline and deterministic — `pytest` runs with `--block-network`.
+- **Conventional Commits are enforced, not stylistic.** `cz bump` derives the version and
+  changelog from history. PRs are squash-merged, so **the PR title becomes the commit on
+  `main`** and must be conventional too. Flag a non-conforming PR title in review.
+- **Documentation ships with the change.** A change to a flag, config key, `EvalCase` field
+  or assertion kind must update the matching page under `docs/`; a change to a protocol or
+  an invariant must update `ARCHITECTURE.md`.
+- Line length 100, `ruff` for lint and format.

skill_lens-0.2.0/.github/instructions/cli-gating.instructions.md

@@ -0,0 +1,27 @@
+---
+applyTo: "src/skill_lens/cli.py,src/skill_lens/gating.py,src/skill_lens/orchestrator.py,src/skill_lens/reporters/**"
+---
+
+# Reviewing the CLI, gate, orchestrator and reporters
+
+This is the contract surface. CI depends on the exit code; humans depend on the output.
+
+- **Exit codes are the contract:** gate passed `0`, gate failed `1`, user or authoring error
+  `2`. A JSON-write failure escalates to 2 **only when the gate itself passed** — it must
+  never mask an already-failing gate. Flag any change that widens or reorders these.
+- **A run executing zero cases fails the gate.** "Nothing ran" is a broken run, not a pass —
+  otherwise a mistyped path reports success forever. The reason must name the cause: no
+  skills found, all skills skipped for having no cases, or every case filtered out by
+  `--tag`.
+- **Errored cases fail the gate by default**, so CI never goes green on a run that did not
+  happen.
+- **Authoring errors abort the run.** `orchestrator.run_evals` deliberately lets them
+  propagate; `cli.py` catches them via `_AUTHORING_ERRORS` and prints the message without a
+  traceback. Flag any `try`/`except` in the orchestrator that would swallow one into a
+  failing score.
+- **`skill_lens` (underscore) never appears in user-facing output.** The name is
+  `skill-lens`: command, config file, distribution, prose.
+- A new runner must be registered in `cli._RUNNERS`, and if it spends money it needs the
+  preflight key check before construction so a missing key costs nothing and exits 2.
+- Gate reasons are read by whoever is staring at a red pipeline. Each should say what failed
+  and against which threshold.

skill_lens-0.2.0/.github/instructions/docs.instructions.md

@@ -0,0 +1,27 @@
+---
+applyTo: "docs/**,*.md,mkdocs.yml"
+---
+
+# Reviewing documentation
+
+- **`README.md` is a landing page.** Reference prose lives in `docs/`. Flag any PR that
+  reintroduces command, config or eval-file reference material into the README — it will
+  drift.
+- **`ARCHITECTURE.md` has exactly one copy**, at the repository root.
+  `docs/architecture.md` includes it with a `pymdownx.snippets` directive. Flag any
+  duplication of its content.
+- **`ARCHITECTURE.md` must use absolute links only (full URLs), never relative links.**
+  `tests/test_docs.py` resolves its relative links from the repo root, but mkdocs resolves
+  them from `docs/` because the file is inlined into `docs/architecture.md` by a snippet —
+  a relative link can pass the test and still fail `mkdocs build --strict`.
+- **A new page must be added to `nav:` in `mkdocs.yml`.** `tests/test_docs.py` fails on an
+  orphan page, and `mkdocs build --strict` fails on a nav entry with no file.
+- **`docs/superpowers/` is a historical archive** of specs and plans. It is excluded from
+  the built site and does not count as documenting a change. Its contents were superseded by
+  what shipped — read `src/` as the source of truth.
+- Documentation ships **with** the change: a new flag updates `docs/cli.md`, a new config
+  key updates `docs/configuration.md`, a new assertion kind updates `docs/eval-files.md`, a
+  new invariant updates `ARCHITECTURE.md`.
+- Do not add `pymdownx.emoji` or mermaid `custom_fences` to `mkdocs.yml`: both need
+  `!!python/name:` YAML tags, which break the plain-YAML nav parsing in `tests/test_docs.py`.
+- Relative links must resolve — there is a test for it.

skill_lens-0.2.0/.github/instructions/evaluators.instructions.md

@@ -0,0 +1,34 @@
+---
+applyTo: "src/skill_lens/evaluators/**"
+---
+
+# Reviewing evaluator code
+
+An evaluator turns a `RunResult` into a pass/fail verdict with a score and human-readable
+detail. Its *score* is about the skill; a low score is never used to report a harness
+problem, and a harness problem is never reported as a low score.
+
+- **A check that could not be performed is not a pass.** `BudgetEvaluator` *skips* an
+  unpriceable cost limit rather than passing it, so a case whose only budget check is an
+  unpriced cost limit fails — nothing was verified. `JudgeEvaluator` follows the same rule:
+  an unjudged rubric, and a check that passes without citing evidence, are not passes. Flag
+  any "if we can't check it, assume it's fine" branch.
+- **`errored` is for infra, and it must set `passed=False`.** An evaluator sets
+  `EvalScore.errored` when the harness broke — a judge endpoint returning 500, a verdict
+  that does not match its rubric, an offered case on a runner that cannot report triggering
+  — never when the skill scored badly. The orchestrator turns any errored score into an
+  errored case. A model validator makes `errored=True, passed=True` unconstructable.
+- **No agent framework in this directory.** `JudgeEvaluator` reaches a model only through
+  the injected `Judge` protocol. Flag any direct client or framework import.
+- **Eval-side spend goes on `EvalScore.cost_usd`, never on `RunResult`.** `budget:` measures
+  the skill's efficiency; what we spent grading it is the harness's cost and is reported
+  separately.
+- **Authoring errors propagate; they never become a failing score.** An unknown assertion
+  kind, a malformed regex, or a tool name in a `trajectory` block that the case never
+  declared is a mistake in the user's files. Scoring it as a failure would be a lie about
+  the skill. These raise and `cli.py` exits 2.
+- **New assertion kinds go in `_CHECKS`** in `assertion.py` and must be documented in
+  `docs/eval-files.md`. `tests/test_docs.py` fails until both happen.
+- `detail` is read by a human staring at a red CI run. It should name what failed and with
+  what value, not just that something did.
+- Every assertion in a case must hold for the case to pass. A case with no assertions passes.

skill_lens-0.2.0/.github/instructions/models-loaders.instructions.md

@@ -0,0 +1,25 @@
+---
+applyTo: "src/skill_lens/models.py,src/skill_lens/config.py,src/skill_lens/yaml_loading.py,src/skill_lens/cases/**,src/skill_lens/skills/**"
+---
+
+# Reviewing models and loaders
+
+This is where user-authored files become typed objects. Every mistake here is silent.
+
+- **`extra="forbid"` on every user-authored model** — `EvalCase`, `AssertionSpec`,
+  `ToolSpec`, `TrajectorySpec`, `BudgetSpec`, `Config`. Without it a typo like `assertion:`
+  yields a case that passes vacuously, the worst failure mode an eval tool has. Flag any
+  removal or any new user-authored model that omits it.
+- **`models.py` holds every data shape.** Other modules import from it; they do not define
+  their own.
+- **All file IO pins `encoding="utf-8"`** and re-raises `OSError`/`UnicodeDecodeError` as a
+  typed parse error (`SkillParseError`, `CaseParseError`, `ConfigError`) naming the file and
+  the field. A raw traceback reaching the user is a bug.
+- **YAML goes through `yaml_loading.safe_load`.** PyYAML's `SafeLoader` is YAML 1.1 and
+  turns bare `yes`/`no`/`on`/`off` into booleans; an assertion `value: yes` is meant as the
+  string.
+- **Secrets never come from `skill-lens.toml`.** A config file is committed; a key must not
+  be. Flag any new config field that would hold a credential.
+- **Derived values are properties, not stored fields** — `RunResult.tokens`, `errored`, and
+  the `RunReport` aggregates. A stored copy can disagree with its source.
+- Skills with no eval files are reported as **skipped**, visibly — never silently dropped.

skill_lens-0.2.0/.github/instructions/runners.instructions.md

@@ -0,0 +1,46 @@
+---
+applyTo: "src/skill_lens/runners/**,src/skill_lens/judges/**"
+---
+
+# Reviewing runner and judge code
+
+These two directories are the framework boundary. Everything the core sees is a plain
+`RunResult` or `JudgeVerdict`. The two share a contract, so the rules below apply to both
+unless one is named.
+
+- **Never raise for a provider failure.** Timeouts, rate limits, auth errors and malformed
+  responses become `RunResult(error=...)` or `JudgeVerdict(error=...)`. Raising turns an
+  infra problem into an unhandled crash and loses the errored/failed distinction the gate
+  depends on.
+- **The whole capture path belongs inside the guard.** Reading messages, summing usage,
+  pricing and constructing the result must all sit inside the `try`, not just the model
+  call — a serialisation failure on a real provider's message shape must report errored,
+  not crash the run. This was a real bug once.
+- **Only `runners/pydantic_ai.py` and `judges/pydantic_ai.py` may import an agent
+  framework.** `runners/tools.py` builds framework-neutral `MockTool`s — a name, a JSON
+  schema and a callable — and the adapters wrap them.
+  `tests/test_framework_isolation.py` enforces the boundary across the whole package.
+- **A judge reports per-check verdicts only** — never an overall verdict or a blended
+  score. skill-lens derives `passed` and `score` from the per-check results, because an
+  unsupported PASS hidden inside a single number is invisible. `JudgeOutput` is deliberately
+  narrower than `JudgeVerdict` so the model cannot assert its own cost or outcome.
+- **`FakeJudge` unscripted returns an error, not a pass.** That is what makes
+  `judge = "fake"` safe as the built-in default: a rubric with no configured judge is
+  errored, never a quiet green.
+- **`judges/prompt.py` is pure and deterministic.** Same input, byte-identical output, every
+  process — cassettes match on the request body, so any instability makes them unmatchable.
+  No `hash()`, no randomness, no time.
+- **Mock tools accept any arguments.** A model hallucinating an argument is an eval signal
+  about the skill; rejecting it would surface as an infra error instead.
+- **Nothing executes in a mock tool.** It records the call and returns its canned value, so
+  the trajectory is genuinely the model's choice and a run has no side effects.
+- **Pricing never fails a run.** An unpriced model yields `cost_usd = 0.0` plus a
+  `cost_note`. Flag any code path where a pricing lookup can raise.
+- **`RunResult.tokens` is derived** from the input/output split. Flag any attempt to set it.
+- **`FakeRunner.run` returns `model_copy(deep=True)`** so a caller cannot corrupt scripted
+  state.
+- **Cassettes are replay-only and secret-free.** Recording is a deliberate, key-bearing act.
+  A missing cassette skips; a mismatched request fails rather than reaching the network.
+  Flag any credential or account-identifying header that could reach a recorded file.
+- A runner that spends money sets `needs_api_key = True` so the preflight check runs before
+  any request.

skill_lens-0.2.0/.github/instructions/tests.instructions.md

@@ -0,0 +1,23 @@
+---
+applyTo: "tests/**"
+---
+
+# Reviewing tests
+
+- **Test-driven:** the failing test comes first. A PR adding behavior with no test that
+  would have failed before it is incomplete.
+- **The default tier is zero-cost, offline and deterministic.** `pytest` runs with
+  `--block-network` and deselects `integration`. Flag any default-tier test that could reach
+  the network, need a key, or depend on wall-clock time or ordering.
+- **Marker tiers:** unmarked = offline pipeline (`FakeRunner`); `cassette` = replays
+  recorded provider traffic, free, selected by default; `integration` = real API spend,
+  opt-in only.
+- **`conftest.py` chdirs every test into a fresh `tmp_path`** so config upward-discovery
+  cannot pick up an ambient `skill-lens.toml`. A test that reads repository files must
+  anchor on `Path(__file__).resolve().parents[1]`, never `Path.cwd()`. Flag any test relying
+  on the working directory.
+- **Cassettes must be secret-free.** Both request and response headers are scrubbed, on two
+  different vcrpy hooks. Flag anything that could write a credential or an account
+  identifier to disk.
+- A test asserting an error path should assert the message a user actually sees, not just
+  the exception type.

skill_lens-0.2.0/.github/pull_request_template.md

@@ -0,0 +1,14 @@
+## What and why
+
+<!-- What changes, and what problem it solves. The diff shows what; explain why. -->
+
+## Checklist
+
+- [ ] The PR title follows [Conventional Commits](https://www.conventionalcommits.org/) —
+      it becomes the commit on `main` when this is squash-merged.
+- [ ] Documentation is updated in the same PR: `docs/` for user-facing behavior,
+      `ARCHITECTURE.md` for design or invariants, `mkdocs.yml` for a new page.
+      (If genuinely not needed, add the `no-docs-needed` label.)
+- [ ] `uv run pytest` passes.
+- [ ] `uv run ruff check .` and `uv run ruff format --check .` pass.
+- [ ] `uv run mkdocs build --strict` passes if any documentation changed.