agent-crucible 0.4.2__tar.gz

agent_crucible-0.4.2/.gitattributes

@@ -0,0 +1 @@
+* text=auto eol=lf

agent_crucible-0.4.2/.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python: ["3.10", "3.11", "3.12"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install
+        run: pip install -e ".[dev]"
+      - name: Test
+        # The suite is hermetic: it stubs the agent, so no ANTHROPIC_API_KEY or
+        # claude CLI is needed. This is the real cross-platform proof, Linux
+        # included, that the runner, assertions, and stats work everywhere.
+        run: python -m pytest -q

agent_crucible-0.4.2/.github/workflows/release.yml

@@ -0,0 +1,25 @@
+name: Release to PyPI
+
+# Publishes on a version tag (e.g. v0.1.0). Uses PyPI Trusted Publishing
+# (OIDC) — configure the publisher at pypi.org first; no token secret needed.
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build-and-publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # for trusted publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Build
+        run: |
+          pip install build
+          python -m build
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

agent_crucible-0.4.2/.gitignore

@@ -0,0 +1,25 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.venv/
+venv/
+
+# Benchmark output lives inside each skill directory.
+**/.agent-skill/runs/
+
+# Third-party agent skills installed via `npx skills add` (e.g. strix).
+.agents/
+.claude/
+
+# strix lockfile from npx skills add
+skills-lock.json
+
+# build artifacts
+dist/
+
+# local benchmark scratch (raw run records are committed under examples/)
+bench_logs/
+run_*.sh

agent_crucible-0.4.2/BENCHMARK.md

@@ -0,0 +1,125 @@
+# Benchmark results
+
+Real runs against a live Claude Code agent. These are the actual numbers this
+tool produced — including the ones that show **no effect**. A benchmark you can
+only pass is not a benchmark.
+
+## How to reproduce
+
+```bash
+pip install agent-crucible
+git clone https://github.com/ayah-spec/agent-crucible
+cd agent-crucible
+crucible test examples/error-handling -n 10
+crucible test examples/negative-control -n 5
+```
+
+Each run copies the task fixture into a throwaway git repo, runs the agent once
+without the skill and once with it installed to `.claude/skills/`, then checks
+the result with deterministic assertions. Raw per-run records for every number
+below are committed under each skill's `.crucible/runs/`.
+
+## Conditions
+
+| | |
+|---|---|
+| Agent / harness | Claude Code CLI, version 2.1.251 |
+| Model | `claude-sonnet-4-5-20250929` (pinned) |
+| OS | Windows |
+| Python | 3.11.9 |
+| Repeats (N) | per-condition, stated per row |
+| Success test | Fisher exact on per-run outcomes, conclusive at p < 0.05 |
+| Date | 2026-08-31 |
+
+Numbers are model-, harness-, task-, and environment-dependent. The correct
+reading of a row is *"on this eval suite, under this model and this Claude Code
+version, the skill did X"* — not *"this skill makes the AI X% better"*.
+
+## Task success
+
+| Skill | N | Without | With | Δ | p-value | Verdict |
+|---|---|---|---|---|---|---|
+| error-handling | 5 | 0% | 40% | +40pp | 0.444 | inconclusive |
+| **error-handling** | **10** | **0%** | **50%** | **+50pp** | **0.033** | **improvement** |
+| sql-parameterization | 5 | 100% | 100% | 0pp | 1.000 | no effect |
+| type-hints | 5 | 40% | 80% | +40pp | 0.524 | inconclusive |
+| type-hints | 10 | 20% | 60% | +40pp | 0.170 | inconclusive |
+| negative-control | 5 | 100% | 100% | 0pp | 1.000 | no effect (expected) |
+
+## What these results actually say
+
+**error-handling is the headline — and it is a lesson about sample size, not
+just a win.** At N=5 the skill looked like a +40pp improvement, but Fisher's
+exact test returned p=0.444: with five runs, 0/5 vs 2/5 is well within chance.
+Only at N=10 (0/10 vs 5/10, p=0.033) did the effect cross into conclusive
+territory. A tool that reported the N=5 point estimate as fact would have
+published a "+40%" number that its own statistics did not support. This is the
+single clearest demonstration of why the tool repeats runs and tests
+significance instead of subtracting two scores.
+
+**sql-parameterization shows no effect, honestly.** The baseline agent already
+wrote parameterized queries and passed every assertion (1.00 → 1.00), so there
+was nothing for the skill to add on this task. That is a true negative, not a
+tuning failure — reported as-is.
+
+**negative-control behaves exactly as a control should.** Its skill only changes
+docstring wording, so it cannot affect whether the tests pass, and it didn't
+(100% → 100%, p=1.000). Notably it *raised* token use (184k → 207k) and wall
+time (61s → 92s): an irrelevant skill is not free, it is overhead. An instrument
+that reported an improvement here would be untrustworthy everywhere.
+
+**type-hints stays unproven even at N=10.** At N=5 it looked like +40pp
+(p=0.524); at N=10 it was still +40pp but p=0.170 — and the baseline itself moved
+(40% → 20%), which is exactly the run-to-run wobble that makes a single small
+sample untrustworthy. Doubling the runs did not manufacture significance, and the
+tool did not pretend otherwise. An honest "we cannot tell yet, spend more runs or
+accept the uncertainty" is the correct scientific output here.
+
+## Efficiency (with vs without, N shown above)
+
+| Skill | Tokens | Wall time | Cost/run |
+|---|---|---|---|
+| error-handling (N=10) | 178k → 193k | 35s → 38s | $0.145 → $0.154 |
+| sql-parameterization | 179k → 193k | 38s → 42s | $0.145 → $0.151 |
+| type-hints | 260k → 258k | 70s → 61s | $0.185 → $0.178 |
+| negative-control | 184k → 207k | 61s → 92s | $0.147 → $0.159 |
+
+None of the token/time deltas here are statistically conclusive at these sample
+sizes; they are reported for transparency, not as claims. (Token counts are
+dominated by cheap cache reads, so per-run cost is the more meaningful figure.)
+
+## Trigger evaluation — and an honest surprise
+
+`crucible trigger` detects whether the agent invoked the **`Skill` tool** for
+a skill. Run against these three convention skills, the result was the same each
+time:
+
+| Skill | Should-fire cases | Fired (Skill tool) | Quiet cases correct |
+|---|---|---|---|
+| error-handling | 2 | 0 | 2 / 2 |
+| sql-parameterization | 2 | 0 | 2 / 2 |
+| type-hints | 2 | 0 | 2 / 2 |
+
+Taken alone that reads like "the skills never trigger". But the A/B section
+proves error-handling **does** change behaviour (N=10, p=0.033). Both are true,
+and reconciling them is the real finding:
+
+**Capturing one such run's event stream shows why.** Given "Add error handling to
+src/calc.py", sonnet-4-5 in headless mode used `Read` and `Edit` and finished the
+task directly — it never called the `Skill` tool. The skill still shaped the
+result (the with-skill condition followed the `code=` convention more often),
+because Claude Code surfaces a simple convention skill by putting it in context,
+not by making the model formally invoke it.
+
+So explicit `Skill`-tool invocation is a **lower bound on influence, not a measure
+of it.** Trigger evaluation is meaningful for skills *designed to be invoked* (a
+multi-step procedure the model calls on demand — a canary skill with a unique
+output was detected correctly in testing), and near-zero for convention skills
+that work by being in context — even when, as here, they measurably help. The
+true-negative side still holds: none of the skills fired on unrelated prompts
+(math, docstring questions), so there were no false positives.
+
+This is the kind of thing a benchmark exists to surface. The honest headline is
+not "trigger recall is 0%" — it is "for these skills, influence and explicit
+invocation are different things, and only the A/B test measures influence."
+

agent_crucible-0.4.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Elfan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_crucible-0.4.2/PKG-INFO

@@ -0,0 +1,409 @@
+Metadata-Version: 2.5
+Name: agent-crucible
+Version: 0.4.2
+Summary: Reproducible testing, benchmarking, and regression testing for Agent Skills.
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Agent Crucible
+
+Reproducible A/B testing for Agent Skills.
+
+A `SKILL.md` is a prompt you ship to other people. This tool answers the only
+question that matters about one: **does it actually make the agent better, or
+does it just feel like it does?**
+
+It runs the same task twice — once with a baseline agent, once with the skill
+installed — repeats that N times, checks the results with deterministic
+assertions, and reports the difference with a confidence interval attached.
+
+**Real results are in [BENCHMARK.md](BENCHMARK.md)** — including a skill that only
+becomes conclusive at N=10, two skills with no measurable effect, and an honest
+surprise about what trigger detection can and cannot see.
+
+```
+                    Without       With
+Task success            50%       100%
+Assertion score        0.83       1.00
+Tokens                8,700      7,100
+Time                     55s        41s
+
+IMPACT   (95% bootstrap CI on the delta)
+
+  Task success        +50.0pp   [+20.0, +80.0]   p=0.033
+~ Tokens              -23.0%    [-41.2%, +2.1%]
+
+CONFIDENCE  MEDIUM  (N=5 per condition)
+~ marks a delta not distinguishable from run-to-run noise.
+
+PER TASK  (success rate)
+
+task-001                  0%       100%
+task-002                100%       100%
+```
+
+The per-task breakdown matters: an aggregate `+50pp` driven entirely by one
+task is a different finding from one spread across all of them.
+
+## Install
+
+```bash
+pip install agent-crucible
+```
+
+Or from a clone: `pip install -e .`
+
+Requires Python 3.10+, `git`, and an authenticated [Claude Code](https://claude.com/claude-code)
+CLI on `PATH`. Tested on Linux, macOS, and Windows (CI runs the suite on all
+three across Python 3.10–3.12). Verify the agent side works before benchmarking
+anything:
+
+```bash
+claude -p "reply with OK" --output-format json
+```
+
+If that returns `"Failed to authenticate"`, log in first — every run in this tool
+shells out to that same command.
+
+## Use
+
+```bash
+crucible init my-skill        # scaffold SKILL.md + evals/
+crucible validate my-skill    # static checks, no tokens spent
+crucible test my-skill        # the A/B benchmark
+crucible report my-skill      # re-render the last run
+crucible regression my-skill  # did the latest run drop below the baseline?
+```
+
+### 1. Validate
+
+Runs before every benchmark, and refuses to spend tokens on a broken suite.
+
+```
+  [ok]   SKILL.md exists
+  [ok]   valid YAML frontmatter
+  [ok]   frontmatter has 'name'
+  [ok]   frontmatter has 'description'
+  [ok]   referenced files exist (2/2)
+  [ok]   [task-001] prompt is customised, not the scaffold placeholder
+  [warn] model 'sonnet' is a pinned version (aliases drift between releases)
+```
+
+### 2. Write eval tasks
+
+```yaml
+id: task-001
+name: Make calc errors typed
+
+prompt: |
+  The test suite in tests/ fails. Fix src/calc.py so every test passes.
+  Do not modify anything under tests/.
+
+fixture: fixtures/task-001
+
+assertions:
+  - type: command          # did it solve the task?
+    run: "python -m pytest -q"
+    expect_exit: 0
+
+  - type: file_contains    # did it follow the convention the skill teaches?
+    path: src/calc.py
+    pattern: "raise CalcError\\([^)]*code="
+
+  - type: file_unchanged   # anti-gaming: editing the tests does not count
+    path: tests/test_calc.py
+```
+
+Assertion types: `command`, `file_exists`, `file_absent`, `file_contains`,
+`file_changed`, `file_unchanged`.
+
+Before burning tokens, check two things — neither spends a cent:
+
+```bash
+crucible test my-skill --dry-run    # can the suite even measure work?
+crucible test my-skill --estimate   # what will a real run cost?
+```
+
+`--dry-run` runs the assertions against the untouched fixture. Anything that
+passes there is flagged `[TRIVIAL]` — it would pass even if the agent did
+nothing. (Assertions that check for the *absence* of something — `file_absent`,
+`file_unchanged`, or a `file_contains` with `expect: false` — are guards, and
+correctly pass on the pristine fixture.)
+
+`--estimate` projects cost and wall time as a range. With no history it is a
+coarse prior; after one real run it uses your own recorded per-run cost. A real
+`test` prints the estimate first, then runs.
+
+### 3. Benchmark
+
+```bash
+crucible test my-skill -n 10
+```
+
+## How it works
+
+Per run:
+
+1. The task fixture is copied into a throwaway temp directory.
+2. That directory is `git init`-ed and committed, which is what makes
+   `file_changed` / `file_unchanged` assertions possible.
+3. In the **with-skill** condition only, the skill is copied to
+   `.claude/skills/<name>/` inside that workspace. The `evals/` directory is
+   excluded — leaking it would hand the agent its own grading criteria.
+4. `claude -p "<prompt>" --output-format json` runs with the workspace as cwd,
+   with `--setting-sources project`. (Note: in practice this does not fully hide
+   your personal `~/.claude` skills — see "Known gap" above. They appear in both
+   conditions, so the delta stays attributable to the skill under test.)
+5. Assertions run against the resulting workspace; tokens, cost and wall time
+   come from the agent's own JSON result.
+6. The workspace is deleted (`--keep-workspace` to inspect it).
+
+Results land in `<skill>/.crucible/runs/*.json`, with one summary line per
+invocation appended to `<skill>/.crucible/history.jsonl`.
+
+## Design decisions worth knowing
+
+**No LLM judge.** Scoring is deterministic: assertions pass or they don't. An
+LLM grader adds a second noisy component on top of an already noisy agent, and
+you can no longer tell which one moved. Quality rubrics are a later addition,
+not the foundation.
+
+**Every delta carries an uncertainty estimate.** Agent runs are
+non-deterministic; the same prompt gives different token counts and sometimes
+different outcomes. A single number invites false regression alarms. Anything
+not separable from noise is printed with `~` and called inconclusive.
+
+**Task success is judged by Fisher's exact test, not by the bootstrap.** This
+was not the original design — it was forced by testing the tool against a
+deliberately noisy stub agent with a weak 60%-vs-50% effect. The percentile
+bootstrap called that a *conclusive regression* on the first attempt: five
+Bernoulli samples per condition is exactly the regime where it is
+anti-conservative. Fisher's exact test on the 2×2 outcome table has no such
+problem at small N, and still detects a real effect at N=5 (0/5 vs 5/5 gives
+p=0.008). Continuous metrics keep the bootstrap interval, at 95% to match the
+same 0.05 threshold.
+
+**Secondary metrics are not corrected for multiple comparisons.** Five metrics
+per report at a 5% threshold means roughly one report in seven contains a
+secondary claim that is not real. The report says so out loud when it makes one
+below N=10, rather than letting you believe all five numbers equally.
+
+**Confidence is labelled by sample size.** N<5 is `LOW` and says so in the
+verdict. The tool will not pretend five runs are evidence — at N=1 it reports
+the observed delta and then explicitly refuses to conclude from it.
+
+**Pin your model.** `model: sonnet` is an alias that moves between releases. A
+comparison against a run from last month is meaningless if the model changed
+underneath it. `validate` warns about this.
+
+**The baseline carries Claude Code's bundled skills.** They appear in both
+conditions, so the delta stays attributable to the skill under test, and they
+are identical across installs of the same version. `isolation: bare` removes
+them for a clean-room baseline (needs `ANTHROPIC_API_KEY`).
+
+**This is decision support, not an oracle.** `--strict` exists for CI, but
+consider making the CI job a warning rather than a hard block, and gate it on
+`SKILL.md` actually having changed — every invocation spends real tokens.
+`N tasks × N repeats × 2 conditions` agent runs is not free the way a unit test
+is free.
+
+## Examples
+
+Complete, runnable skills live in `examples/`, each mixing a "did it work"
+command assertion with a "did it the right way" content assertion, and each
+carrying trigger cases:
+
+| Skill | Domain | The convention it checks |
+|---|---|---|
+| `error-handling` | code style | raises carry a machine-readable `code=` |
+| `sql-parameterization` | security | queries use placeholders, never f-strings |
+| `type-hints` | typing | functions get annotated params + return type |
+| `negative-control` | — | **deliberately irrelevant** to task success |
+
+```bash
+crucible validate examples/sql-parameterization
+crucible test examples/sql-parameterization --dry-run
+crucible test examples/sql-parameterization --estimate
+crucible test examples/sql-parameterization -n 5
+```
+
+Each positive example is built so the difference is visible: a baseline agent
+usually makes the tests pass, but only the skilled agent reliably follows the
+convention. "Tests pass" and "did it the right way" are different questions —
+which is exactly why a single assertion is not enough.
+
+`negative-control` is the opposite on purpose: its skill only changes docstring
+wording, so it *cannot* affect whether the tests pass. Running it should report
+**no conclusive effect** — a measurement instrument you can trust reads zero when
+the effect is zero.
+
+## CI
+
+`examples/ci/skill-eval.yml` is a GitHub Actions template. It triggers only when
+`SKILL.md` or `evals/` changed, and posts the report as a PR comment instead of
+failing the build.
+
+## Validated against a live agent
+
+Run end-to-end against the real `claude` CLI, not just a stub. Two things the
+example run demonstrated, both worth internalising before you trust any number:
+
+- **A lucky small sample lies, and the tool catches it.** At N=3 the example
+  skill scored a clean 3/3 vs 0/3 — an apparent +100pp. At N=5 the same skill,
+  same prompt, scored 1/5 (+20pp, p=1.000). The effect is real but weak and
+  inconsistent; three runs happened to catch a good streak. A tool that reports
+  one number would have published "+100% success". This one reported "not
+  distinguishable from noise" and asked for more runs. That is the whole point.
+
+- **Availability is not invocation.** A skill in `.claude/skills/` is *discovered*
+  by the agent but only *used* when its description matches the task. The prompt
+  "make the tests pass" did not trigger an error-handling skill; "add proper
+  error handling following this project's conventions" did. Your eval prompt is
+  a real variable — the same skill looks useless or useful depending on it.
+
+Baseline isolation, found in the same run: the headless agent always sees the
+skills that ship with Claude Code (design, code-review, …). They are **not** your
+personal `~/.claude` skills — they are bundled with the install, so they are the
+same for anyone on the same Claude Code version, and they appear in *both*
+conditions, cancelling out of the delta. For a true clean-room baseline set
+`isolation: bare` in `evals.yaml` (runs the agent with `--bare`); it needs
+`ANTHROPIC_API_KEY`, since bare mode never reads your OAuth login. The default
+keeps the bundled skills and works with OAuth.
+
+### 4. Regression
+
+Each `test` writes a run record. `regression` compares the newest run against an
+earlier baseline — by default the most recent run whose skill content differs, so
+it answers "did my last edit make the skill worse?"
+
+```bash
+crucible regression my-skill
+```
+
+```
+With-skill success:  86%  ->  62%   (-24pp, p=0.021)
+--------------------------------------------------------
+x  REGRESSION — with-skill success dropped beyond run-to-run noise.
+```
+
+Crucially, it is judged by the same Fisher exact test as the benchmark, not by
+subtracting two scores. A drop from 100% to 80% on five runs prints
+`~ PASS (inconclusive)` with `p=1.000`, because one flipped run is exactly what
+noise produces — a naive point-vs-point check would have raised a false alarm.
+It exits `1` **only** on a conclusive drop, so a CI gate blocks a genuine
+regression without failing on the dice:
+
+```yaml
+- run: crucible test ./my-skill -n 10
+- run: crucible regression ./my-skill   # exit 1 blocks the merge
+```
+
+### 5. Trigger evaluation
+
+A skill can be excellent when used and still be worthless because the agent never
+reaches for it — or noisy because it fires on unrelated prompts. `trigger`
+measures that directly. In `evals/triggers.yaml` you label prompts:
+
+```yaml
+cases:
+  - prompt: "Add error handling to src/calc.py."
+    should_trigger: true
+  - prompt: "What is 27 times 34?"
+    should_trigger: false
+```
+
+```bash
+crucible trigger my-skill
+```
+
+```
+  Precision (when it fired, was it right): 100%   FP=0
+  Recall (of prompts that should fire it):  75%   FN=1
+  ! 1 prompt that should have used the skill did not — available but not reached for.
+```
+
+Detection is deterministic, not an LLM judging "was it probably used": the run
+uses `--output-format stream-json`, and a trigger is a `Skill` tool-use event for
+this skill in the stream.
+
+**Scope, learned from real runs (see [BENCHMARK.md](BENCHMARK.md)):** this
+measures *explicit `Skill`-tool invocation*. That is the right signal for skills
+designed to be invoked as a procedure. But Claude Code activates a simple
+*convention* skill by putting it in context, not by making the model call the
+Skill tool — so such a skill can measurably change behaviour (proven by the A/B
+test) while its explicit-invocation rate is zero. Read trigger recall as a lower
+bound on influence, not as "the skill did nothing"; the A/B test is what measures
+influence. The false-positive side is still meaningful: a skill firing on
+unrelated prompts is a real defect this catches.
+
+## Real results
+
+Four skills were run against a live agent (Claude Code, `claude-sonnet-4-5`,
+N=5). The full write-up with methodology and raw data is in
+[BENCHMARK.md](BENCHMARK.md); the short version:
+
+- **No skill reached statistical significance on task success at N=5.** Two
+  showed promising but inconclusive swings (0→40%, 40→80%, both p≈0.5); the tool
+  reported them as inconclusive rather than publishing the point estimate.
+- **The negative control read exactly zero** (0.00 delta, p=1.000) — the
+  instrument does not manufacture improvements.
+- **The only conclusive deltas were token *costs*** — including a skill whose
+  advice the baseline agent already followed, so it added tokens and nothing else.
+- **Trigger recall was 0–50%**: the skills often were not invoked even when
+  clearly relevant, which likely explains the weak outcome effects. Availability
+  is not invocation.
+
+This is the intended behaviour: run it on your own skill and it will tell you the
+truth, including "no effect" or "costs more than it helps."
+
+## Reproducibility
+
+Every `test` writes a full run record to `.crucible/runs/*.json` — not just
+the summary, but every per-run outcome, and an `environment` block capturing what
+the numbers depend on: tool version, agent (`claude`) version, model, OS,
+Python version, and separate fingerprints for the **skill** and the **eval
+suite**. The suite fingerprint exists because a result can flip when you reword a
+prompt without touching `SKILL.md`; comparing across a changed suite fingerprint
+is comparing two different experiments. `crucible report` prints the
+environment block so a reader knows the conditions.
+
+## Limitations
+
+Read these before quoting any number from this tool.
+
+- **Results are model-, harness-, task-, and environment-dependent.** A correct
+  claim is *"on this eval suite, under this model and Claude Code version, the
+  skill raised task success by N percentage points"* — **not** *"this skill makes
+  the AI N% better"*. The run record stores exactly those conditions so the
+  scoped claim is the easy one to make.
+- **A benchmark is only as good as its eval suite.** The tool measures whether
+  the agent satisfied *your* assertions on *your* tasks. A weak or unrepresentative
+  suite produces a confident, meaningless number. `--dry-run` guards against
+  assertions that pass on the untouched fixture; nothing guards against tasks that
+  don't represent real use.
+- **Cost.** Each run is a real agent session spending real tokens. `N tasks × N
+  repeats × 2 conditions` is not free the way a unit test is; `--estimate` prints
+  the projected spend before you commit.
+- **Small N is a smoke test, not evidence.** Below ~5 runs per condition the tool
+  labels the result `LOW` confidence and refuses to call effects conclusive, on
+  purpose.
+
+## Status
+
+Claude Code only: `init`, `validate`, `test` (with `--dry-run`, `--estimate`,
+`--strict`), `report`, `regression`, `trigger`. Tested on Linux/macOS/Windows
+across Python 3.10–3.12; the test suite stubs the agent, so CI needs no API key.
+
+Deliberately not built yet: LLM-as-judge quality scoring, cross-agent
+`benchmark` / `compare`, and `audit` for skill security. These are held back on
+purpose — the differentiator is deterministic, statistically honest A/B testing,
+not breadth.
+
+## License
+
+MIT