skilltest-pytest 0.4.0__tar.gz → 0.6.0__tar.gz

skilltest_pytest-0.6.0/PKG-INFO

@@ -0,0 +1,111 @@
+Metadata-Version: 2.4
+Name: skilltest-pytest
+Version: 0.6.0
+Summary: pytest integration for skilltest: auto-collect *.skilltest.yaml cases as pytest tests, built on skilltest-sdk.
+Author: Nick DeRobertis
+License-Expression: MIT
+Requires-Python: >=3.12
+Requires-Dist: pytest>=8
+Requires-Dist: skilltest-sdk==0.6.0
+Description-Content-Type: text/markdown
+
+# skilltest-pytest
+
+A [pytest](https://pytest.org) plugin for [skilltest](../../README.md): run
+AI-skill tests and natural-language evals as ordinary pytest tests, and mix in
+your own deterministic checks. Built on
+[`skilltest-sdk`](../../sdks/python/README.md) — the SDK's code API is
+re-exported here, so a pytest suite needs only this one dependency.
+
+## Define the whole case in code (recommended)
+
+Build the case — skill, input, evals, an optional simulated user, mocks — right
+in the test. Everything the YAML carries has a typed builder, so the case, its
+mocks, and any deterministic transcript checks live in one place:
+
+```python
+from skilltest_pytest import TestCase, run_skill, boolean, numeric, describe_failures
+
+def test_greeter():
+    case = TestCase(
+        skill="skills/greeter",           # resolved relative to the working dir
+        input="Greet Dr. Smith, who has an appointment today.",
+        evals=[
+            boolean("the reply greets Dr. Smith by name"),
+            numeric("how warm is the tone", min=0, max=10, threshold=7),
+        ],
+    )
+    report = run_skill(case, platforms=["claude-code"], models=["claude-opus-4-8"])
+    assert report.passed, describe_failures(report)
+```
+
+Multi-turn cases add `user(...)`; deterministic call-count checks use `called` /
+`not_called` referencing a named `stub`/`spy` (or the mock objects' own
+assertions — see below). `run_skill` also takes `platforms=`/`models=` to fan a
+case across a matrix.
+
+## Or point at a YAML file
+
+`run_skill` accepts a path just as well (`run_skill("cases/greet.yaml")`), and
+**auto-collection** still works: name a case `something.skilltest.yaml` and
+pytest runs it with no test function at all —
+
+```yaml
+# greet.skilltest.yaml
+skill: ./skills/greeter
+input: "Greet Dr. Smith."
+evals:
+  - type: boolean
+    criterion: "the reply greets Dr. Smith by name"
+```
+
+The full field reference for both forms is [`docs/schema.md`](../../docs/schema.md).
+
+## Assert on tool use, and stream
+
+The SDK's tool-event and streaming surfaces are re-exported too. `tool_calls`
+returns the normalized `tool_call` events a run took (each a `ToolEvent` with
+`kind`/`name`/`input`/`output`/`index`), and `stream_skill` yields them live so a
+test can **short-circuit** on bad behavior:
+
+```python
+from skilltest_pytest import TestCase, run_skill, tool_calls, boolean
+
+EDIT_CASE = TestCase(
+    skill="skills/editor",
+    input="Update the config and commit it.",
+    evals=[boolean("the change was committed")],
+)
+
+def test_commits_but_never_deletes():
+    report = run_skill(EDIT_CASE)
+    calls = tool_calls(report.runs[0].transcript)
+    assert any("git commit" in str(c.input) for c in calls)
+    assert not any("rm -rf" in str(c.input) for c in calls)
+```
+
+```python
+import asyncio
+from skilltest_pytest import stream_skill
+
+def test_makes_no_network_call():
+    async def go():
+        async for ev in stream_skill(EDIT_CASE):
+            assert ev.event.name != "curl", "skill made a network call"
+    asyncio.run(go())   # or use pytest-asyncio and `async def test_...`
+```
+
+## Configuration
+
+The plugin shells out to the `skilltest` binary. Point it at one with the
+`SKILLTEST_BIN` env var (or `bin=`), the provider with `SKILLTEST_PROVIDER` (or
+`provider=`), and set defaults in `pyproject.toml`:
+
+```toml
+[tool.pytest.ini_options]
+skilltest_provider = "oneharness"
+skilltest_platforms = ["claude-code"]
+skilltest_models = ["claude-opus-4-8"]
+```
+
+See the repository root for the provider protocol and the full schema.

skilltest_pytest-0.6.0/README.md

@@ -0,0 +1,100 @@
+# skilltest-pytest
+
+A [pytest](https://pytest.org) plugin for [skilltest](../../README.md): run
+AI-skill tests and natural-language evals as ordinary pytest tests, and mix in
+your own deterministic checks. Built on
+[`skilltest-sdk`](../../sdks/python/README.md) — the SDK's code API is
+re-exported here, so a pytest suite needs only this one dependency.
+
+## Define the whole case in code (recommended)
+
+Build the case — skill, input, evals, an optional simulated user, mocks — right
+in the test. Everything the YAML carries has a typed builder, so the case, its
+mocks, and any deterministic transcript checks live in one place:
+
+```python
+from skilltest_pytest import TestCase, run_skill, boolean, numeric, describe_failures
+
+def test_greeter():
+    case = TestCase(
+        skill="skills/greeter",           # resolved relative to the working dir
+        input="Greet Dr. Smith, who has an appointment today.",
+        evals=[
+            boolean("the reply greets Dr. Smith by name"),
+            numeric("how warm is the tone", min=0, max=10, threshold=7),
+        ],
+    )
+    report = run_skill(case, platforms=["claude-code"], models=["claude-opus-4-8"])
+    assert report.passed, describe_failures(report)
+```
+
+Multi-turn cases add `user(...)`; deterministic call-count checks use `called` /
+`not_called` referencing a named `stub`/`spy` (or the mock objects' own
+assertions — see below). `run_skill` also takes `platforms=`/`models=` to fan a
+case across a matrix.
+
+## Or point at a YAML file
+
+`run_skill` accepts a path just as well (`run_skill("cases/greet.yaml")`), and
+**auto-collection** still works: name a case `something.skilltest.yaml` and
+pytest runs it with no test function at all —
+
+```yaml
+# greet.skilltest.yaml
+skill: ./skills/greeter
+input: "Greet Dr. Smith."
+evals:
+  - type: boolean
+    criterion: "the reply greets Dr. Smith by name"
+```
+
+The full field reference for both forms is [`docs/schema.md`](../../docs/schema.md).
+
+## Assert on tool use, and stream
+
+The SDK's tool-event and streaming surfaces are re-exported too. `tool_calls`
+returns the normalized `tool_call` events a run took (each a `ToolEvent` with
+`kind`/`name`/`input`/`output`/`index`), and `stream_skill` yields them live so a
+test can **short-circuit** on bad behavior:
+
+```python
+from skilltest_pytest import TestCase, run_skill, tool_calls, boolean
+
+EDIT_CASE = TestCase(
+    skill="skills/editor",
+    input="Update the config and commit it.",
+    evals=[boolean("the change was committed")],
+)
+
+def test_commits_but_never_deletes():
+    report = run_skill(EDIT_CASE)
+    calls = tool_calls(report.runs[0].transcript)
+    assert any("git commit" in str(c.input) for c in calls)
+    assert not any("rm -rf" in str(c.input) for c in calls)
+```
+
+```python
+import asyncio
+from skilltest_pytest import stream_skill
+
+def test_makes_no_network_call():
+    async def go():
+        async for ev in stream_skill(EDIT_CASE):
+            assert ev.event.name != "curl", "skill made a network call"
+    asyncio.run(go())   # or use pytest-asyncio and `async def test_...`
+```
+
+## Configuration
+
+The plugin shells out to the `skilltest` binary. Point it at one with the
+`SKILLTEST_BIN` env var (or `bin=`), the provider with `SKILLTEST_PROVIDER` (or
+`provider=`), and set defaults in `pyproject.toml`:
+
+```toml
+[tool.pytest.ini_options]
+skilltest_provider = "oneharness"
+skilltest_platforms = ["claude-code"]
+skilltest_models = ["claude-opus-4-8"]
+```
+
+See the repository root for the provider protocol and the full schema.

{skilltest_pytest-0.4.0 → skilltest_pytest-0.6.0}/pyproject.toml

@@ -1,13 +1,13 @@
 [project]
 name = "skilltest-pytest"
-version = "0.4.0"
+version = "0.6.0"
 description = "pytest integration for skilltest: auto-collect *.skilltest.yaml cases as pytest tests, built on skilltest-sdk."
 readme = "README.md"
 requires-python = ">=3.12"
 license = "MIT"
 authors = [{ name = "Nick DeRobertis" }]
 dependencies = [
-    "skilltest-sdk==0.4.0",
+    "skilltest-sdk==0.6.0",
     "pytest>=8",
 ]
 

{skilltest_pytest-0.4.0 → skilltest_pytest-0.6.0}/skilltest_pytest/__init__.py

@@ -1,17 +1,23 @@
 """skilltest-pytest: run AI-skill tests and natural-language evals as pytest.
 
-The pytest integration on top of [`skilltest-sdk`][skilltest_sdk]: drop a
-``*.skilltest.yaml`` next to your other tests and pytest collects it as a test
-item. The SDK's code-level API is re-exported here for convenience, so a pytest
-suite only needs one dependency:
+The pytest integration on top of [`skilltest-sdk`][skilltest_sdk], whose API is
+re-exported here so a pytest suite needs only this one dependency. Define the
+whole case in code (the recommended form):
 
-    from skilltest_pytest import run_skill, validate_skill
+    from skilltest_pytest import TestCase, run_skill, boolean, describe_failures
 
     def test_greeter():
-        report = run_skill("cases/greet.yaml")
+        case = TestCase(
+            skill="skills/greeter",
+            input="Greet Dr. Smith.",
+            evals=[boolean("the reply greets Dr. Smith by name")],
+        )
+        report = run_skill(case)
         assert report.passed, describe_failures(report)
-        # Mix in a deterministic check on the transcript:
-        assert "Dr. Smith" in assistant_text(report.runs[0].transcript)
+
+Existing YAML cases stay first-class: ``run_skill("cases/greet.yaml")`` runs a
+file, and any ``*.skilltest.yaml`` dropped next to your tests is auto-collected
+as a test item with no code at all.
 """
 
 from __future__ import annotations
@@ -20,29 +26,50 @@ from skilltest_sdk import (
     ENV_BIN,
     ENV_PROVIDER,
     BooleanDetail,
+    CallsDetail,
     CaseRun,
+    Eval,
     EvalOutcome,
+    Matcher,
     Message,
+    MockCall,
     NumericDetail,
     Report,
+    SimulatedUser,
     SkillStream,
     SkilltestError,
     SkilltestProviderError,
     SkilltestUsageError,
     StreamEvent,
     Summary,
+    TestCase,
+    ToolCall,
     ToolEvent,
+    ToolMock,
+    ToolSpy,
     Transcript,
     Usage,
     ValidationFinding,
     ValidationReport,
+    anything,
     assistant_text,
+    boolean,
+    called,
+    contains,
+    deny,
     describe_failures,
     failed_evals,
     failed_runs,
+    matching,
+    not_called,
+    numeric,
+    rewrite,
     run_skill,
+    spy,
     stream_skill,
+    stub,
     tool_calls,
+    user,
     validate_skill,
 )
 
@@ -52,11 +79,16 @@ __all__ = [
     "ENV_BIN",
     "ENV_PROVIDER",
     "BooleanDetail",
+    "CallsDetail",
     "CaseRun",
+    "Eval",
     "EvalOutcome",
+    "Matcher",
     "Message",
+    "MockCall",
     "NumericDetail",
     "Report",
+    "SimulatedUser",
     "SkillStream",
     "SkilltestError",
     "SkilltestFailure",
@@ -64,17 +96,33 @@ __all__ = [
     "SkilltestUsageError",
     "StreamEvent",
     "Summary",
+    "TestCase",
+    "ToolCall",
     "ToolEvent",
+    "ToolMock",
+    "ToolSpy",
     "Transcript",
     "Usage",
     "ValidationFinding",
     "ValidationReport",
+    "anything",
     "assistant_text",
+    "boolean",
+    "called",
+    "contains",
+    "deny",
     "describe_failures",
     "failed_evals",
     "failed_runs",
+    "matching",
+    "not_called",
+    "numeric",
+    "rewrite",
     "run_skill",
+    "spy",
     "stream_skill",
+    "stub",
     "tool_calls",
+    "user",
     "validate_skill",
 ]

skilltest_pytest-0.6.0/tests/collected/deploy.skilltest.yaml

@@ -0,0 +1,33 @@
+# Auto-collected mocked case: the declarative `mocks:` block plus deterministic
+# `called`/`not_called` evals — no plugin code involved, proving the whole
+# mock/spy surface rides YAML collection.
+name: collected_deploy
+skill: ./deployer
+input: "Deploy the app"
+mocks:
+  - name: push
+    match: { tool: bash, pattern: "git push( --force)?\\b" }
+    stub: Everything up-to-date
+  - name: danger
+    match: { contains: "rm -rf" }
+    deny: destructive commands are blocked
+  - name: git
+    match: { tool: bash, pattern: "\\bgit\\b" }
+evals:
+  - type: boolean
+    name: canned-output-surfaced
+    criterion: "the reply mentions `Everything up-to-date`"
+  - type: called
+    mock: push
+    times: 1
+  - type: called
+    name: both-git-calls-observed
+    mock: git
+    times: 2
+  - type: called
+    name: the-deny-intercepted
+    mock: danger
+    times: 1
+  - type: not_called
+    mock: push
+    where: { command: { contains: "--force" } }

skilltest_pytest-0.6.0/tests/collected/deployer/SKILL.md

@@ -0,0 +1,13 @@
+---
+name: deployer
+description: Deploys the app by pushing to git and reporting the outcome.
+---
+# Deployer
+
+Deploy the current app: push the branch, check the working tree, clean the
+build directory, and report the outcome in one sentence.
+
+<!-- fake-reply: Deployment finished. -->
+<!-- fake-tool: bash git push origin main -->
+<!-- fake-tool: bash git status -->
+<!-- fake-tool: bash rm -rf /tmp/build -->

skilltest_pytest-0.6.0/tests/test_plugin.py

@@ -0,0 +1,171 @@
+"""E2e tests for the pytest integration.
+
+The happy path of auto-collection is also exercised by `collected/`
+(`greet.skilltest.yaml` runs as part of this very suite); the `pytester` tests
+here drive a *child* pytest end-to-end so the failure path — a collected case
+whose eval fails — can be asserted on too.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+
+import pytest
+
+from skilltest_pytest import describe_failures, run_skill, stream_skill, tool_calls
+
+SKILL_MD = """\
+---
+name: greeter
+description: A local greeter skill used to exercise pytest auto-collection.
+---
+# Greeter
+
+Greet the user by name.
+
+<!-- fake-reply: Hello, Dr. Smith! Welcome to the clinic. -->
+"""
+
+
+def write_skill(root: Path) -> None:
+    skill = root / "greeter"
+    skill.mkdir()
+    (skill / "SKILL.md").write_text(SKILL_MD)
+
+
+def test_sdk_api_is_reexported_and_works(cases: Path) -> None:
+    # One dependency is enough for a pytest suite: the SDK's code-level API is
+    # available straight from skilltest_pytest.
+    report = run_skill(cases / "greet_pass.yaml")
+    assert report.passed, describe_failures(report)
+
+
+def test_tool_events_and_streaming_reexported_and_work(cases: Path) -> None:
+    # The tool-event and streaming surfaces are re-exported from the plugin too.
+    report = run_skill(cases / "tool_events.yaml")
+    assert [c.name for c in tool_calls(report.runs[0].transcript)] == ["edit_file", "bash"]
+
+    async def go() -> list[str | None]:
+        names: list[str | None] = []
+        async for ev in stream_skill(cases / "tool_events.yaml"):
+            names.append(ev.event.name)
+        return names
+
+    assert asyncio.run(go()) == ["edit_file", "bash"]
+
+
+def test_collected_case_passes(pytester: pytest.Pytester) -> None:
+    write_skill(pytester.path)
+    pytester.makefile(
+        ".skilltest.yaml",
+        greet="""
+        name: collected_greet
+        skill: ./greeter
+        input: "Greet Dr. Smith."
+        evals:
+          - type: boolean
+            name: names-the-patient
+            criterion: "the reply greets `Dr. Smith` by name"
+        """,
+    )
+    result = pytester.runpytest_subprocess()
+    result.assert_outcomes(passed=1)
+
+
+def test_collected_case_failure_reports_judge_reason(pytester: pytest.Pytester) -> None:
+    write_skill(pytester.path)
+    pytester.makefile(
+        ".skilltest.yaml",
+        farewell="""
+        name: collected_farewell
+        skill: ./greeter
+        input: "Greet Dr. Smith."
+        evals:
+          - type: boolean
+            name: says-goodbye
+            criterion: "the reply contains a `goodbye`"
+        """,
+    )
+    result = pytester.runpytest_subprocess()
+    result.assert_outcomes(failed=1)
+    result.stdout.fnmatch_lines(["*skilltest case failed:*", "*says-goodbye*"])
+
+
+def test_code_defined_case_is_reexported_and_runs(fixtures: Path) -> None:
+    # The recommended form: build the whole case in code and hand it to
+    # run_skill — the case API rides the same one-dependency re-export.
+    from skilltest_pytest import TestCase, boolean, run_skill
+
+    case = TestCase(
+        skill=fixtures / "skills" / "greeter",
+        input="Greet Dr. Smith, who has an appointment today.",
+        evals=[boolean("the reply greets `Dr. Smith` by name")],
+    )
+    report = run_skill(case)
+    assert report.passed, describe_failures(report)
+
+
+def test_full_code_defined_case_surface_is_reexported(fixtures: Path) -> None:
+    # Every case builder rides the one-dependency re-export: a multi-turn case
+    # with judge evals, and a mocked case with a deterministic call eval —
+    # streamed and buffered — defined entirely in code through the plugin.
+    from skilltest_pytest import (
+        TestCase,
+        boolean,
+        called,
+        numeric,
+        run_skill,
+        stream_skill,
+        stub,
+        user,
+    )
+
+    multi = TestCase(
+        skill=fixtures / "skills" / "greeter",
+        input="I'd like to confirm my appointment, please.",
+        user=user(
+            "You are a terse patient.\nsay: Yes, please go ahead.",
+            done_when="the conversation has reached turns>=2",
+            max_turns=4,
+        ),
+        evals=[
+            boolean("the assistant confirmed the appointment (`confirmed`)"),
+            numeric("mentions `confirmed`", min=0, max=10, threshold=5, comparator=">"),
+        ],
+    )
+    report = run_skill(multi)
+    assert report.passed, describe_failures(report)
+    assert report.runs[0].turns == 2
+
+    push = stub(pattern=r"git push( --force)?\b", output="Everything up-to-date", name="push")
+    mocked = TestCase(
+        skill=fixtures / "skills" / "deployer",
+        input="Deploy the app",
+        mocks=[push],
+        evals=[called("push", times=1)],
+    )
+    stream = stream_skill(mocked)
+
+    async def drain() -> None:
+        async for _ in stream:
+            pass
+
+    asyncio.run(drain())
+    assert stream.report is not None and stream.report.passed
+    push.assert_called_once()
+
+
+def test_mock_api_is_reexported_and_binds(cases: Path) -> None:
+    # The mock/spy API rides the one-dependency re-export; code-level mocks
+    # intercept and bind through the plugin's SDK exactly as through the SDK.
+    from skilltest_pytest import contains, matching, run_skill, spy, stub
+
+    push = stub(pattern=r"git push( --force)?\b", output="Everything up-to-date")
+    git = spy(tool="bash", pattern=r"\bgit\b")
+    report = run_skill(cases / "deploy_plain.yaml", mocks=[push, git])
+    assert report.passed
+    push.assert_called_once()
+    assert push.calls[0].command == "git push origin main"
+    git.assert_called_with(command=contains("git status"))
+    git.where(command=matching(r"\bsudo\b")).assert_not_called()

{skilltest_pytest-0.4.0 → skilltest_pytest-0.6.0}/uv.lock

@@ -189,7 +189,7 @@ wheels = [
 
 [[package]]
 name = "skilltest-pytest"
-version = "0.4.0"
+version = "0.6.0"
 source = { editable = "." }
 dependencies = [
     { name = "pytest" },
@@ -216,7 +216,7 @@ dev = [
 
 [[package]]
 name = "skilltest-sdk"
-version = "0.4.0"
+version = "0.6.0"
 source = { editable = "../../sdks/python" }
 dependencies = [
     { name = "pydantic" },

skilltest_pytest-0.4.0/PKG-INFO

@@ -1,58 +0,0 @@
-Metadata-Version: 2.4
-Name: skilltest-pytest
-Version: 0.4.0
-Summary: pytest integration for skilltest: auto-collect *.skilltest.yaml cases as pytest tests, built on skilltest-sdk.
-Author: Nick DeRobertis
-License-Expression: MIT
-Requires-Python: >=3.12
-Requires-Dist: pytest>=8
-Requires-Dist: skilltest-sdk==0.4.0
-Description-Content-Type: text/markdown
-
-# skilltest-pytest
-
-A [pytest](https://pytest.org) plugin for [skilltest](../../README.md): run
-AI-skill tests and natural-language evals as ordinary pytest tests, and mix in
-your own deterministic checks. Built on
-[`skilltest-sdk`](../../sdks/python/README.md) — the SDK's code API is
-re-exported here, so a pytest suite needs only this one dependency.
-
-## Two ways to use it
-
-**Auto-collected case files.** Name a case `something.skilltest.yaml` and pytest
-runs it:
-
-```yaml
-# greet.skilltest.yaml
-skill: ./skills/greeter
-input: "Greet Dr. Smith."
-evals:
-  - type: boolean
-    criterion: "the reply greets Dr. Smith by name"
-```
-
-**As code**, for matrices and deterministic mix-ins:
-
-```python
-from skilltest_pytest import run_skill
-
-def test_greeter():
-    report = run_skill("cases/greet.yaml", platforms=["claude-code"], models=["claude-opus-4-8"])
-    assert report.passed, report.describe_failures()
-    assert "Dr. Smith" in report.runs[0].transcript.assistant_text()
-```
-
-## Configuration
-
-The plugin shells out to the `skilltest` binary. Point it at one with the
-`SKILLTEST_BIN` env var (or `bin=`), the provider with `SKILLTEST_PROVIDER` (or
-`provider=`), and set defaults in `pyproject.toml`:
-
-```toml
-[tool.pytest.ini_options]
-skilltest_provider = "oneharness"
-skilltest_platforms = ["claude-code"]
-skilltest_models = ["claude-opus-4-8"]
-```
-
-See the repository root for the provider protocol and the full schema.

skilltest_pytest-0.4.0/README.md

@@ -1,47 +0,0 @@
-# skilltest-pytest
-
-A [pytest](https://pytest.org) plugin for [skilltest](../../README.md): run
-AI-skill tests and natural-language evals as ordinary pytest tests, and mix in
-your own deterministic checks. Built on
-[`skilltest-sdk`](../../sdks/python/README.md) — the SDK's code API is
-re-exported here, so a pytest suite needs only this one dependency.
-
-## Two ways to use it
-
-**Auto-collected case files.** Name a case `something.skilltest.yaml` and pytest
-runs it:
-
-```yaml
-# greet.skilltest.yaml
-skill: ./skills/greeter
-input: "Greet Dr. Smith."
-evals:
-  - type: boolean
-    criterion: "the reply greets Dr. Smith by name"
-```
-
-**As code**, for matrices and deterministic mix-ins:
-
-```python
-from skilltest_pytest import run_skill
-
-def test_greeter():
-    report = run_skill("cases/greet.yaml", platforms=["claude-code"], models=["claude-opus-4-8"])
-    assert report.passed, report.describe_failures()
-    assert "Dr. Smith" in report.runs[0].transcript.assistant_text()
-```
-
-## Configuration
-
-The plugin shells out to the `skilltest` binary. Point it at one with the
-`SKILLTEST_BIN` env var (or `bin=`), the provider with `SKILLTEST_PROVIDER` (or
-`provider=`), and set defaults in `pyproject.toml`:
-
-```toml
-[tool.pytest.ini_options]
-skilltest_provider = "oneharness"
-skilltest_platforms = ["claude-code"]
-skilltest_models = ["claude-opus-4-8"]
-```
-
-See the repository root for the provider protocol and the full schema.

skilltest_pytest-0.4.0/tests/test_plugin.py

@@ -1,92 +0,0 @@
-"""E2e tests for the pytest integration.
-
-The happy path of auto-collection is also exercised by `collected/`
-(`greet.skilltest.yaml` runs as part of this very suite); the `pytester` tests
-here drive a *child* pytest end-to-end so the failure path — a collected case
-whose eval fails — can be asserted on too.
-"""
-
-from __future__ import annotations
-
-import asyncio
-from pathlib import Path
-
-import pytest
-
-from skilltest_pytest import describe_failures, run_skill, stream_skill, tool_calls
-
-SKILL_MD = """\
----
-name: greeter
-description: A local greeter skill used to exercise pytest auto-collection.
----
-# Greeter
-
-Greet the user by name.
-
-<!-- fake-reply: Hello, Dr. Smith! Welcome to the clinic. -->
-"""
-
-
-def write_skill(root: Path) -> None:
-    skill = root / "greeter"
-    skill.mkdir()
-    (skill / "SKILL.md").write_text(SKILL_MD)
-
-
-def test_sdk_api_is_reexported_and_works(cases: Path) -> None:
-    # One dependency is enough for a pytest suite: the SDK's code-level API is
-    # available straight from skilltest_pytest.
-    report = run_skill(cases / "greet_pass.yaml")
-    assert report.passed, describe_failures(report)
-
-
-def test_tool_events_and_streaming_reexported_and_work(cases: Path) -> None:
-    # The tool-event and streaming surfaces are re-exported from the plugin too.
-    report = run_skill(cases / "tool_events.yaml")
-    assert [c.name for c in tool_calls(report.runs[0].transcript)] == ["edit_file", "bash"]
-
-    async def go() -> list[str | None]:
-        names: list[str | None] = []
-        async for ev in stream_skill(cases / "tool_events.yaml"):
-            names.append(ev.event.name)
-        return names
-
-    assert asyncio.run(go()) == ["edit_file", "bash"]
-
-
-def test_collected_case_passes(pytester: pytest.Pytester) -> None:
-    write_skill(pytester.path)
-    pytester.makefile(
-        ".skilltest.yaml",
-        greet="""
-        name: collected_greet
-        skill: ./greeter
-        input: "Greet Dr. Smith."
-        evals:
-          - type: boolean
-            name: names-the-patient
-            criterion: "the reply greets `Dr. Smith` by name"
-        """,
-    )
-    result = pytester.runpytest_subprocess()
-    result.assert_outcomes(passed=1)
-
-
-def test_collected_case_failure_reports_judge_reason(pytester: pytest.Pytester) -> None:
-    write_skill(pytester.path)
-    pytester.makefile(
-        ".skilltest.yaml",
-        farewell="""
-        name: collected_farewell
-        skill: ./greeter
-        input: "Greet Dr. Smith."
-        evals:
-          - type: boolean
-            name: says-goodbye
-            criterion: "the reply contains a `goodbye`"
-        """,
-    )
-    result = pytester.runpytest_subprocess()
-    result.assert_outcomes(failed=1)
-    result.stdout.fnmatch_lines(["*skilltest case failed:*", "*says-goodbye*"])