skilltest-pytest 0.5.0__tar.gz → 0.7.0__tar.gz

{skilltest_pytest-0.5.0 → skilltest_pytest-0.7.0}/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.4
 Name: skilltest-pytest
-Version: 0.5.0
+Version: 0.7.0
 Summary: pytest integration for skilltest: auto-collect *.skilltest.yaml cases as pytest tests, built on skilltest-sdk.
 Author: Nick DeRobertis
 License-Expression: MIT
 Requires-Python: >=3.12
 Requires-Dist: pytest>=8
-Requires-Dist: skilltest-sdk==0.5.0
+Requires-Dist: skilltest-sdk==0.7.0
 Description-Content-Type: text/markdown
 
 # skilltest-pytest
@@ -17,10 +17,38 @@ your own deterministic checks. Built on
 [`skilltest-sdk`](../../sdks/python/README.md) — the SDK's code API is
 re-exported here, so a pytest suite needs only this one dependency.
 
-## Two ways to use it
+## Define the whole case in code (recommended)
 
-**Auto-collected case files.** Name a case `something.skilltest.yaml` and pytest
-runs it:
+Build the case — skill, input, evals, an optional simulated user, mocks — right
+in the test. Everything the YAML carries has a typed builder, so the case, its
+mocks, and any deterministic transcript checks live in one place:
+
+```python
+from skilltest_pytest import TestCase, run_skill, boolean, numeric, describe_failures
+
+def test_greeter():
+    case = TestCase(
+        skill="skills/greeter",           # resolved relative to the working dir
+        input="Greet Dr. Smith, who has an appointment today.",
+        evals=[
+            boolean("the reply greets Dr. Smith by name"),
+            numeric("how warm is the tone", min=0, max=10, threshold=7),
+        ],
+    )
+    report = run_skill(case, platforms=["claude-code"], models=["claude-opus-4-8"])
+    assert report.passed, describe_failures(report)
+```
+
+Multi-turn cases add `user(...)`; deterministic call-count checks use `called` /
+`not_called` taking the `stub`/`spy` object itself or a name you gave it (or
+use the mock objects' own assertions — see below). `run_skill` also takes `platforms=`/`models=` to fan a
+case across a matrix.
+
+## Or point at a YAML file
+
+`run_skill` accepts a path just as well (`run_skill("cases/greet.yaml")`), and
+**auto-collection** still works: name a case `something.skilltest.yaml` and
+pytest runs it with no test function at all —
 
 ```yaml
 # greet.skilltest.yaml
@@ -31,16 +59,7 @@ evals:
     criterion: "the reply greets Dr. Smith by name"
 ```
 
-**As code**, for matrices and deterministic mix-ins:
-
-```python
-from skilltest_pytest import run_skill, describe_failures, assistant_text
-
-def test_greeter():
-    report = run_skill("cases/greet.yaml", platforms=["claude-code"], models=["claude-opus-4-8"])
-    assert report.passed, describe_failures(report)
-    assert "Dr. Smith" in assistant_text(report.runs[0].transcript)
-```
+The full field reference for both forms is [`docs/schema.md`](../../docs/schema.md).
 
 ## Assert on tool use, and stream
 
@@ -50,10 +69,16 @@ returns the normalized `tool_call` events a run took (each a `ToolEvent` with
 test can **short-circuit** on bad behavior:
 
 ```python
-from skilltest_pytest import run_skill, tool_calls
+from skilltest_pytest import TestCase, run_skill, tool_calls, boolean
+
+EDIT_CASE = TestCase(
+    skill="skills/editor",
+    input="Update the config and commit it.",
+    evals=[boolean("the change was committed")],
+)
 
 def test_commits_but_never_deletes():
-    report = run_skill("cases/edit.skilltest.yaml")
+    report = run_skill(EDIT_CASE)
     calls = tool_calls(report.runs[0].transcript)
     assert any("git commit" in str(c.input) for c in calls)
     assert not any("rm -rf" in str(c.input) for c in calls)
@@ -65,7 +90,7 @@ from skilltest_pytest import stream_skill
 
 def test_makes_no_network_call():
     async def go():
-        async for ev in stream_skill("cases/edit.skilltest.yaml"):
+        async for ev in stream_skill(EDIT_CASE):
             assert ev.event.name != "curl", "skill made a network call"
     asyncio.run(go())   # or use pytest-asyncio and `async def test_...`
 ```

{skilltest_pytest-0.5.0 → skilltest_pytest-0.7.0}/README.md

@@ -6,10 +6,38 @@ your own deterministic checks. Built on
 [`skilltest-sdk`](../../sdks/python/README.md) — the SDK's code API is
 re-exported here, so a pytest suite needs only this one dependency.
 
-## Two ways to use it
+## Define the whole case in code (recommended)
 
-**Auto-collected case files.** Name a case `something.skilltest.yaml` and pytest
-runs it:
+Build the case — skill, input, evals, an optional simulated user, mocks — right
+in the test. Everything the YAML carries has a typed builder, so the case, its
+mocks, and any deterministic transcript checks live in one place:
+
+```python
+from skilltest_pytest import TestCase, run_skill, boolean, numeric, describe_failures
+
+def test_greeter():
+    case = TestCase(
+        skill="skills/greeter",           # resolved relative to the working dir
+        input="Greet Dr. Smith, who has an appointment today.",
+        evals=[
+            boolean("the reply greets Dr. Smith by name"),
+            numeric("how warm is the tone", min=0, max=10, threshold=7),
+        ],
+    )
+    report = run_skill(case, platforms=["claude-code"], models=["claude-opus-4-8"])
+    assert report.passed, describe_failures(report)
+```
+
+Multi-turn cases add `user(...)`; deterministic call-count checks use `called` /
+`not_called` taking the `stub`/`spy` object itself or a name you gave it (or
+use the mock objects' own assertions — see below). `run_skill` also takes `platforms=`/`models=` to fan a
+case across a matrix.
+
+## Or point at a YAML file
+
+`run_skill` accepts a path just as well (`run_skill("cases/greet.yaml")`), and
+**auto-collection** still works: name a case `something.skilltest.yaml` and
+pytest runs it with no test function at all —
 
 ```yaml
 # greet.skilltest.yaml
@@ -20,16 +48,7 @@ evals:
     criterion: "the reply greets Dr. Smith by name"
 ```
 
-**As code**, for matrices and deterministic mix-ins:
-
-```python
-from skilltest_pytest import run_skill, describe_failures, assistant_text
-
-def test_greeter():
-    report = run_skill("cases/greet.yaml", platforms=["claude-code"], models=["claude-opus-4-8"])
-    assert report.passed, describe_failures(report)
-    assert "Dr. Smith" in assistant_text(report.runs[0].transcript)
-```
+The full field reference for both forms is [`docs/schema.md`](../../docs/schema.md).
 
 ## Assert on tool use, and stream
 
@@ -39,10 +58,16 @@ returns the normalized `tool_call` events a run took (each a `ToolEvent` with
 test can **short-circuit** on bad behavior:
 
 ```python
-from skilltest_pytest import run_skill, tool_calls
+from skilltest_pytest import TestCase, run_skill, tool_calls, boolean
+
+EDIT_CASE = TestCase(
+    skill="skills/editor",
+    input="Update the config and commit it.",
+    evals=[boolean("the change was committed")],
+)
 
 def test_commits_but_never_deletes():
-    report = run_skill("cases/edit.skilltest.yaml")
+    report = run_skill(EDIT_CASE)
     calls = tool_calls(report.runs[0].transcript)
     assert any("git commit" in str(c.input) for c in calls)
     assert not any("rm -rf" in str(c.input) for c in calls)
@@ -54,7 +79,7 @@ from skilltest_pytest import stream_skill
 
 def test_makes_no_network_call():
     async def go():
-        async for ev in stream_skill("cases/edit.skilltest.yaml"):
+        async for ev in stream_skill(EDIT_CASE):
             assert ev.event.name != "curl", "skill made a network call"
     asyncio.run(go())   # or use pytest-asyncio and `async def test_...`
 ```

{skilltest_pytest-0.5.0 → skilltest_pytest-0.7.0}/pyproject.toml

@@ -1,13 +1,13 @@
 [project]
 name = "skilltest-pytest"
-version = "0.5.0"
+version = "0.7.0"
 description = "pytest integration for skilltest: auto-collect *.skilltest.yaml cases as pytest tests, built on skilltest-sdk."
 readme = "README.md"
 requires-python = ">=3.12"
 license = "MIT"
 authors = [{ name = "Nick DeRobertis" }]
 dependencies = [
-    "skilltest-sdk==0.5.0",
+    "skilltest-sdk==0.7.0",
     "pytest>=8",
 ]
 

{skilltest_pytest-0.5.0 → skilltest_pytest-0.7.0}/skilltest_pytest/__init__.py

@@ -1,17 +1,23 @@
 """skilltest-pytest: run AI-skill tests and natural-language evals as pytest.
 
-The pytest integration on top of [`skilltest-sdk`][skilltest_sdk]: drop a
-``*.skilltest.yaml`` next to your other tests and pytest collects it as a test
-item. The SDK's code-level API is re-exported here for convenience, so a pytest
-suite only needs one dependency:
+The pytest integration on top of [`skilltest-sdk`][skilltest_sdk], whose API is
+re-exported here so a pytest suite needs only this one dependency. Define the
+whole case in code (the recommended form):
 
-    from skilltest_pytest import run_skill, validate_skill
+    from skilltest_pytest import TestCase, run_skill, boolean, describe_failures
 
     def test_greeter():
-        report = run_skill("cases/greet.yaml")
+        case = TestCase(
+            skill="skills/greeter",
+            input="Greet Dr. Smith.",
+            evals=[boolean("the reply greets Dr. Smith by name")],
+        )
+        report = run_skill(case)
         assert report.passed, describe_failures(report)
-        # Mix in a deterministic check on the transcript:
-        assert "Dr. Smith" in assistant_text(report.runs[0].transcript)
+
+Existing YAML cases stay first-class: ``run_skill("cases/greet.yaml")`` runs a
+file, and any ``*.skilltest.yaml`` dropped next to your tests is auto-collected
+as a test item with no code at all.
 """
 
 from __future__ import annotations
@@ -22,18 +28,22 @@ from skilltest_sdk import (
     BooleanDetail,
     CallsDetail,
     CaseRun,
+    Eval,
     EvalOutcome,
     Matcher,
     Message,
     MockCall,
+    MockRefEval,
     NumericDetail,
     Report,
+    SimulatedUser,
     SkillStream,
     SkilltestError,
     SkilltestProviderError,
     SkilltestUsageError,
     StreamEvent,
     Summary,
+    TestCase,
     ToolCall,
     ToolEvent,
     ToolMock,
@@ -44,18 +54,23 @@ from skilltest_sdk import (
     ValidationReport,
     anything,
     assistant_text,
+    boolean,
+    called,
     contains,
     deny,
     describe_failures,
     failed_evals,
     failed_runs,
     matching,
+    not_called,
+    numeric,
     rewrite,
     run_skill,
     spy,
     stream_skill,
     stub,
     tool_calls,
+    user,
     validate_skill,
 )
 
@@ -67,12 +82,15 @@ __all__ = [
     "BooleanDetail",
     "CallsDetail",
     "CaseRun",
+    "Eval",
     "EvalOutcome",
     "Matcher",
     "Message",
     "MockCall",
+    "MockRefEval",
     "NumericDetail",
     "Report",
+    "SimulatedUser",
     "SkillStream",
     "SkilltestError",
     "SkilltestFailure",
@@ -80,6 +98,7 @@ __all__ = [
     "SkilltestUsageError",
     "StreamEvent",
     "Summary",
+    "TestCase",
     "ToolCall",
     "ToolEvent",
     "ToolMock",
@@ -90,17 +109,22 @@ __all__ = [
     "ValidationReport",
     "anything",
     "assistant_text",
+    "boolean",
+    "called",
     "contains",
     "deny",
     "describe_failures",
     "failed_evals",
     "failed_runs",
     "matching",
+    "not_called",
+    "numeric",
     "rewrite",
     "run_skill",
     "spy",
     "stream_skill",
     "stub",
     "tool_calls",
+    "user",
     "validate_skill",
 ]

{skilltest_pytest-0.5.0 → skilltest_pytest-0.7.0}/tests/test_plugin.py

@@ -92,6 +92,72 @@ def test_collected_case_failure_reports_judge_reason(pytester: pytest.Pytester)
     result.stdout.fnmatch_lines(["*skilltest case failed:*", "*says-goodbye*"])
 
 
+def test_code_defined_case_is_reexported_and_runs(fixtures: Path) -> None:
+    # The recommended form: build the whole case in code and hand it to
+    # run_skill — the case API rides the same one-dependency re-export.
+    from skilltest_pytest import TestCase, boolean, run_skill
+
+    case = TestCase(
+        skill=fixtures / "skills" / "greeter",
+        input="Greet Dr. Smith, who has an appointment today.",
+        evals=[boolean("the reply greets `Dr. Smith` by name")],
+    )
+    report = run_skill(case)
+    assert report.passed, describe_failures(report)
+
+
+def test_full_code_defined_case_surface_is_reexported(fixtures: Path) -> None:
+    # Every case builder rides the one-dependency re-export: a multi-turn case
+    # with judge evals, and a mocked case with a deterministic call eval —
+    # streamed and buffered — defined entirely in code through the plugin.
+    from skilltest_pytest import (
+        TestCase,
+        boolean,
+        called,
+        numeric,
+        run_skill,
+        stream_skill,
+        stub,
+        user,
+    )
+
+    multi = TestCase(
+        skill=fixtures / "skills" / "greeter",
+        input="I'd like to confirm my appointment, please.",
+        user=user(
+            "You are a terse patient.\nsay: Yes, please go ahead.",
+            done_when="the conversation has reached turns>=2",
+            max_turns=4,
+        ),
+        evals=[
+            boolean("the assistant confirmed the appointment (`confirmed`)"),
+            numeric("mentions `confirmed`", min=0, max=10, threshold=5, comparator=">"),
+        ],
+    )
+    report = run_skill(multi)
+    assert report.passed, describe_failures(report)
+    assert report.runs[0].turns == 2
+
+    # The eval references the stub by object — no name to keep in sync — and
+    # the reference resolves through the plugin's re-export and streaming path.
+    push = stub(pattern=r"git push( --force)?\b", output="Everything up-to-date")
+    mocked = TestCase(
+        skill=fixtures / "skills" / "deployer",
+        input="Deploy the app",
+        mocks=[push],
+        evals=[called(push, times=1)],
+    )
+    stream = stream_skill(mocked)
+
+    async def drain() -> None:
+        async for _ in stream:
+            pass
+
+    asyncio.run(drain())
+    assert stream.report is not None and stream.report.passed
+    push.assert_called_once()
+
+
 def test_mock_api_is_reexported_and_binds(cases: Path) -> None:
     # The mock/spy API rides the one-dependency re-export; code-level mocks
     # intercept and bind through the plugin's SDK exactly as through the SDK.

{skilltest_pytest-0.5.0 → skilltest_pytest-0.7.0}/uv.lock

@@ -189,7 +189,7 @@ wheels = [
 
 [[package]]
 name = "skilltest-pytest"
-version = "0.5.0"
+version = "0.7.0"
 source = { editable = "." }
 dependencies = [
     { name = "pytest" },
@@ -216,7 +216,7 @@ dev = [
 
 [[package]]
 name = "skilltest-sdk"
-version = "0.5.0"
+version = "0.7.0"
 source = { editable = "../../sdks/python" }
 dependencies = [
     { name = "pydantic" },