ancha 0.1.0__tar.gz

ancha-0.1.0/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: ancha
+Version: 0.1.0
+Summary: Ancha for Python: test your agent's own tools against production's recorded calls, and pytest worlds for agent tests
+License-Expression: MIT
+Classifier: Framework :: Pytest
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: pytest
+Requires-Dist: pytest>=7; extra == "pytest"
+
+# ancha
+
+Ancha for Python. Two parts, one `import ancha`:
+
+- **Tool wrappers** (no dependencies): wrap your agent's own tools once with `@ancha.tool` or `ancha.wrap_tools`, and
+  run the agent through `ancha.entry`. In production nothing changes. In an [Ancha](https://ancha.io) test made from a
+  production trace, each call answers as production did. See
+  [Tool wrappers](https://docs.ancha.io/reference/tool-wrappers).
+- **A pytest plugin** (`pip install "ancha[pytest]"`): a fresh simulated world for each test. See
+  [Test with pytest](https://docs.ancha.io/guides/pytest).
+
+```sh
+pip install ancha
+```
+
+```python
+import ancha
+
+@ancha.tool
+def lookup_order(order_id: str) -> dict: ...
+
+ancha.entry(lambda input: run_agent(input["task"]))
+```
+
+## The pytest plugin
+
+pytest for Ancha: a fresh simulated world for each test, your agent run against it, and checks on what people learned and what the tools hold. It drives the `ancha` CLI. This development version needs the matching CLI with `world capture` and `world record`; published CLI 0.1.4 does not support run recording. `ANCHA_CLI` names another CLI command.
+
+```python
+import ancha, pytest
+
+@pytest.mark.ancha(test="ancha/tests/relay-export-bug.yaml")
+@ancha.attempts(10, min_pass_rate=0.9)
+def test_relay(world):
+    world.start("npm start")                 # the agent gets world.env: the tools' URLs and fake credentials, ANCHA_TASK
+    world.wait_until_quiet()
+    assert world.people["ben"].knows("laura.beta_on")
+    assert world.slack.messages(to="laura", in_thread_of="$laura_report", from_="$slack_bot")
+    assert not world.people["laura"].knows("ben.root_cause")
+```
+
+- `@pytest.mark.ancha(test=…, scene=…, world=…, on="local"|"cloud", seed=…, ttl=…)` says what the `world` fixture starts (`ancha world up`). Without it, the one scene `ancha.yaml` names.
+- `world.env`, `world.task`; `world.start(command)`, `world.wait_until_quiet(timeout)`, `world.log()`, `world.stop()`.
+- `world.check()` applies the test's own checks; `world.check({...})` applies check objects (the kinds in `ancha docs scenes`). A report is truthy when every check passed; `report.failures` has expected and actual.
+- `world.people[ref].knows(fact)`, `.decided(decision, option)`; `world.slack.messages(to=, in_=, in_thread_of=, from_=, text_contains=, fact=)`.
+- `world.state()`, `world.trace()`; `world.fresh()` for another world; the fixture removes its world (`ancha world down`).
+- `@ancha.attempts(n, min_pass_rate)` runs the test in `n` fresh worlds and passes when enough of them do.
+
+An evaluated check that fails returns a false `CheckReport`; `report.failures` explains the expected and actual values. A check that cannot be evaluated raises `ancha.AnchaError` instead. For example, a `knows` check naming an unknown person raises an error with `code == "USAGE"` and a message naming the invalid reference. Correct the test definition; this error is not a verdict on the agent. The people and Slack helpers use the same check path and can raise the same error. CLI failures also raise `AnchaError`, retaining the CLI's `code`, `message` and optional `hint`.
+
+Worlds start locally by default, including after `ancha login`. Run `pytest --on cloud` to put worlds in your hosted workspace; pytest and the agent still run on your machine. `--on local` forces local worlds. The pytest option overrides the marker's `on="local"` or `on="cloud"`; without either, the CLI's normal world selection applies. An explicit server URL can also select a world server. Never give the agent the test files or the checks.
+
+## Recorded runs
+
+While the plugin is enabled, each executed pytest item creates one local run under `.ancha/runs/`, with `source: pytest` and `target: local`. This includes tests without a world, parameterized cases and skipped tests. Collection alone creates no records. Disable the plugin with `-p no:ancha` for suites that should not record. Run files stay local even when worlds run in the cloud; uploading pytest records is not supported.
+
+```sh
+ancha --local runs list --json
+ancha --local runs show <run-id> --json
+ancha --local explain <run-id> --json
+```
+
+Pytest assertions decide the test outcome. A deliberately false `world.check()` remains evidence and does not fail a passing negative assertion. Records preserve setup, call and teardown outcomes separately. Skipped and expected-failure cases are ungraded; setup, capture and cleanup failures are invalid, not failed agent answers. `@ancha.attempts` records each invocation and its fresh world under the same run, including failed attempts when the overall pass-rate threshold is met. The requested repetition count is recorded before execution. Reports keep every requested attempt in the pass-rate denominator, including invalid, skipped and missing attempts; all-k requires all requested attempts to finish validly and pass. Thresholds use exact integer counts. An old baseline without verified requested counts must be regenerated before comparison.
+
+The plugin stops its agent, captures a coherent world observation, then removes the world. This observation is not a sealed managed-trial capture. Capture failures remain visible and cleanup is still attempted. Recording failures are reported and make an otherwise successful pytest session fail.
+
+Parameter values in test IDs are replaced by a hash. Records omit environment values, raw exception messages, captured pytest output and agent logs. Known credentials are redacted from evidence; keep `.ancha/` private and out of version control. Abrupt termination such as SIGKILL cannot finish a record or guarantee cleanup; world TTLs still apply.

ancha-0.1.0/README.md

@@ -0,0 +1,67 @@
+# ancha
+
+Ancha for Python. Two parts, one `import ancha`:
+
+- **Tool wrappers** (no dependencies): wrap your agent's own tools once with `@ancha.tool` or `ancha.wrap_tools`, and
+  run the agent through `ancha.entry`. In production nothing changes. In an [Ancha](https://ancha.io) test made from a
+  production trace, each call answers as production did. See
+  [Tool wrappers](https://docs.ancha.io/reference/tool-wrappers).
+- **A pytest plugin** (`pip install "ancha[pytest]"`): a fresh simulated world for each test. See
+  [Test with pytest](https://docs.ancha.io/guides/pytest).
+
+```sh
+pip install ancha
+```
+
+```python
+import ancha
+
+@ancha.tool
+def lookup_order(order_id: str) -> dict: ...
+
+ancha.entry(lambda input: run_agent(input["task"]))
+```
+
+## The pytest plugin
+
+pytest for Ancha: a fresh simulated world for each test, your agent run against it, and checks on what people learned and what the tools hold. It drives the `ancha` CLI. This development version needs the matching CLI with `world capture` and `world record`; published CLI 0.1.4 does not support run recording. `ANCHA_CLI` names another CLI command.
+
+```python
+import ancha, pytest
+
+@pytest.mark.ancha(test="ancha/tests/relay-export-bug.yaml")
+@ancha.attempts(10, min_pass_rate=0.9)
+def test_relay(world):
+    world.start("npm start")                 # the agent gets world.env: the tools' URLs and fake credentials, ANCHA_TASK
+    world.wait_until_quiet()
+    assert world.people["ben"].knows("laura.beta_on")
+    assert world.slack.messages(to="laura", in_thread_of="$laura_report", from_="$slack_bot")
+    assert not world.people["laura"].knows("ben.root_cause")
+```
+
+- `@pytest.mark.ancha(test=…, scene=…, world=…, on="local"|"cloud", seed=…, ttl=…)` says what the `world` fixture starts (`ancha world up`). Without it, the one scene `ancha.yaml` names.
+- `world.env`, `world.task`; `world.start(command)`, `world.wait_until_quiet(timeout)`, `world.log()`, `world.stop()`.
+- `world.check()` applies the test's own checks; `world.check({...})` applies check objects (the kinds in `ancha docs scenes`). A report is truthy when every check passed; `report.failures` has expected and actual.
+- `world.people[ref].knows(fact)`, `.decided(decision, option)`; `world.slack.messages(to=, in_=, in_thread_of=, from_=, text_contains=, fact=)`.
+- `world.state()`, `world.trace()`; `world.fresh()` for another world; the fixture removes its world (`ancha world down`).
+- `@ancha.attempts(n, min_pass_rate)` runs the test in `n` fresh worlds and passes when enough of them do.
+
+An evaluated check that fails returns a false `CheckReport`; `report.failures` explains the expected and actual values. A check that cannot be evaluated raises `ancha.AnchaError` instead. For example, a `knows` check naming an unknown person raises an error with `code == "USAGE"` and a message naming the invalid reference. Correct the test definition; this error is not a verdict on the agent. The people and Slack helpers use the same check path and can raise the same error. CLI failures also raise `AnchaError`, retaining the CLI's `code`, `message` and optional `hint`.
+
+Worlds start locally by default, including after `ancha login`. Run `pytest --on cloud` to put worlds in your hosted workspace; pytest and the agent still run on your machine. `--on local` forces local worlds. The pytest option overrides the marker's `on="local"` or `on="cloud"`; without either, the CLI's normal world selection applies. An explicit server URL can also select a world server. Never give the agent the test files or the checks.
+
+## Recorded runs
+
+While the plugin is enabled, each executed pytest item creates one local run under `.ancha/runs/`, with `source: pytest` and `target: local`. This includes tests without a world, parameterized cases and skipped tests. Collection alone creates no records. Disable the plugin with `-p no:ancha` for suites that should not record. Run files stay local even when worlds run in the cloud; uploading pytest records is not supported.
+
+```sh
+ancha --local runs list --json
+ancha --local runs show <run-id> --json
+ancha --local explain <run-id> --json
+```
+
+Pytest assertions decide the test outcome. A deliberately false `world.check()` remains evidence and does not fail a passing negative assertion. Records preserve setup, call and teardown outcomes separately. Skipped and expected-failure cases are ungraded; setup, capture and cleanup failures are invalid, not failed agent answers. `@ancha.attempts` records each invocation and its fresh world under the same run, including failed attempts when the overall pass-rate threshold is met. The requested repetition count is recorded before execution. Reports keep every requested attempt in the pass-rate denominator, including invalid, skipped and missing attempts; all-k requires all requested attempts to finish validly and pass. Thresholds use exact integer counts. An old baseline without verified requested counts must be regenerated before comparison.
+
+The plugin stops its agent, captures a coherent world observation, then removes the world. This observation is not a sealed managed-trial capture. Capture failures remain visible and cleanup is still attempted. Recording failures are reported and make an otherwise successful pytest session fail.
+
+Parameter values in test IDs are replaced by a hash. Records omit environment values, raw exception messages, captured pytest output and agent logs. Known credentials are redacted from evidence; keep `.ancha/` private and out of version control. Abrupt termination such as SIGKILL cannot finish a record or guarantee cleanup; world TTLs still apply.

ancha-0.1.0/pyproject.toml

@@ -0,0 +1,23 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+# One distribution for `import ancha`: the tool wrappers need nothing; the pytest plugin needs pytest (ancha[pytest]).
+name = "ancha"
+version = "0.1.0"
+description = "Ancha for Python: test your agent's own tools against production's recorded calls, and pytest worlds for agent tests"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+dependencies = []
+classifiers = ["Framework :: Pytest"]
+
+[project.optional-dependencies]
+pytest = ["pytest>=7"]
+
+[project.entry-points.pytest11]
+ancha = "ancha.plugin"
+
+[tool.setuptools.packages.find]
+where = ["src"]

ancha-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ancha-0.1.0/src/ancha/__init__.py

@@ -0,0 +1,19 @@
+"""Ancha for Python: wrap your agent's own tools so a test made from a production trace answers them as production did
+(`ancha.tool`, `ancha.wrap_tools`, `ancha.entry`; see ancha/tools.py), and pytest for Ancha: a fresh simulated world for each test, your agent run against it, and checks on what people learned
+and what the tools hold.
+
+    import ancha, pytest
+
+    @pytest.mark.ancha(test="ancha/tests/relay-export-bug.yaml")
+    @ancha.attempts(10, min_pass_rate=0.9)
+    def test_relay(world):
+        world.start("npm start")
+        world.wait_until_quiet()
+        assert world.people["ben"].knows("laura.beta_on")
+        assert world.slack.messages(to="laura", in_thread_of="$laura_report", from_="$slack_bot")
+        assert not world.people["laura"].knows("ben.root_cause")
+"""
+from .tools import ToolError, call_tool, entry, tool, wrap_tools
+from .world import AnchaError, CheckReport, People, Person, Slack, World, attempts
+
+__all__ = ["AnchaError", "CheckReport", "People", "Person", "Slack", "ToolError", "World", "attempts", "call_tool", "entry", "tool", "wrap_tools"]

ancha-0.1.0/src/ancha/plugin.py

@@ -0,0 +1,85 @@
+"""The pytest plugin: the `world` fixture, configured per test with `@pytest.mark.ancha(...)`."""
+from __future__ import annotations
+
+import os
+from typing import Any, Iterator
+
+import pytest
+
+from .recording import Recorder, current, error_code
+from .world import World
+
+OPTIONS = ("test", "scene", "world", "on", "seed", "ttl")
+
+
+def pytest_addoption(parser: pytest.Parser) -> None:
+    parser.getgroup("ancha").addoption("--on", choices=("local", "cloud"), default=None,
+        help="where Ancha worlds run; overrides the ancha marker (pytest and the agent stay local)")
+
+
+def pytest_configure(config: pytest.Config) -> None:
+    config.addinivalue_line("markers", "ancha(test=None, scene=None, world=None, on=None, seed=None, ttl=None, cwd=None): "
+                                       "what the `world` fixture starts: a test or scene file, or a saved world; where (local or cloud)")
+    config._ancha_recording_failures = []
+    config._ancha_recorded = 0
+
+
+@pytest.hookimpl(hookwrapper=True, tryfirst=True)
+def pytest_runtest_protocol(item: pytest.Item, nextitem: pytest.Item | None):
+    marker = item.get_closest_marker("ancha")
+    recorder = Recorder(item.nodeid, str((marker.kwargs.get("cwd") if marker else None) or os.getcwd()))
+    item._ancha_recorder = recorder
+    token = current.set(recorder)
+    try:
+        yield
+    finally:
+        try:
+            recorder.finish()
+            recorder.write()
+            item.config._ancha_recorded += 1
+            if recorder.infrastructure_failed:
+                item.config._ancha_recording_failures.append((recorder.payload["runId"], "INFRASTRUCTURE_ERROR"))
+        except Exception as error:
+            # Recording cannot turn a failing test into an internal pytest error or expose its exception text.
+            item.config._ancha_recording_failures.append((recorder.payload["runId"], error_code(error)))
+        finally:
+            current.reset(token)
+
+
+@pytest.hookimpl(hookwrapper=True, tryfirst=True)
+def pytest_runtest_makereport(item: pytest.Item, call: pytest.CallInfo):
+    outcome = yield
+    recorder = getattr(item, "_ancha_recorder", None)
+    if recorder is not None:
+        recorder.report(outcome.get_result(), call.excinfo.value if call.excinfo is not None else None)
+
+
+def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
+    if session.config._ancha_recording_failures and session.exitstatus == pytest.ExitCode.OK:
+        session.exitstatus = pytest.ExitCode.TESTS_FAILED
+
+
+def pytest_terminal_summary(terminalreporter: Any) -> None:
+    config = terminalreporter.config
+    if config._ancha_recorded:
+        terminalreporter.write_line(f"Ancha recorded {config._ancha_recorded} pytest run(s) locally in .ancha/runs.")
+    for run_id, code in config._ancha_recording_failures:
+        terminalreporter.write_line(f"Ancha run {run_id}: infrastructure or recording failed ({code}); pytest session failed.", red=True)
+
+
+@pytest.fixture
+def world(request: pytest.FixtureRequest) -> Iterator[World]:
+    """A fresh world for this test (`ancha world up`), removed afterwards. Without a marker: the one scene ancha.yaml names."""
+    marker = request.node.get_closest_marker("ancha")
+    options = dict(marker.kwargs) if marker else {}
+    unknown = set(options) - {*OPTIONS, "cwd"}
+    if unknown:
+        raise pytest.UsageError(f"@pytest.mark.ancha takes {', '.join(OPTIONS)} and cwd, not {', '.join(sorted(unknown))}")
+    if request.config.getoption("--on") is not None:
+        options["on"] = request.config.getoption("--on")
+    args = [arg for key in OPTIONS if options.get(key) is not None for arg in (f"--{key}", str(options[key]))]
+    live = World(args, cwd=str(options.get("cwd") or os.getcwd()))
+    try:
+        yield live
+    finally:
+        live.down()

ancha-0.1.0/src/ancha/recording.py

@@ -0,0 +1,173 @@
+"""Private per-test observations. The CLI validates and stores the finished run."""
+from __future__ import annotations
+
+import contextvars
+import datetime as dt
+import hashlib
+import json
+import os
+import re
+import tempfile
+import uuid
+from typing import Any
+
+
+current: contextvars.ContextVar[Recorder | None] = contextvars.ContextVar("ancha_pytest_recorder", default=None)
+_SECRET_NAME = re.compile(r"secret|token|password|passwd|api.?key|private.?key|credential|cookie|authorization", re.I)
+_CODE = re.compile(r"^[A-Za-z_][A-Za-z0-9_.:-]{0,127}$")
+
+
+def now() -> str:
+    return dt.datetime.now(dt.timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
+
+
+def error_code(error: BaseException | None) -> str:
+    code = getattr(error, "code", None) or (type(error).__name__ if error is not None else "PYTEST_ERROR")
+    return code if isinstance(code, str) and _CODE.fullmatch(code) else "PYTEST_ERROR"
+
+
+def node_id(value: str) -> str:
+    # Parameter IDs can include credentials, payloads or user data. Keep test identity without their contents.
+    if "[" not in value:
+        return value
+    plain, parameters = value.split("[", 1)
+    return f"{plain}[case-{hashlib.sha256(parameters.encode()).hexdigest()[:16]}]"
+
+
+class Recorder:
+    def __init__(self, node: str, cwd: str) -> None:
+        self.cwd = cwd
+        self.secrets: set[str] = set()
+        self.add_secrets(os.environ)
+        self.payload: dict[str, Any] = {
+            "version": 1, "runId": f"run_pytest_{uuid.uuid4().hex}", "nodeId": node_id(node),
+            "startedAt": now(), "finishedAt": now(), "outcome": "passed", "minPassRate": 1.0,
+            "phases": [], "attempts": [], "requestedTrials": 1,
+        }
+        self.worlds: list[Any] = []
+        self.repeated = False
+        self.infrastructure_failed = False
+        self.begin_attempt()
+
+    @property
+    def attempt(self) -> dict[str, Any]:
+        return self.payload["attempts"][-1]
+
+    def add_secrets(self, values: Any) -> None:
+        self.secrets.update(value for key, value in values.items()
+                            if _SECRET_NAME.search(key) and isinstance(value, str) and value)
+
+    def begin_attempt(self) -> None:
+        self.payload["attempts"].append({"number": len(self.payload["attempts"]) + 1,
+            "outcome": "passed", "startedAt": now(), "finishedAt": now(), "worlds": [], "errors": []})
+
+    def add_world(self, world: Any) -> dict[str, Any]:
+        value: dict[str, Any] = {"envId": world.id, "workspace": world.workspace, "checks": [], "errors": []}
+        self.worlds.append(world)
+        self.attempt["worlds"].append(value)
+        self.add_secrets(world.env)
+        return value
+
+    def infrastructure_error(self, phase: str, error: BaseException, world: dict | None = None) -> None:
+        issue = {"phase": phase, "code": error_code(error)}
+        if world is not None:
+            world["errors"].append(issue)
+        else:
+            self.attempt["errors"].append(issue)
+        self.attempt["outcome"] = "invalid"
+        self.infrastructure_failed = True
+
+    def close_worlds(self) -> BaseException | None:
+        first: BaseException | None = None
+        # World.down owns error recording and is idempotent, including after a failed removal.
+        for world in self.worlds:
+            try:
+                world.down()
+            except BaseException as error:
+                first = first or error
+        return first
+
+    def finish_attempt(self, outcome: str) -> BaseException | None:
+        if self.attempt["outcome"] != "invalid":
+            self.attempt["outcome"] = outcome
+        error = self.close_worlds()
+        self.attempt["finishedAt"] = now()
+        return error
+
+    def report(self, report: Any, error: BaseException | None) -> None:
+        phase: dict[str, Any] = {"when": report.when, "outcome": report.outcome,
+                                "durationMs": max(0, report.duration * 1000)}
+        if report.failed:
+            phase["errorCode"] = error_code(error)
+        self.payload["phases"].append(phase)
+        if report.skipped:
+            outcome = "xfailed" if getattr(report, "wasxfail", None) is not None else "skipped"
+            if self.payload["outcome"] not in ("failed", "error"):
+                self.payload["outcome"] = outcome
+            if not self.repeated:
+                self.attempt["outcome"] = outcome
+        elif report.failed:
+            # A failed assertion/pytest.fail is a test verdict. Other exceptions and fixture errors are invalid.
+            import pytest
+            assertion = isinstance(error, (AssertionError, pytest.fail.Exception))
+            strict_xpass = error is None and report.when == "call" and str(report.longrepr).startswith("[XPASS(strict)]")
+            if report.when != "call" or (not assertion and not strict_xpass):
+                stage = getattr(error, "_ancha_phase", report.when)
+                self.infrastructure_error(stage, error or RuntimeError())
+                self.payload["outcome"] = "error"
+            elif self.payload["outcome"] != "error":
+                self.payload["outcome"] = "failed"
+                if not self.repeated and not strict_xpass:
+                    self.attempt["outcome"] = "failed"
+        elif report.when == "call" and getattr(report, "wasxfail", None) is not None:
+            self.payload["outcome"] = "xpassed"
+            if not self.repeated:
+                self.attempt["outcome"] = "xpassed"
+
+    def _redact(self, value: Any) -> Any:
+        if isinstance(value, str):
+            for secret in sorted(self.secrets, key=len, reverse=True):
+                if len(secret) >= 4:
+                    value = value.replace(secret, "[redacted]")
+                elif value == secret:
+                    value = "[redacted]"
+            return value
+        if isinstance(value, list):
+            return [self._redact(item) for item in value]
+        if isinstance(value, dict):
+            return {self._redact(key): self._redact(item) for key, item in value.items()}
+        return value
+
+    def safe_payload(self) -> dict:
+        # Keep protocol enums/timestamps/IDs intact even if a credential happens to equal one of their values.
+        def code(value: str) -> str:
+            return value if self._redact(value) == value else "REDACTED_ERROR"
+
+        def errors(values: list[dict]) -> list[dict]:
+            return [{**value, "code": code(value["code"])} for value in values]
+
+        def world(value: dict) -> dict:
+            return {**{key: self._redact(item) for key, item in value.items() if key != "errors"}, "errors": errors(value["errors"])}
+
+        return {**self.payload, "nodeId": self._redact(self.payload["nodeId"]),
+                "phases": [{**phase, **({"errorCode": code(phase["errorCode"])} if "errorCode" in phase else {})}
+                           for phase in self.payload["phases"]],
+                "attempts": [{**attempt, "worlds": [world(value) for value in attempt["worlds"]], "errors": errors(attempt["errors"])}
+                             for attempt in self.payload["attempts"]]}
+
+    def finish(self) -> None:
+        self.close_worlds()
+        if self.infrastructure_failed:
+            self.payload["outcome"] = "error"
+        self.attempt["finishedAt"] = now()
+        self.payload["finishedAt"] = now()
+
+    def write(self) -> dict:
+        from .world import run_cli
+        fd, path = tempfile.mkstemp(prefix="ancha-pytest-", suffix=".json")
+        try:
+            with os.fdopen(fd, "w", encoding="utf-8") as file:
+                json.dump(self.safe_payload(), file, ensure_ascii=False)
+            return run_cli(["world", "record", "--file", path], self.cwd)
+        finally:
+            os.unlink(path)