convy 0.1.0__tar.gz

convy-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ivan Deyna
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

convy-0.1.0/PKG-INFO

@@ -0,0 +1,422 @@
+Metadata-Version: 2.4
+Name: convy
+Version: 0.1.0
+Summary: Check conversational agents the way a person would: a simulated user talks to the agent, a judge rates the dialogue, and convy reports pass rate, tokens and response time.
+Keywords: llm,agents,ai-agents,evaluation,llm-evaluation,agent-evaluation,testing,conversational-ai,chatbot-testing,simulated-user,llm-as-a-judge,benchmark
+Author: deyna256
+Author-email: deyna256 <literallybugcreator@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Classifier: Framework :: AsyncIO
+Requires-Dist: httpx2>=2.13.1
+Requires-Dist: msgspec>=0.22.0
+Requires-Dist: pydantic-settings>=2.15.0
+Requires-Dist: yamlrocks>=0.6.1,<0.7
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/deyna256/convy
+Project-URL: Documentation, https://github.com/deyna256/convy#readme
+Project-URL: Changelog, https://github.com/deyna256/convy/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/deyna256/convy/issues
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+<h1>convy</h1>
+
+<h3>Conversation tests for AI agents</h3>
+
+<p>A simulated user talks to your agent, a judge rates the dialogue, and convy reports what you need to
+decide: <strong>which scenarios the agent passes</strong>, <strong>how many tokens it spends</strong> and
+<strong>how long it takes to answer</strong>. Connect any agent that runs as a service — convy needs
+neither its code nor its model.</p>
+
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
+[![Status: pre-alpha](https://img.shields.io/badge/status-pre--alpha-orange)](#status)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-3776ab)](pyproject.toml)
+<br>
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
+
+[Status](#status) · [Why convy](#why-convy) · [Quick start](#quick-start) · [Connect an agent](#connect-an-agent) · [Scenarios](#scenarios) · [Report](#the-report) · [Python](#run-from-python) · [FAQ](#faq) · [Contributing](CONTRIBUTING.md)
+
+</div>
+
+---
+
+## Status
+
+convy is not released yet. This README describes the first release, which is being built now.
+Nothing below can be installed from PyPI until `0.1.0` is out.
+
+## Why convy
+
+Before a team puts an agent in front of people, it wants to know how the agent behaves in a real
+conversation — not on a single prompt, but over several turns with someone who asks vague questions,
+changes their mind or never gives the details up front. And when there are several agents, or several
+builds of one, it wants to compare them on the same conversations.
+
+convy does exactly that and nothing more:
+
+- **Any agent, connected in minutes.** An agent with a JSON API needs a few lines of settings. Anything
+  else — streaming, a two-step login, a custom protocol — is two small Python classes.
+- **Conversations, not prompts.** A scenario tells a simulated user who to be and what to want; the
+  judge checks plain-language claims about the dialogue.
+- **Numbers for a decision.** Pass rate over repeated attempts, the agent's own tokens and its response
+  time per turn, side by side for every agent and version.
+- **Small and readable.** About a thousand lines, no evaluation framework and no containers underneath.
+
+## How it works
+
+```text
+             scenario                                       claims
+                │                                              │
+                ▼                                              ▼
+       ┌─────────────────┐  message  ┌─────────┐       ┌─────────────┐
+       │ simulated user  │ ────────▶ │  agent  │       │    judge    │──▶ verdict
+       │     (model)     │ ◀──────── │         │       │   (model)   │
+       └─────────────────┘  answer   └─────────┘       └─────────────┘
+                │     time and tokens of every turn           ▲
+                └───────────────── transcript ────────────────┘
+```
+
+For every attempt convy opens a fresh conversation with the agent, lets the simulated user talk to it
+until the user is done or the turns run out, and asks the judge whether every claim holds. Each attempt
+is written to a journal as soon as it ends, and `convy report` turns the journals into one static HTML
+page.
+
+## Quick start
+
+```sh
+uvx convy init my-bench              # a project with examples
+cd my-bench
+uv run convy run echo --smoke        # check convy itself: free, no models called
+cp .env.example .env                 # add the address and key of your model gateway
+uv run convy run echo                # a real run with the simulated user and the judge
+uv run convy report                  # results/index.html
+```
+
+`convy init` creates a project of your own:
+
+```text
+my-bench/
+├── gateway.py        # the model gateway, shared by models.py and the agents
+├── models.py         # the models that play the user and the judge
+├── agents/           # one file per agent
+├── scenarios/        # one YAML file per scenario
+└── results/          # journals and the report
+```
+
+## Connect an agent
+
+Add `agents/support_bot.py`. For an agent with a JSON API, settings are enough:
+
+```python
+from pydantic import SecretStr
+from convy import Env, JsonAgent, JsonEndpoint
+
+
+class BotEnv(Env):
+    bot_token: SecretStr  # BOT_TOKEN in .env
+
+
+env = BotEnv()
+agent = JsonAgent(
+    JsonEndpoint(
+        "https://bot.example.com/api/chat",
+        headers={"Authorization": f"Bearer {env.bot_token.get_secret_value()}"},
+    ),
+    body={"session_id": "{session}", "message": "{text}"},
+    reply="data.answer",
+    tokens=("data.usage.input_tokens", "data.usage.output_tokens"),
+)
+```
+
+convy sends the body on every turn with `{text}` replaced by the user's message and `{session}` by an id
+that stays the same for the whole conversation. For an agent that does not remember the dialogue, put
+`"{history}"` where it expects the list of messages. `reply` and `tokens` are paths in the JSON answer.
+
+Files of the project import each other, as in a script run from its folder: settings that several
+agents share go into one module, such as `gateway.py` in a new project (`from gateway import gateway`).
+
+Then check the connection — the agent answers for real, nothing else is called:
+
+```sh
+uv run convy run support_bot --smoke
+```
+
+Anything a JSON template cannot express is two classes in the same file: an agent that opens a
+conversation, and a conversation that answers a message.
+
+```python
+from contextlib import asynccontextmanager
+
+import httpx2
+
+from convy import Answer, Message, Usage
+
+
+class SupportBot:
+    @asynccontextmanager
+    async def conversation(self):
+        async with httpx2.AsyncClient(base_url="https://bot.example.com") as http:
+            login = await http.post("/login", json={"user": "convy"})
+            yield SupportChat(http, login.raise_for_status().json()["token"])
+
+
+class SupportChat:
+    def __init__(self, http: httpx2.AsyncClient, token: str):
+        self.http = http
+        self.token = token
+
+    async def answer(self, message: Message) -> Answer:
+        response = await self.http.post(
+            "/chat",
+            json={"text": message.text},
+            headers={"Authorization": f"Bearer {self.token}"},
+        )
+        data = response.raise_for_status().json()
+        return Answer(data["text"], Usage(input=data["tokens_in"], output=data["tokens_out"]))
+
+
+agent = SupportBot()
+```
+
+There is no need to catch errors in `answer`: any exception counts as a failed turn, and convy
+records it in the dialogue. The text of an exception from your agent goes to the journal and the
+report, so keep keys out of it; `Env`'s own errors never show the values read.
+
+### A client without async
+
+convy's interface is async. If your agent's client only has blocking calls, run them in a thread
+with `asyncio.to_thread`, so other conversations go on while one waits:
+
+```python
+import asyncio
+from contextlib import asynccontextmanager
+
+from support_sdk import Client  # a client with blocking calls only
+
+from convy import Answer, Message
+
+
+class SupportBot:
+    @asynccontextmanager
+    async def conversation(self):
+        client = Client("https://bot.example.com")
+        session = await asyncio.to_thread(client.start_session)
+        try:
+            yield SupportChat(client, session)
+        finally:
+            await asyncio.to_thread(client.close)
+
+
+class SupportChat:
+    def __init__(self, client: Client, session: str):
+        self.client = client
+        self.session = session
+
+    async def answer(self, message: Message) -> Answer:
+        text = await asyncio.to_thread(self.client.send, self.session, message.text)
+        return Answer(text)
+
+
+agent = SupportBot()
+```
+
+When a turn runs out of time, convy stops waiting for it, but a thread cannot be stopped: the call
+goes on in the background until it returns. Give the client a timeout of its own if it has one.
+
+### Tokens counted from the start
+
+Some agents report only the tokens spent so far, not those of one answer. Read the counter before
+and after the turn; the difference is the turn's tokens:
+
+```python
+import httpx2
+
+from convy import Answer, Message, Usage
+
+
+class SupportChat:
+    def __init__(self, http: httpx2.AsyncClient):
+        self.http = http
+
+    async def answer(self, message: Message) -> Answer:
+        before = await self.spent()
+        response = await self.http.post("/chat", json={"text": message.text})
+        after = await self.spent()
+        text = response.raise_for_status().json()["text"]
+        return Answer(text, Usage(input=after[0] - before[0], output=after[1] - before[1]))
+
+    async def spent(self) -> tuple[int, int]:
+        """The tokens this conversation has spent so far: input and output."""
+        usage = (await self.http.get("/usage")).raise_for_status().json()
+        return usage["input_tokens"], usage["output_tokens"]
+```
+
+The counter must belong to the conversation. A counter shared by conversations that run at the
+same time mixes their tokens; run such an agent with `--parallel 1`.
+
+### Corporate certificates
+
+An agent or a model gateway behind a corporate certificate authority takes `Tls` in its endpoint: `ca`
+is an extra root certificate, `cert` and `key` a client certificate. Declare the paths in your `Env` as
+pydantic's `FilePath`, so a wrong path stops convy before the run (`error: …`, exit code 2) instead of
+failing every attempt:
+
+```python
+from pydantic import FilePath
+from convy import Env, JsonAgent, JsonEndpoint, Tls
+
+
+class CorpEnv(Env):
+    corp_ca: FilePath  # CORP_CA in .env
+
+
+env = CorpEnv()
+agent = JsonAgent(
+    JsonEndpoint("https://bot.corp.example/api/chat", tls=Tls(ca=str(env.corp_ca))),
+    body={"message": "{text}"},
+    reply="answer",
+)
+```
+
+`FilePath` does not expand `~`: write the full path. A file that exists but is damaged shows only on
+connection — check an agent with `--smoke`.
+
+## Scenarios
+
+A scenario is a YAML file in `scenarios/`:
+
+```yaml
+id: clarify-backup
+max_turns: 6
+user: |
+  You need a backup script. Start short: "Make me a backup script".
+  Give details only when the assistant asks for them:
+  back up ~/photos to /mnt/backup, once a day.
+judge:
+  - Before giving a finished script, the agent asked what to back up and where to
+  - The final script copies ~/photos to /mnt/backup and runs once a day
+```
+
+`user` tells the simulated user who to be and what to want; `judge` lists claims that must all hold for
+the scenario to pass. An attempt in which the agent fails a turn does not pass, whatever it said before.
+When one of convy's own models fails instead — it does not answer, or the judge answers without a
+verdict in JSON (`"pass"` of `true` or `false`) — the attempt gets no verdict and is left out of the
+pass rate.
+Write scenarios in any language — the simulated user speaks the language of its instructions.
+
+```sh
+uv run convy run support_bot -k 3                  # every scenario, three attempts each
+uv run convy run support_bot --scenarios 'refund-*'
+uv run convy run support_bot new_bot               # several agents, one after another
+uv run convy run support_bot --turn-timeout 60
+```
+
+`--turn-timeout` (600 by default) is the seconds for each step of the agent: opening a conversation,
+a turn, closing. A step that takes longer fails the attempt as an agent error.
+
+## The report
+
+`results/index.html` is a single static page that works offline:
+
+- **agents** — for each agent and version: the share of attempts passed, every attempt as a mark
+  (`●` passed, `✕` failed, `○` no verdict), the agent's tokens and time per scenario, its answer
+  time, and problems such as agent errors; the best value in each column is in bold;
+- **scenarios** — scenarios × agents, split into those where the agents' results differ and those
+  where they are the same; a filter such as `refund-*` narrows the list;
+- **a result** — select a cell to read each attempt: the claims, the judge's reasoning, and the
+  dialogue with the time and tokens of each answer. The address keeps the selected result, so a link
+  to the page opens it.
+
+Tokens and time are the agent's own: the simulated user and the judge are not counted.
+
+## Run from Python
+
+Everything the command does is in the library. A run is a `Bench` played against an agent into a
+journal; the report is built from the journals:
+
+```python
+import asyncio
+from datetime import datetime
+from pathlib import Path
+
+from convy import Bench, JsonlJournal, Models, Report, RunHeader, Runs, Scenario
+from convy.fakes import Echo, FakeModel
+
+scenario = Scenario(
+    id="greet",
+    max_turns=3,
+    instructions="Say hello to the assistant, then thank it.",
+    claims=("The agent answered the greeting",),
+)
+models = Models(  # fakes: nothing is called; use OpenAiModel for real ones
+    user=FakeModel("Hello!", "Thank you!", "###STOP###"),
+    judge=FakeModel('{"pass": true, "reason": "it answered"}'),
+)
+bench = Bench((scenario,), models)
+header = RunHeader(
+    agent="echo",
+    version="",
+    user=models.user.name,
+    judge=models.judge.name,
+    attempts=bench.attempts,
+    planned=bench.planned(),
+    started=datetime.now().astimezone(),
+)
+runs = Path("results/runs")
+outcomes = asyncio.run(bench.run(Echo(), JsonlJournal(runs, header)))
+Path("results/index.html").write_text(Report(Runs(runs)).html(), encoding="utf-8")
+```
+
+Put your own agent in place of `Echo()`. The fakes in `convy.fakes` are public too, for testing your
+agent classes without a model.
+
+## FAQ
+
+**Does convy need to know which model my agent uses?**
+No. convy talks to the agent, not to its model. Only convy's own models — the simulated user and the
+judge — are set in `models.py`.
+
+**My agent does not report tokens.**
+Then the report shows time and results without tokens. If tokens matter, ask for them in the agent's
+answer, or read them from the model gateway the agent uses.
+
+**What do the exit codes mean?**
+- `0` — every attempt finished;
+- `1` — an attempt ended with an agent error or a failure of convy's models;
+- `2` — the project could not be loaded, and nothing ran.
+
+**Is it safe to run convy on someone else's project?**
+`convy run` executes `models.py` and `agents/*.py`. Treat a project like its tests: run only what you
+trust. See the [security policy](SECURITY.md).
+
+## Documentation
+
+| document | covers |
+|---|---|
+| [ARCHITECTURE.md](ARCHITECTURE.md) | how the code is laid out and the rules it keeps |
+| [docs/decisions/](docs/decisions/) | why the main choices were made |
+| [CHANGELOG.md](CHANGELOG.md) | what changed in each release |
+
+## Contributing
+
+Issues and pull requests are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md) first, and follow the
+[Code of Conduct](CODE_OF_CONDUCT.md). Report vulnerabilities privately, as the
+[security policy](SECURITY.md) describes.
+
+## License
+
+[MIT](LICENSE)