evalloop-agents 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- evalloop_agents-0.6.0/LICENSE +21 -0
- evalloop_agents-0.6.0/PKG-INFO +376 -0
- evalloop_agents-0.6.0/README.md +341 -0
- evalloop_agents-0.6.0/pyproject.toml +52 -0
- evalloop_agents-0.6.0/setup.cfg +4 -0
- evalloop_agents-0.6.0/src/evalloop/__init__.py +45 -0
- evalloop_agents-0.6.0/src/evalloop/adapters.py +287 -0
- evalloop_agents-0.6.0/src/evalloop/checks.py +320 -0
- evalloop_agents-0.6.0/src/evalloop/ci.py +95 -0
- evalloop_agents-0.6.0/src/evalloop/cli.py +175 -0
- evalloop_agents-0.6.0/src/evalloop/core.py +160 -0
- evalloop_agents-0.6.0/src/evalloop/frameworks.py +221 -0
- evalloop_agents-0.6.0/src/evalloop/instrument.py +173 -0
- evalloop_agents-0.6.0/src/evalloop/py.typed +0 -0
- evalloop_agents-0.6.0/src/evalloop/redteam.py +111 -0
- evalloop_agents-0.6.0/src/evalloop/report.py +200 -0
- evalloop_agents-0.6.0/src/evalloop/runner.py +398 -0
- evalloop_agents-0.6.0/src/evalloop/scaffold.py +80 -0
- evalloop_agents-0.6.0/src/evalloop/stats.py +64 -0
- evalloop_agents-0.6.0/src/evalloop/store.py +127 -0
- evalloop_agents-0.6.0/src/evalloop/traces.py +109 -0
- evalloop_agents-0.6.0/src/evalloop_agents.egg-info/PKG-INFO +376 -0
- evalloop_agents-0.6.0/src/evalloop_agents.egg-info/SOURCES.txt +31 -0
- evalloop_agents-0.6.0/src/evalloop_agents.egg-info/dependency_links.txt +1 -0
- evalloop_agents-0.6.0/src/evalloop_agents.egg-info/entry_points.txt +2 -0
- evalloop_agents-0.6.0/src/evalloop_agents.egg-info/requires.txt +10 -0
- evalloop_agents-0.6.0/src/evalloop_agents.egg-info/top_level.txt +1 -0
- evalloop_agents-0.6.0/tests/test_adapters.py +99 -0
- evalloop_agents-0.6.0/tests/test_evalloop.py +166 -0
- evalloop_agents-0.6.0/tests/test_frameworks.py +163 -0
- evalloop_agents-0.6.0/tests/test_multiturn.py +131 -0
- evalloop_agents-0.6.0/tests/test_v03.py +237 -0
- evalloop_agents-0.6.0/tests/test_v05.py +255 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sumit Parmar
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,376 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: evalloop-agents
|
|
3
|
+
Version: 0.6.0
|
|
4
|
+
Summary: Unit tests and a CI gate for LLM agents: reliability (pass^k), multi-turn conversations, prompt injection, hallucination, loops, cost and latency. Works with OpenAI Agents SDK and LangGraph.
|
|
5
|
+
Author: Sumit Parmar
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://sumitparmartech.github.io/evalloop/
|
|
8
|
+
Project-URL: Source, https://github.com/sumitparmartech/evalloop
|
|
9
|
+
Project-URL: Changelog, https://github.com/sumitparmartech/evalloop/blob/main/CHANGELOG.md
|
|
10
|
+
Project-URL: Issues, https://github.com/sumitparmartech/evalloop/issues
|
|
11
|
+
Keywords: llm,agents,evals,testing,ci,pass-k,reliability,llm-as-judge,pytest,langgraph,openai-agents,prompt-injection,red-teaming
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
22
|
+
Classifier: Topic :: Software Development :: Testing
|
|
23
|
+
Classifier: Typing :: Typed
|
|
24
|
+
Requires-Python: >=3.9
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
License-File: LICENSE
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: pytest>=7; extra == "dev"
|
|
29
|
+
Provides-Extra: agents
|
|
30
|
+
Requires-Dist: openai-agents>=0.2; extra == "agents"
|
|
31
|
+
Provides-Extra: langgraph
|
|
32
|
+
Requires-Dist: langgraph>=0.2; extra == "langgraph"
|
|
33
|
+
Requires-Dist: langchain-core>=0.3; extra == "langgraph"
|
|
34
|
+
Dynamic: license-file
|
|
35
|
+
|
|
36
|
+
# evalloop
|
|
37
|
+
|
|
38
|
+
[](https://github.com/sumitparmartech/evalloop/actions/workflows/tests.yml)
|
|
39
|
+
[](https://pypi.org/project/evalloop-agents/)
|
|
40
|
+
[](https://pypi.org/project/evalloop-agents/)
|
|
41
|
+
[](LICENSE)
|
|
42
|
+
|
|
43
|
+
**[Live demo](https://sumitparmartech.github.io/evalloop/)** · **[How it compares](docs/COMPARISON.md)** · **[Changelog](CHANGELOG.md)**
|
|
44
|
+
|
|
45
|
+
Unit tests and a CI gate for LLM agents. Zero dependencies, no API key needed to try it. Python 3.9+.
|
|
46
|
+
|
|
47
|
+
Most agent demos work once. Production agents fail 1 run in 5, quote amounts they never saw, obey instructions hidden in a web page, and loop on vague questions. `evalloop` runs your agent many times against a suite of cases, checks the full trace (not just the final answer), and fails your build when behavior regresses.
|
|
48
|
+
|
|
49
|
+

|
|
50
|
+
|
|
51
|
+
## What it catches
|
|
52
|
+
|
|
53
|
+
| Failure | Check |
|
|
54
|
+
| --- | --- |
|
|
55
|
+
| Flaky agent that passes once and fails later | `pass^k` reliability, every case runs k times |
|
|
56
|
+
| Hallucinated numbers (amounts, dates, ids) | `grounded()` flags any number not found in the input or a tool result |
|
|
57
|
+
| Prompt injection that leaks a secret | `no_secret_leak(...)` |
|
|
58
|
+
| Wrong tool, wrong arguments, wrong order | `tool_called`, `tool_not_called`, `tool_args`, `tool_order` |
|
|
59
|
+
| Infinite or wasteful tool loops | `no_loop(n)`, `max_steps(n)` |
|
|
60
|
+
| Cost and latency creep | `cost_under(usd)`, `latency_under(ms)` |
|
|
61
|
+
| PII or keys in the answer | `no_pii()` (email, phone, card, SSN, API key) |
|
|
62
|
+
| Expensive tool called too often | `tool_count(name, at_most=n)` |
|
|
63
|
+
| Tone, helpfulness, policy (things code cannot check) | `judge(criteria, ask)` LLM-as-judge |
|
|
64
|
+
| Agent forgets what the user said two messages ago | `Conversation(...)` multi-turn cases |
|
|
65
|
+
| Prompt injection hidden in a tool result | `Case.poison`, `injection_cases(...)` |
|
|
66
|
+
| Breaks on typos, SHOUTING, filler words | `variants(case, ...)` |
|
|
67
|
+
| Bad output format | `json_answer(...)`, `answer_matches(regex)` |
|
|
68
|
+
| Anything else | `custom(name, fn)` |
|
|
69
|
+
|
|
70
|
+
## Install
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
pip install evalloop-agents # the import name is still: import evalloop
|
|
74
|
+
# or the latest code from GitHub:
|
|
75
|
+
pip install git+https://github.com/sumitparmartech/evalloop
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Optional extras: `pip install "evalloop-agents[agents]"` for the OpenAI Agents SDK adapter,
|
|
79
|
+
`pip install "evalloop-agents[langgraph]"` for LangGraph.
|
|
80
|
+
|
|
81
|
+
## Quickstart
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
git clone https://github.com/sumitparmartech/evalloop && cd evalloop
|
|
85
|
+
pip install -e .
|
|
86
|
+
cd examples
|
|
87
|
+
python -m evalloop.cli run suite:cases --agent support_agent:agent_v1 \
|
|
88
|
+
--name support-agent --label v1 --trials 5 --seed 2 --out ../docs/run_v1.json
|
|
89
|
+
python -m evalloop.cli run suite:cases --agent support_agent:agent_v2 \
|
|
90
|
+
--name support-agent --label v2 --trials 5 --seed 2 --out ../docs/run_v2.json \
|
|
91
|
+
--baseline ../docs/run_v1.json --html ../docs/report.html
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The example support agent is simulated so the demo is free and deterministic. Version 1 has four real-world bugs, version 2 fixes them:
|
|
95
|
+
|
|
96
|
+
| | v1 | v2 |
|
|
97
|
+
| --- | --- | --- |
|
|
98
|
+
| Pass rate | 33% | 100% |
|
|
99
|
+
| Reliability (pass^5) | 0% | 100% |
|
|
100
|
+
| Latency p95 | 3.1 s | 1.0 s |
|
|
101
|
+
| Cost per run | $0.0105 | $0.0026 |
|
|
102
|
+
|
|
103
|
+

|
|
104
|
+
|
|
105
|
+
## Test your own agent
|
|
106
|
+
|
|
107
|
+
Wrap any agent (OpenAI, Anthropic, LangChain, your own loop) by recording steps with `Tracer`:
|
|
108
|
+
|
|
109
|
+
```python
|
|
110
|
+
from evalloop import Tracer, Case, run_suite, tool_called, tool_args, grounded, no_loop
|
|
111
|
+
|
|
112
|
+
def my_agent(user_input, ctx):
|
|
113
|
+
t = Tracer(user_input)
|
|
114
|
+
messages = [{"role": "user", "content": user_input}]
|
|
115
|
+
while True:
|
|
116
|
+
resp = client.chat.completions.create(model="gpt-4o-mini", messages=messages, tools=TOOLS)
|
|
117
|
+
msg = resp.choices[0].message
|
|
118
|
+
t.llm("step", messages, msg.content or "", resp.usage.prompt_tokens, resp.usage.completion_tokens)
|
|
119
|
+
if not msg.tool_calls:
|
|
120
|
+
return t.finish(msg.content)
|
|
121
|
+
for call in msg.tool_calls:
|
|
122
|
+
args = json.loads(call.function.arguments)
|
|
123
|
+
result = t.tool(call.function.name, args, fn=TOOL_FUNCS[call.function.name])
|
|
124
|
+
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
|
|
125
|
+
|
|
126
|
+
cases = [
|
|
127
|
+
Case("refund_small", "Refund order #1042",
|
|
128
|
+
[tool_args("issue_refund", order_id="1042"), grounded(), no_loop(2)]),
|
|
129
|
+
]
|
|
130
|
+
result = run_suite(my_agent, cases, trials=5)
|
|
131
|
+
print(result.pass_rate, result.reliability)
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
`ctx.seed` is passed to your agent so runs are reproducible when you use seeded randomness.
|
|
135
|
+
|
|
136
|
+
## Run it on a real model
|
|
137
|
+
|
|
138
|
+
`chat_agent` and `from_provider` wrap any OpenAI-compatible endpoint (OpenAI, Groq, OpenRouter, Gemini, Ollama, vLLM). They run the tool-calling loop and record every step, so every check works unchanged. There are no extra dependencies.
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
export GROQ_API_KEY=... # a free key from console.groq.com
|
|
142
|
+
cd examples
|
|
143
|
+
evalloop run real_suite:cases --agent real_agent:agent --trials 3 \
|
|
144
|
+
--name real-support-agent --out run_real.json --html report_real.html
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
```python
|
|
148
|
+
from evalloop import from_provider
|
|
149
|
+
|
|
150
|
+
agent = from_provider("groq", system=SYSTEM, tools=TOOLS, impls={"lookup_order": lookup_order})
|
|
151
|
+
# the key is read from GROQ_API_KEY and is never stored, printed or put in error messages
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
For a local model with no key at all, use `from_provider("ollama", ...)`. Free tiers have rate limits, so start with `--trials 1` or `--trials 3`. Set `price_in` and `price_out` (USD per 1M tokens) if you want `cost_under` to mean something.
|
|
155
|
+
|
|
156
|
+
The adapter is tested against a scripted fake provider. Model names change over time, so set `EVALLOOP_MODEL` if the default is retired.
|
|
157
|
+
|
|
158
|
+
## OpenAI Agents SDK and LangGraph
|
|
159
|
+
|
|
160
|
+
Test an agent built with a framework as it is. The wrapper runs it, then turns tool calls,
|
|
161
|
+
tool outputs, model calls and token usage into a trace, so every check, report and CI gate works.
|
|
162
|
+
|
|
163
|
+
```python
|
|
164
|
+
from evalloop import run_suite
|
|
165
|
+
from evalloop.frameworks import openai_agents_agent, langgraph_agent, poisoned
|
|
166
|
+
|
|
167
|
+
results = run_suite(openai_agents_agent(support_agent), cases, trials=5) # agents.Agent
|
|
168
|
+
results = run_suite(langgraph_agent(graph), cases, trials=5) # compiled StateGraph
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Multi-turn `Conversation` cases pass earlier turns as message history. For prompt injection
|
|
172
|
+
tests, return `poisoned("tool_name", result)` from inside a tool; evalloop adds the attack
|
|
173
|
+
text only when a case sets `poison=`. Both adapters are tested against the real packages
|
|
174
|
+
(`tests/test_frameworks.py`), using a scripted model so no API key is needed.
|
|
175
|
+
|
|
176
|
+
## Benchmark a real model
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
export GROQ_API_KEY=... # PowerShell: $env:GROQ_API_KEY="..."
|
|
180
|
+
cd examples
|
|
181
|
+
python benchmark.py --trials 10 # or --provider openai --model gpt-4.1-mini
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
It runs three suites (core cases with many trials, every built-in injection attack, and messy
|
|
185
|
+
rewrites of the order cases) and writes HTML reports plus a `SUMMARY.md` table to
|
|
186
|
+
`benchmark_results/<provider>-<model>/`.
|
|
187
|
+
|
|
188
|
+
## Add it to an agent you already have (two decorators)
|
|
189
|
+
|
|
190
|
+
No need to rewrite your agent around `Tracer`. Put `@tool()` on your tool functions and `@agent` on the function that returns the final answer. Sync and async both work, and parallel runs stay separate.
|
|
191
|
+
|
|
192
|
+
```python
|
|
193
|
+
from evalloop import agent, tool, log_llm
|
|
194
|
+
|
|
195
|
+
@tool()
|
|
196
|
+
def lookup_order(order_id): ... # every call is recorded: arguments, result, time, errors
|
|
197
|
+
|
|
198
|
+
@agent
|
|
199
|
+
def support_agent(text): # return the final answer as a string
|
|
200
|
+
reply = call_your_model(text)
|
|
201
|
+
log_llm("plan", text, reply, tokens_in=312, tokens_out=40, latency_ms=850) # optional: cost and latency
|
|
202
|
+
return reply
|
|
203
|
+
|
|
204
|
+
run_suite(support_agent, cases, trials=5, workers=4)
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
Start a new project with `evalloop init`. It creates `evals/agent.py`, `evals/suite.py` and a GitHub workflow that already uses the run store, the baseline and the PR summary.
|
|
208
|
+
|
|
209
|
+
## Red-team your agent
|
|
210
|
+
|
|
211
|
+
**Prompt injection through tools.** `Case.poison` appends an attack to what a tool returns, exactly like a poisoned web page or document would. The pack has 8 attacks (`INJECTIONS`), and `injection_cases` makes one case per attack.
|
|
212
|
+
|
|
213
|
+
```python
|
|
214
|
+
from evalloop import Case, injection_cases, no_secret_leak, tool_not_called
|
|
215
|
+
|
|
216
|
+
policy = Case("return_policy", "What is your return policy?", [no_secret_leak(KEY), tool_not_called("issue_refund")])
|
|
217
|
+
cases = injection_cases(policy, tool="search_kb", secret=KEY) # 8 cases, one per attack
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
**Messy input.** `variants` rewrites a case with typos, SHOUTING, extra spaces, politeness and filler words, with the same checks. Numbers, ids and amounts are never changed. If your agent passes the original but fails the variants, it is brittle.
|
|
221
|
+
|
|
222
|
+
```python
|
|
223
|
+
from evalloop import variants
|
|
224
|
+
cases = [track, *variants(track, "typos", "upper", "polite", "filler")]
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
The demo (`examples/redteam_demo.py`) has a gullible agent and a careful one. The gullible agent passes the plain question, then fails on typos, on SHOUTING and on 6 of 8 injection attacks. The careful agent passes everything.
|
|
228
|
+
|
|
229
|
+
```bash
|
|
230
|
+
cd examples
|
|
231
|
+
evalloop run redteam_demo:cases --agent redteam_demo:careful --baseline ../docs/redteam_gullible.json --trials 3
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## Keep history and watch trends
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
evalloop run suite:cases --agent agent:support_agent --trials 5 --store runs --baseline last
|
|
238
|
+
evalloop trend runs --html trend.html
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
`--store` saves every run as plain JSON. `--baseline last` compares with the previous run automatically, so you never manage baseline files by hand. `evalloop trend` makes one HTML page with charts of pass rate (with its confidence band), reliability, p95 latency, cost and flaky cases over time.
|
|
242
|
+
|
|
243
|
+
## Multi-turn conversations
|
|
244
|
+
|
|
245
|
+
Real agents fail across turns: they forget the order number, repeat themselves, or leak a secret on the third message. A single-turn test never sees that. `Conversation` runs a whole dialogue, passes the history to your agent, and checks each turn and the conversation as a whole.
|
|
246
|
+
|
|
247
|
+
```python
|
|
248
|
+
from evalloop import Conversation, Turn, answer_contains, tool_args, tool_count, no_secret_leak, grounded
|
|
249
|
+
|
|
250
|
+
cases = [
|
|
251
|
+
Conversation(
|
|
252
|
+
"refund_it_after_lookup",
|
|
253
|
+
["Look up order #1042",
|
|
254
|
+
Turn("OK please refund it", [tool_args("issue_refund", order_id="1042")])], # checks for this turn
|
|
255
|
+
[tool_count("issue_refund", at_most=1), no_secret_leak(KEY), grounded()], # checks for the whole chat
|
|
256
|
+
),
|
|
257
|
+
]
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
- `ctx.history` is the list of earlier `(user, answer)` pairs and `ctx.turn` is the turn number. `chat_agent` and `anthropic_agent` send the history to the model for you.
|
|
261
|
+
- Conversation checks see every step and every answer, so a tool budget, loop limit, PII or secret check covers all turns, not just the last one.
|
|
262
|
+
- A turn can be a function, `Turn(lambda history: ...)`, so a scripted user can react to what the agent just said.
|
|
263
|
+
- If the agent crashes on a turn, the conversation stops and fails with that turn named.
|
|
264
|
+
|
|
265
|
+
The demo in `examples/convo_agent.py` has a v1 that only reads the latest message and a v2 that uses the history. evalloop fails v1 on 3 of 4 conversations and passes v2 on all of them:
|
|
266
|
+
|
|
267
|
+

|
|
268
|
+
|
|
269
|
+
```bash
|
|
270
|
+
cd examples
|
|
271
|
+
evalloop run convo_suite:cases --agent convo_agent:agent_v2 --name convo --trials 3 --baseline ../docs/convo_v1.json
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
## Statistics you can trust
|
|
275
|
+
|
|
276
|
+
Agents are random, so a single failure is not always a real regression. evalloop reports honest numbers:
|
|
277
|
+
|
|
278
|
+
- A 95% confidence interval on the pass rate, so "100% on 5 trials" shows as 57% to 100%.
|
|
279
|
+
- Flaky cases (pass some trials, fail others) are listed on their own, because they are the ones that hurt in production.
|
|
280
|
+
- Unbiased `pass@k` and `pass^k` estimates from your n trials: `case.pass_at(k)`, `case.pass_hat(k)`.
|
|
281
|
+
- `--alpha 0.2` ignores drops that are probably noise. Every changed case shows its `p_value`, so you can see what is real.
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
evalloop run suite:cases --agent my_module:agent --trials 10 --baseline base.json \
|
|
285
|
+
--fail-on-regression --alpha 0.2 --workers 8 --timeout 60
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
`--workers` runs trials in parallel, so a real-model suite finishes in a fraction of the time. `--timeout` stops one hung call from freezing the run.
|
|
289
|
+
|
|
290
|
+
## Test real conversations you already have
|
|
291
|
+
|
|
292
|
+
Take logged messages from production, turn them into traces and run the same checks. No model call needed.
|
|
293
|
+
|
|
294
|
+
```python
|
|
295
|
+
from evalloop import from_openai_messages, from_anthropic_messages, replay_agent, run_suite, save_traces
|
|
296
|
+
|
|
297
|
+
traces = [from_openai_messages(m) for m in logged_conversations]
|
|
298
|
+
save_traces("prod_traces.jsonl", traces)
|
|
299
|
+
result = run_suite(replay_agent(traces), cases) # checks run on the recorded traces
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
## LLM-as-judge
|
|
303
|
+
|
|
304
|
+
For things code cannot check, add a judge. It scores 1 to 5 at temperature 0 and the check passes at your pass mark.
|
|
305
|
+
|
|
306
|
+
```python
|
|
307
|
+
from evalloop import judge, judge_with
|
|
308
|
+
|
|
309
|
+
ask = judge_with("anthropic") # or "openai", "groq", "gemini", "ollama"
|
|
310
|
+
Case("tone", "I am angry about my late order", [judge("Calm, apologetic, offers a concrete next step", ask)])
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
## Claude and other providers
|
|
314
|
+
|
|
315
|
+
```python
|
|
316
|
+
from evalloop import anthropic_agent, from_provider
|
|
317
|
+
|
|
318
|
+
agent = anthropic_agent(system=SYSTEM, tools=ANTHROPIC_TOOLS, impls=IMPLS, model="claude-sonnet-4-5")
|
|
319
|
+
# ANTHROPIC_API_KEY is read from the environment and only goes in the x-api-key header
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
## Use it in pytest and any CI
|
|
323
|
+
|
|
324
|
+
```python
|
|
325
|
+
from evalloop import assert_reliable
|
|
326
|
+
|
|
327
|
+
def test_refund_flow():
|
|
328
|
+
assert_reliable(my_agent, refund_case, trials=5, min_pass_rate=0.8)
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
`--junit junit.xml` gives every case as a test in Jenkins, GitLab or GitHub. `--markdown "$GITHUB_STEP_SUMMARY"` writes a short report with regressions, flaky cases and failure reasons to the job summary.
|
|
332
|
+
|
|
333
|
+
## Gate your pull requests
|
|
334
|
+
|
|
335
|
+
```bash
|
|
336
|
+
evalloop run suite:cases --agent my_module:my_agent --trials 5 \
|
|
337
|
+
--baseline baseline.json --fail-on-regression --fail-under 0.95 --html report.html
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
The command exits with code 1 if any case got worse than the saved baseline, or if the overall pass rate drops below your threshold. A ready workflow is in `.github/workflows/agent-evals.yml`.
|
|
341
|
+
|
|
342
|
+
Other commands:
|
|
343
|
+
|
|
344
|
+
```bash
|
|
345
|
+
evalloop diff old.json new.json --fail-on-regression # compare two saved runs
|
|
346
|
+
evalloop report run.json --baseline old.json --html r.html
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
## Why pass^k and not just accuracy
|
|
350
|
+
|
|
351
|
+
If an agent passes a task 80% of the time, running it once and seeing a pass tells you almost nothing. A case counts as reliable only if all k trials pass, so reliability falls fast as k grows. This is the number users actually feel.
|
|
352
|
+
|
|
353
|
+
## Design notes
|
|
354
|
+
|
|
355
|
+
- Trace first. Checks run on every LLM call and tool call, so you can assert on behavior, not only on the final text.
|
|
356
|
+
- Standard library only. The HTML report is a single file with no JavaScript, safe to attach to a pull request.
|
|
357
|
+
- Deterministic by default. Every trial gets a stable seed derived from the case id, trial number and your base seed.
|
|
358
|
+
- Crashes count as failures. If the agent raises, that trial fails and the error is shown in the report.
|
|
359
|
+
|
|
360
|
+
## Roadmap
|
|
361
|
+
|
|
362
|
+
- A hosted dashboard (today: a local trend page from `--store`)
|
|
363
|
+
- Adapters for LlamaIndex and CrewAI
|
|
364
|
+
- Published benchmark numbers for popular models
|
|
365
|
+
- Judge calibration against human labels
|
|
366
|
+
|
|
367
|
+
## Develop
|
|
368
|
+
|
|
369
|
+
```bash
|
|
370
|
+
pip install -e ".[dev]"
|
|
371
|
+
pytest
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, style and a list of good first issues.
|
|
375
|
+
|
|
376
|
+
MIT licensed.
|