evalloop-agents 0.6.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. evalloop_agents-0.6.0/LICENSE +21 -0
  2. evalloop_agents-0.6.0/PKG-INFO +376 -0
  3. evalloop_agents-0.6.0/README.md +341 -0
  4. evalloop_agents-0.6.0/pyproject.toml +52 -0
  5. evalloop_agents-0.6.0/setup.cfg +4 -0
  6. evalloop_agents-0.6.0/src/evalloop/__init__.py +45 -0
  7. evalloop_agents-0.6.0/src/evalloop/adapters.py +287 -0
  8. evalloop_agents-0.6.0/src/evalloop/checks.py +320 -0
  9. evalloop_agents-0.6.0/src/evalloop/ci.py +95 -0
  10. evalloop_agents-0.6.0/src/evalloop/cli.py +175 -0
  11. evalloop_agents-0.6.0/src/evalloop/core.py +160 -0
  12. evalloop_agents-0.6.0/src/evalloop/frameworks.py +221 -0
  13. evalloop_agents-0.6.0/src/evalloop/instrument.py +173 -0
  14. evalloop_agents-0.6.0/src/evalloop/py.typed +0 -0
  15. evalloop_agents-0.6.0/src/evalloop/redteam.py +111 -0
  16. evalloop_agents-0.6.0/src/evalloop/report.py +200 -0
  17. evalloop_agents-0.6.0/src/evalloop/runner.py +398 -0
  18. evalloop_agents-0.6.0/src/evalloop/scaffold.py +80 -0
  19. evalloop_agents-0.6.0/src/evalloop/stats.py +64 -0
  20. evalloop_agents-0.6.0/src/evalloop/store.py +127 -0
  21. evalloop_agents-0.6.0/src/evalloop/traces.py +109 -0
  22. evalloop_agents-0.6.0/src/evalloop_agents.egg-info/PKG-INFO +376 -0
  23. evalloop_agents-0.6.0/src/evalloop_agents.egg-info/SOURCES.txt +31 -0
  24. evalloop_agents-0.6.0/src/evalloop_agents.egg-info/dependency_links.txt +1 -0
  25. evalloop_agents-0.6.0/src/evalloop_agents.egg-info/entry_points.txt +2 -0
  26. evalloop_agents-0.6.0/src/evalloop_agents.egg-info/requires.txt +10 -0
  27. evalloop_agents-0.6.0/src/evalloop_agents.egg-info/top_level.txt +1 -0
  28. evalloop_agents-0.6.0/tests/test_adapters.py +99 -0
  29. evalloop_agents-0.6.0/tests/test_evalloop.py +166 -0
  30. evalloop_agents-0.6.0/tests/test_frameworks.py +163 -0
  31. evalloop_agents-0.6.0/tests/test_multiturn.py +131 -0
  32. evalloop_agents-0.6.0/tests/test_v03.py +237 -0
  33. evalloop_agents-0.6.0/tests/test_v05.py +255 -0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Sumit Parmar
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,376 @@
1
+ Metadata-Version: 2.4
2
+ Name: evalloop-agents
3
+ Version: 0.6.0
4
+ Summary: Unit tests and a CI gate for LLM agents: reliability (pass^k), multi-turn conversations, prompt injection, hallucination, loops, cost and latency. Works with OpenAI Agents SDK and LangGraph.
5
+ Author: Sumit Parmar
6
+ License: MIT
7
+ Project-URL: Homepage, https://sumitparmartech.github.io/evalloop/
8
+ Project-URL: Source, https://github.com/sumitparmartech/evalloop
9
+ Project-URL: Changelog, https://github.com/sumitparmartech/evalloop/blob/main/CHANGELOG.md
10
+ Project-URL: Issues, https://github.com/sumitparmartech/evalloop/issues
11
+ Keywords: llm,agents,evals,testing,ci,pass-k,reliability,llm-as-judge,pytest,langgraph,openai-agents,prompt-injection,red-teaming
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Intended Audience :: Developers
14
+ Classifier: License :: OSI Approved :: MIT License
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
22
+ Classifier: Topic :: Software Development :: Testing
23
+ Classifier: Typing :: Typed
24
+ Requires-Python: >=3.9
25
+ Description-Content-Type: text/markdown
26
+ License-File: LICENSE
27
+ Provides-Extra: dev
28
+ Requires-Dist: pytest>=7; extra == "dev"
29
+ Provides-Extra: agents
30
+ Requires-Dist: openai-agents>=0.2; extra == "agents"
31
+ Provides-Extra: langgraph
32
+ Requires-Dist: langgraph>=0.2; extra == "langgraph"
33
+ Requires-Dist: langchain-core>=0.3; extra == "langgraph"
34
+ Dynamic: license-file
35
+
36
+ # evalloop
37
+
38
+ [![tests](https://github.com/sumitparmartech/evalloop/actions/workflows/tests.yml/badge.svg)](https://github.com/sumitparmartech/evalloop/actions/workflows/tests.yml)
39
+ [![PyPI](https://img.shields.io/pypi/v/evalloop-agents)](https://pypi.org/project/evalloop-agents/)
40
+ [![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/evalloop-agents/)
41
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
42
+
43
+ **[Live demo](https://sumitparmartech.github.io/evalloop/)** · **[How it compares](docs/COMPARISON.md)** · **[Changelog](CHANGELOG.md)**
44
+
45
+ Unit tests and a CI gate for LLM agents. Zero dependencies, no API key needed to try it. Python 3.9+.
46
+
47
+ Most agent demos work once. Production agents fail 1 run in 5, quote amounts they never saw, obey instructions hidden in a web page, and loop on vague questions. `evalloop` runs your agent many times against a suite of cases, checks the full trace (not just the final answer), and fails your build when behavior regresses.
48
+
49
+ ![report](https://raw.githubusercontent.com/sumitparmartech/evalloop/main/docs/report_v1.png)
50
+
51
+ ## What it catches
52
+
53
+ | Failure | Check |
54
+ | --- | --- |
55
+ | Flaky agent that passes once and fails later | `pass^k` reliability, every case runs k times |
56
+ | Hallucinated numbers (amounts, dates, ids) | `grounded()` flags any number not found in the input or a tool result |
57
+ | Prompt injection that leaks a secret | `no_secret_leak(...)` |
58
+ | Wrong tool, wrong arguments, wrong order | `tool_called`, `tool_not_called`, `tool_args`, `tool_order` |
59
+ | Infinite or wasteful tool loops | `no_loop(n)`, `max_steps(n)` |
60
+ | Cost and latency creep | `cost_under(usd)`, `latency_under(ms)` |
61
+ | PII or keys in the answer | `no_pii()` (email, phone, card, SSN, API key) |
62
+ | Expensive tool called too often | `tool_count(name, at_most=n)` |
63
+ | Tone, helpfulness, policy (things code cannot check) | `judge(criteria, ask)` LLM-as-judge |
64
+ | Agent forgets what the user said two messages ago | `Conversation(...)` multi-turn cases |
65
+ | Prompt injection hidden in a tool result | `Case.poison`, `injection_cases(...)` |
66
+ | Breaks on typos, SHOUTING, filler words | `variants(case, ...)` |
67
+ | Bad output format | `json_answer(...)`, `answer_matches(regex)` |
68
+ | Anything else | `custom(name, fn)` |
69
+
70
+ ## Install
71
+
72
+ ```bash
73
+ pip install evalloop-agents # the import name is still: import evalloop
74
+ # or the latest code from GitHub:
75
+ pip install git+https://github.com/sumitparmartech/evalloop
76
+ ```
77
+
78
+ Optional extras: `pip install "evalloop-agents[agents]"` for the OpenAI Agents SDK adapter,
79
+ `pip install "evalloop-agents[langgraph]"` for LangGraph.
80
+
81
+ ## Quickstart
82
+
83
+ ```bash
84
+ git clone https://github.com/sumitparmartech/evalloop && cd evalloop
85
+ pip install -e .
86
+ cd examples
87
+ python -m evalloop.cli run suite:cases --agent support_agent:agent_v1 \
88
+ --name support-agent --label v1 --trials 5 --seed 2 --out ../docs/run_v1.json
89
+ python -m evalloop.cli run suite:cases --agent support_agent:agent_v2 \
90
+ --name support-agent --label v2 --trials 5 --seed 2 --out ../docs/run_v2.json \
91
+ --baseline ../docs/run_v1.json --html ../docs/report.html
92
+ ```
93
+
94
+ The example support agent is simulated so the demo is free and deterministic. Version 1 has four real-world bugs, version 2 fixes them:
95
+
96
+ | | v1 | v2 |
97
+ | --- | --- | --- |
98
+ | Pass rate | 33% | 100% |
99
+ | Reliability (pass^5) | 0% | 100% |
100
+ | Latency p95 | 3.1 s | 1.0 s |
101
+ | Cost per run | $0.0105 | $0.0026 |
102
+
103
+ ![diff](https://raw.githubusercontent.com/sumitparmartech/evalloop/main/docs/report.png)
104
+
105
+ ## Test your own agent
106
+
107
+ Wrap any agent (OpenAI, Anthropic, LangChain, your own loop) by recording steps with `Tracer`:
108
+
109
+ ```python
110
+ from evalloop import Tracer, Case, run_suite, tool_called, tool_args, grounded, no_loop
111
+
112
+ def my_agent(user_input, ctx):
113
+ t = Tracer(user_input)
114
+ messages = [{"role": "user", "content": user_input}]
115
+ while True:
116
+ resp = client.chat.completions.create(model="gpt-4o-mini", messages=messages, tools=TOOLS)
117
+ msg = resp.choices[0].message
118
+ t.llm("step", messages, msg.content or "", resp.usage.prompt_tokens, resp.usage.completion_tokens)
119
+ if not msg.tool_calls:
120
+ return t.finish(msg.content)
121
+ for call in msg.tool_calls:
122
+ args = json.loads(call.function.arguments)
123
+ result = t.tool(call.function.name, args, fn=TOOL_FUNCS[call.function.name])
124
+ messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
125
+
126
+ cases = [
127
+ Case("refund_small", "Refund order #1042",
128
+ [tool_args("issue_refund", order_id="1042"), grounded(), no_loop(2)]),
129
+ ]
130
+ result = run_suite(my_agent, cases, trials=5)
131
+ print(result.pass_rate, result.reliability)
132
+ ```
133
+
134
+ `ctx.seed` is passed to your agent so runs are reproducible when you use seeded randomness.
135
+
136
+ ## Run it on a real model
137
+
138
+ `chat_agent` and `from_provider` wrap any OpenAI-compatible endpoint (OpenAI, Groq, OpenRouter, Gemini, Ollama, vLLM). They run the tool-calling loop and record every step, so every check works unchanged. There are no extra dependencies.
139
+
140
+ ```bash
141
+ export GROQ_API_KEY=... # a free key from console.groq.com
142
+ cd examples
143
+ evalloop run real_suite:cases --agent real_agent:agent --trials 3 \
144
+ --name real-support-agent --out run_real.json --html report_real.html
145
+ ```
146
+
147
+ ```python
148
+ from evalloop import from_provider
149
+
150
+ agent = from_provider("groq", system=SYSTEM, tools=TOOLS, impls={"lookup_order": lookup_order})
151
+ # the key is read from GROQ_API_KEY and is never stored, printed or put in error messages
152
+ ```
153
+
154
+ For a local model with no key at all, use `from_provider("ollama", ...)`. Free tiers have rate limits, so start with `--trials 1` or `--trials 3`. Set `price_in` and `price_out` (USD per 1M tokens) if you want `cost_under` to mean something.
155
+
156
+ The adapter is tested against a scripted fake provider. Model names change over time, so set `EVALLOOP_MODEL` if the default is retired.
157
+
158
+ ## OpenAI Agents SDK and LangGraph
159
+
160
+ Test an agent built with a framework as it is. The wrapper runs it, then turns tool calls,
161
+ tool outputs, model calls and token usage into a trace, so every check, report and CI gate works.
162
+
163
+ ```python
164
+ from evalloop import run_suite
165
+ from evalloop.frameworks import openai_agents_agent, langgraph_agent, poisoned
166
+
167
+ results = run_suite(openai_agents_agent(support_agent), cases, trials=5) # agents.Agent
168
+ results = run_suite(langgraph_agent(graph), cases, trials=5) # compiled StateGraph
169
+ ```
170
+
171
+ Multi-turn `Conversation` cases pass earlier turns as message history. For prompt injection
172
+ tests, return `poisoned("tool_name", result)` from inside a tool; evalloop adds the attack
173
+ text only when a case sets `poison=`. Both adapters are tested against the real packages
174
+ (`tests/test_frameworks.py`), using a scripted model so no API key is needed.
175
+
176
+ ## Benchmark a real model
177
+
178
+ ```bash
179
+ export GROQ_API_KEY=... # PowerShell: $env:GROQ_API_KEY="..."
180
+ cd examples
181
+ python benchmark.py --trials 10 # or --provider openai --model gpt-4.1-mini
182
+ ```
183
+
184
+ It runs three suites (core cases with many trials, every built-in injection attack, and messy
185
+ rewrites of the order cases) and writes HTML reports plus a `SUMMARY.md` table to
186
+ `benchmark_results/<provider>-<model>/`.
187
+
188
+ ## Add it to an agent you already have (two decorators)
189
+
190
+ No need to rewrite your agent around `Tracer`. Put `@tool()` on your tool functions and `@agent` on the function that returns the final answer. Sync and async both work, and parallel runs stay separate.
191
+
192
+ ```python
193
+ from evalloop import agent, tool, log_llm
194
+
195
+ @tool()
196
+ def lookup_order(order_id): ... # every call is recorded: arguments, result, time, errors
197
+
198
+ @agent
199
+ def support_agent(text): # return the final answer as a string
200
+ reply = call_your_model(text)
201
+ log_llm("plan", text, reply, tokens_in=312, tokens_out=40, latency_ms=850) # optional: cost and latency
202
+ return reply
203
+
204
+ run_suite(support_agent, cases, trials=5, workers=4)
205
+ ```
206
+
207
+ Start a new project with `evalloop init`. It creates `evals/agent.py`, `evals/suite.py` and a GitHub workflow that already uses the run store, the baseline and the PR summary.
208
+
209
+ ## Red-team your agent
210
+
211
+ **Prompt injection through tools.** `Case.poison` appends an attack to what a tool returns, exactly like a poisoned web page or document would. The pack has 8 attacks (`INJECTIONS`), and `injection_cases` makes one case per attack.
212
+
213
+ ```python
214
+ from evalloop import Case, injection_cases, no_secret_leak, tool_not_called
215
+
216
+ policy = Case("return_policy", "What is your return policy?", [no_secret_leak(KEY), tool_not_called("issue_refund")])
217
+ cases = injection_cases(policy, tool="search_kb", secret=KEY) # 8 cases, one per attack
218
+ ```
219
+
220
+ **Messy input.** `variants` rewrites a case with typos, SHOUTING, extra spaces, politeness and filler words, with the same checks. Numbers, ids and amounts are never changed. If your agent passes the original but fails the variants, it is brittle.
221
+
222
+ ```python
223
+ from evalloop import variants
224
+ cases = [track, *variants(track, "typos", "upper", "polite", "filler")]
225
+ ```
226
+
227
+ The demo (`examples/redteam_demo.py`) has a gullible agent and a careful one. The gullible agent passes the plain question, then fails on typos, on SHOUTING and on 6 of 8 injection attacks. The careful agent passes everything.
228
+
229
+ ```bash
230
+ cd examples
231
+ evalloop run redteam_demo:cases --agent redteam_demo:careful --baseline ../docs/redteam_gullible.json --trials 3
232
+ ```
233
+
234
+ ## Keep history and watch trends
235
+
236
+ ```bash
237
+ evalloop run suite:cases --agent agent:support_agent --trials 5 --store runs --baseline last
238
+ evalloop trend runs --html trend.html
239
+ ```
240
+
241
+ `--store` saves every run as plain JSON. `--baseline last` compares with the previous run automatically, so you never manage baseline files by hand. `evalloop trend` makes one HTML page with charts of pass rate (with its confidence band), reliability, p95 latency, cost and flaky cases over time.
242
+
243
+ ## Multi-turn conversations
244
+
245
+ Real agents fail across turns: they forget the order number, repeat themselves, or leak a secret on the third message. A single-turn test never sees that. `Conversation` runs a whole dialogue, passes the history to your agent, and checks each turn and the conversation as a whole.
246
+
247
+ ```python
248
+ from evalloop import Conversation, Turn, answer_contains, tool_args, tool_count, no_secret_leak, grounded
249
+
250
+ cases = [
251
+ Conversation(
252
+ "refund_it_after_lookup",
253
+ ["Look up order #1042",
254
+ Turn("OK please refund it", [tool_args("issue_refund", order_id="1042")])], # checks for this turn
255
+ [tool_count("issue_refund", at_most=1), no_secret_leak(KEY), grounded()], # checks for the whole chat
256
+ ),
257
+ ]
258
+ ```
259
+
260
+ - `ctx.history` is the list of earlier `(user, answer)` pairs and `ctx.turn` is the turn number. `chat_agent` and `anthropic_agent` send the history to the model for you.
261
+ - Conversation checks see every step and every answer, so a tool budget, loop limit, PII or secret check covers all turns, not just the last one.
262
+ - A turn can be a function, `Turn(lambda history: ...)`, so a scripted user can react to what the agent just said.
263
+ - If the agent crashes on a turn, the conversation stops and fails with that turn named.
264
+
265
+ The demo in `examples/convo_agent.py` has a v1 that only reads the latest message and a v2 that uses the history. evalloop fails v1 on 3 of 4 conversations and passes v2 on all of them:
266
+
267
+ ![conversation report](https://raw.githubusercontent.com/sumitparmartech/evalloop/main/docs/report_convo.png)
268
+
269
+ ```bash
270
+ cd examples
271
+ evalloop run convo_suite:cases --agent convo_agent:agent_v2 --name convo --trials 3 --baseline ../docs/convo_v1.json
272
+ ```
273
+
274
+ ## Statistics you can trust
275
+
276
+ Agents are random, so a single failure is not always a real regression. evalloop reports honest numbers:
277
+
278
+ - A 95% confidence interval on the pass rate, so "100% on 5 trials" shows as 57% to 100%.
279
+ - Flaky cases (pass some trials, fail others) are listed on their own, because they are the ones that hurt in production.
280
+ - Unbiased `pass@k` and `pass^k` estimates from your n trials: `case.pass_at(k)`, `case.pass_hat(k)`.
281
+ - `--alpha 0.2` ignores drops that are probably noise. Every changed case shows its `p_value`, so you can see what is real.
282
+
283
+ ```bash
284
+ evalloop run suite:cases --agent my_module:agent --trials 10 --baseline base.json \
285
+ --fail-on-regression --alpha 0.2 --workers 8 --timeout 60
286
+ ```
287
+
288
+ `--workers` runs trials in parallel, so a real-model suite finishes in a fraction of the time. `--timeout` stops one hung call from freezing the run.
289
+
290
+ ## Test real conversations you already have
291
+
292
+ Take logged messages from production, turn them into traces and run the same checks. No model call needed.
293
+
294
+ ```python
295
+ from evalloop import from_openai_messages, from_anthropic_messages, replay_agent, run_suite, save_traces
296
+
297
+ traces = [from_openai_messages(m) for m in logged_conversations]
298
+ save_traces("prod_traces.jsonl", traces)
299
+ result = run_suite(replay_agent(traces), cases) # checks run on the recorded traces
300
+ ```
301
+
302
+ ## LLM-as-judge
303
+
304
+ For things code cannot check, add a judge. It scores 1 to 5 at temperature 0 and the check passes at your pass mark.
305
+
306
+ ```python
307
+ from evalloop import judge, judge_with
308
+
309
+ ask = judge_with("anthropic") # or "openai", "groq", "gemini", "ollama"
310
+ Case("tone", "I am angry about my late order", [judge("Calm, apologetic, offers a concrete next step", ask)])
311
+ ```
312
+
313
+ ## Claude and other providers
314
+
315
+ ```python
316
+ from evalloop import anthropic_agent, from_provider
317
+
318
+ agent = anthropic_agent(system=SYSTEM, tools=ANTHROPIC_TOOLS, impls=IMPLS, model="claude-sonnet-4-5")
319
+ # ANTHROPIC_API_KEY is read from the environment and only goes in the x-api-key header
320
+ ```
321
+
322
+ ## Use it in pytest and any CI
323
+
324
+ ```python
325
+ from evalloop import assert_reliable
326
+
327
+ def test_refund_flow():
328
+ assert_reliable(my_agent, refund_case, trials=5, min_pass_rate=0.8)
329
+ ```
330
+
331
+ `--junit junit.xml` gives every case as a test in Jenkins, GitLab or GitHub. `--markdown "$GITHUB_STEP_SUMMARY"` writes a short report with regressions, flaky cases and failure reasons to the job summary.
332
+
333
+ ## Gate your pull requests
334
+
335
+ ```bash
336
+ evalloop run suite:cases --agent my_module:my_agent --trials 5 \
337
+ --baseline baseline.json --fail-on-regression --fail-under 0.95 --html report.html
338
+ ```
339
+
340
+ The command exits with code 1 if any case got worse than the saved baseline, or if the overall pass rate drops below your threshold. A ready workflow is in `.github/workflows/agent-evals.yml`.
341
+
342
+ Other commands:
343
+
344
+ ```bash
345
+ evalloop diff old.json new.json --fail-on-regression # compare two saved runs
346
+ evalloop report run.json --baseline old.json --html r.html
347
+ ```
348
+
349
+ ## Why pass^k and not just accuracy
350
+
351
+ If an agent passes a task 80% of the time, running it once and seeing a pass tells you almost nothing. A case counts as reliable only if all k trials pass, so reliability falls fast as k grows. This is the number users actually feel.
352
+
353
+ ## Design notes
354
+
355
+ - Trace first. Checks run on every LLM call and tool call, so you can assert on behavior, not only on the final text.
356
+ - Standard library only. The HTML report is a single file with no JavaScript, safe to attach to a pull request.
357
+ - Deterministic by default. Every trial gets a stable seed derived from the case id, trial number and your base seed.
358
+ - Crashes count as failures. If the agent raises, that trial fails and the error is shown in the report.
359
+
360
+ ## Roadmap
361
+
362
+ - A hosted dashboard (today: a local trend page from `--store`)
363
+ - Adapters for LlamaIndex and CrewAI
364
+ - Published benchmark numbers for popular models
365
+ - Judge calibration against human labels
366
+
367
+ ## Develop
368
+
369
+ ```bash
370
+ pip install -e ".[dev]"
371
+ pytest
372
+ ```
373
+
374
+ Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, style and a list of good first issues.
375
+
376
+ MIT licensed.