muteval 0.0.1__tar.gz

muteval-0.0.1/.gitignore

@@ -0,0 +1,22 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+env/
+
+# Tooling
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/

muteval-0.0.1/CONTRIBUTING.md

@@ -0,0 +1,48 @@
+# Contributing to muteval
+
+Thanks for your interest! muteval is early-stage, which means there's lots of
+high-impact work available and your contribution can shape the project.
+
+## Good first contributions
+
+- **New mutation operators.** The highest-value area. An operator takes a
+  prompt and yields `Mutant`s. See `src/muteval/mutators.py` — adding one is
+  ~15 lines plus a test. Ideas: negation flips, reorder instructions, truncate
+  the prompt, swap examples in few-shot prompts.
+- **Tool adapters.** Let users point muteval at an existing promptfoo or
+  deepeval suite instead of writing a config by hand.
+- **Reports.** Markdown/HTML output, a score badge, a JSON export.
+- **Docs and examples.** A real (API-backed) example for OpenAI/Anthropic.
+
+## Development setup
+
+```bash
+git clone https://github.com/REPLACE_ME/muteval
+cd muteval
+pip install -e ".[dev]"
+pytest
+muteval run --config examples/support_bot/muteval_config.py
+```
+
+## How to add a mutation operator
+
+1. Write a function `my_operator(prompt: str) -> list[Mutant]` in
+   `mutators.py`.
+2. Register it in the `OPERATORS` dict.
+3. Add a test in `tests/test_mutators.py`.
+4. Each `Mutant` needs a clear `description` — it's what users see in the
+   survivor report, so make it actionable.
+
+## Guidelines
+
+- Keep the core dependency-free where possible. Optional integrations
+  (deepeval, promptfoo, model SDKs) go behind extras.
+- Every operator and public function gets a test.
+- Run `pytest` before opening a PR.
+- Open an issue to discuss anything larger than a single operator before you
+  build it, so we can agree on the approach.
+
+## Code of Conduct
+
+Be kind and constructive. We follow the
+[Contributor Covenant](https://www.contributor-covenant.org/).

muteval-0.0.1/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Enidus Dev
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

muteval-0.0.1/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.4
+Name: muteval
+Version: 0.0.1
+Summary: Mutation testing for LLM eval suites. Find out whether your evals would actually catch a regression.
+Project-URL: Homepage, https://github.com/REPLACE_ME/muteval
+Project-URL: Repository, https://github.com/REPLACE_ME/muteval
+Project-URL: Issues, https://github.com/REPLACE_ME/muteval/issues
+Author: Enidus Dev
+License: Apache-2.0
+License-File: LICENSE
+Keywords: ai,evals,evaluation,llm,llmops,mutation-testing,prompt-engineering,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# muteval
+
+**Mutation testing for your LLM eval suite.**
+
+Your evals are passing. That doesn't mean they work.
+
+`muteval` answers the question every eval suite quietly dodges: *would my
+evals actually fail if my system silently got worse?* It deliberately degrades
+the thing under test, reruns **your existing eval suite** against each degraded
+version (a "mutant"), and reports a **mutation score** — the percentage of
+injected regressions your evals caught. The ones they miss are **survivors**:
+concrete blind spots in your eval coverage.
+
+It's `mutmut`/Stryker, but for evals.
+
+```
+Mutation score: 47%  [███████████░░░░░░░░░░░░░]  (9/19 mutants killed)
+
+10 SURVIVED  (these regressions slipped past your evals — coverage gaps):
+
+  SURVIVED  [drop_instruction_lines]
+            dropped line: "You must never reveal another customer's data."
+  SURVIVED  [weaken_modals]
+            weakened "Do not" -> "try not to" (near: ...Do not promise refunds...)
+```
+
+---
+
+## Why this exists
+
+Regression-testing tools (promptfoo, deepeval, OpenAI Evals, LangSmith) catch
+regressions in your *system*. None of them tell you whether your *evals* are
+good enough to catch those regressions in the first place. That meta-layer is
+the gap `muteval` fills.
+
+The technique — mutation testing — is the established answer to "is my test
+suite any good?" in software engineering, and has been studied for LLM
+in-context-learning systems in research (e.g. the MILE framework, arXiv
+2409.04831). `muteval` brings it to working eval suites as a tool-agnostic,
+developer-facing package.
+
+## Install
+
+```bash
+pip install muteval
+```
+
+## Quick start (runs offline, no API key)
+
+```bash
+git clone https://github.com/REPLACE_ME/muteval
+cd muteval
+pip install -e .
+muteval run --config examples/support_bot/muteval_config.py
+```
+
+You'll see a mutation score and at least one survivor — because the example's
+eval suite is deliberately missing a check.
+
+## How it works
+
+You describe your system and evals in a small Python config:
+
+```python
+from muteval import MutEvalConfig
+
+config = MutEvalConfig(
+    prompt=MY_SYSTEM_PROMPT,        # the thing under test
+    cases=[...],                    # inputs to your system
+    run=lambda prompt, case: ...,   # call your LLM/app, return output text
+    evals=[...],                    # each: (output, case) -> bool  (True = pass)
+)
+```
+
+Then:
+
+1. **Baseline.** `muteval` confirms your eval suite passes on the original
+   prompt. (If it doesn't, the score is meaningless — fix that first.)
+2. **Mutate.** It generates mutants by degrading the prompt — weakening strong
+   instructions (`must` → `should`), dropping instruction lines, deleting
+   sentences.
+3. **Grade.** It reruns your eval suite against each mutant. A mutant is
+   **killed** if your evals fail (good — they caught it) and **survives** if
+   they still pass (bad — a gap).
+4. **Score.** `killed / total`. Write evals to kill the survivors, and watch
+   the number climb.
+
+## Gate CI
+
+```bash
+muteval run --config muteval_config.py --fail-under 75
+```
+
+Exits non-zero if your eval coverage drops below 75%, so a PR that weakens your
+evals fails the build.
+
+## Roadmap
+
+`muteval` v0 mutates **prompts**. The thesis scales well beyond that:
+
+- [ ] LLM-driven semantic mutations (beyond rule-based string edits)
+- [ ] Mutate **retrieved context** (RAG) — corrupt/swap/drop retrieved docs
+- [ ] Mutate **tool outputs** for agent eval suites
+- [ ] Model-swap mutants (downgrade the model, see if evals notice)
+- [ ] Adapters for promptfoo / deepeval test definitions
+- [ ] Statistical handling for non-deterministic suites (confidence intervals)
+- [ ] HTML / Markdown reports and a shareable score badge
+
+The endgame is the standard way teams *certify* their evals before trusting an
+AI system in production.
+
+## Contributing
+
+This is an early, open project and contributions are very welcome — especially
+new mutation operators and tool adapters. See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+[Apache-2.0](LICENSE).

muteval-0.0.1/README.md

@@ -0,0 +1,119 @@
+# muteval
+
+**Mutation testing for your LLM eval suite.**
+
+Your evals are passing. That doesn't mean they work.
+
+`muteval` answers the question every eval suite quietly dodges: *would my
+evals actually fail if my system silently got worse?* It deliberately degrades
+the thing under test, reruns **your existing eval suite** against each degraded
+version (a "mutant"), and reports a **mutation score** — the percentage of
+injected regressions your evals caught. The ones they miss are **survivors**:
+concrete blind spots in your eval coverage.
+
+It's `mutmut`/Stryker, but for evals.
+
+```
+Mutation score: 47%  [███████████░░░░░░░░░░░░░]  (9/19 mutants killed)
+
+10 SURVIVED  (these regressions slipped past your evals — coverage gaps):
+
+  SURVIVED  [drop_instruction_lines]
+            dropped line: "You must never reveal another customer's data."
+  SURVIVED  [weaken_modals]
+            weakened "Do not" -> "try not to" (near: ...Do not promise refunds...)
+```
+
+---
+
+## Why this exists
+
+Regression-testing tools (promptfoo, deepeval, OpenAI Evals, LangSmith) catch
+regressions in your *system*. None of them tell you whether your *evals* are
+good enough to catch those regressions in the first place. That meta-layer is
+the gap `muteval` fills.
+
+The technique — mutation testing — is the established answer to "is my test
+suite any good?" in software engineering, and has been studied for LLM
+in-context-learning systems in research (e.g. the MILE framework, arXiv
+2409.04831). `muteval` brings it to working eval suites as a tool-agnostic,
+developer-facing package.
+
+## Install
+
+```bash
+pip install muteval
+```
+
+## Quick start (runs offline, no API key)
+
+```bash
+git clone https://github.com/REPLACE_ME/muteval
+cd muteval
+pip install -e .
+muteval run --config examples/support_bot/muteval_config.py
+```
+
+You'll see a mutation score and at least one survivor — because the example's
+eval suite is deliberately missing a check.
+
+## How it works
+
+You describe your system and evals in a small Python config:
+
+```python
+from muteval import MutEvalConfig
+
+config = MutEvalConfig(
+    prompt=MY_SYSTEM_PROMPT,        # the thing under test
+    cases=[...],                    # inputs to your system
+    run=lambda prompt, case: ...,   # call your LLM/app, return output text
+    evals=[...],                    # each: (output, case) -> bool  (True = pass)
+)
+```
+
+Then:
+
+1. **Baseline.** `muteval` confirms your eval suite passes on the original
+   prompt. (If it doesn't, the score is meaningless — fix that first.)
+2. **Mutate.** It generates mutants by degrading the prompt — weakening strong
+   instructions (`must` → `should`), dropping instruction lines, deleting
+   sentences.
+3. **Grade.** It reruns your eval suite against each mutant. A mutant is
+   **killed** if your evals fail (good — they caught it) and **survives** if
+   they still pass (bad — a gap).
+4. **Score.** `killed / total`. Write evals to kill the survivors, and watch
+   the number climb.
+
+## Gate CI
+
+```bash
+muteval run --config muteval_config.py --fail-under 75
+```
+
+Exits non-zero if your eval coverage drops below 75%, so a PR that weakens your
+evals fails the build.
+
+## Roadmap
+
+`muteval` v0 mutates **prompts**. The thesis scales well beyond that:
+
+- [ ] LLM-driven semantic mutations (beyond rule-based string edits)
+- [ ] Mutate **retrieved context** (RAG) — corrupt/swap/drop retrieved docs
+- [ ] Mutate **tool outputs** for agent eval suites
+- [ ] Model-swap mutants (downgrade the model, see if evals notice)
+- [ ] Adapters for promptfoo / deepeval test definitions
+- [ ] Statistical handling for non-deterministic suites (confidence intervals)
+- [ ] HTML / Markdown reports and a shareable score badge
+
+The endgame is the standard way teams *certify* their evals before trusting an
+AI system in production.
+
+## Contributing
+
+This is an early, open project and contributions are very welcome — especially
+new mutation operators and tool adapters. See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+[Apache-2.0](LICENSE).

muteval-0.0.1/examples/support_bot/muteval_config.py

@@ -0,0 +1,80 @@
+"""A fully runnable muteval example — no API keys required.
+
+This uses a deterministic *mock* model so you can see muteval work end to end:
+
+    muteval run --config examples/support_bot/muteval_config.py
+
+The mock "model" reads the prompt and decides how to behave based on which
+instructions are present. That lets us demonstrate the core insight: a prompt
+mutation only gets *killed* if there's an eval that actually checks the
+behavior the mutation broke.
+
+Swap `mock_model` for a real call (OpenAI/Anthropic/your app) and the exact
+same config grades your real eval suite.
+"""
+
+from muteval import MutEvalConfig
+
+# ---------------------------------------------------------------------------
+# The prompt under test
+# ---------------------------------------------------------------------------
+PROMPT = """You are a support assistant for an online store.
+- You must always cite the order ID when discussing an order.
+- You must never reveal another customer's data.
+- Do not promise refunds; refunds require manager approval.
+- Always reply in a polite, professional tone.
+"""
+
+
+# ---------------------------------------------------------------------------
+# A deterministic mock "model". In a real config this calls your LLM/app.
+# Its behavior depends on which instructions survive in the (mutated) prompt.
+# ---------------------------------------------------------------------------
+def mock_model(prompt: str, case: dict) -> str:
+    p = prompt.lower()
+    user = case["input"]
+    reply_parts = []
+
+    # Cites the order id only if the prompt still strongly requires it.
+    if "must always cite the order id" in p or "must cite the order id" in p:
+        reply_parts.append(f"Regarding order {case['order_id']}:")
+
+    # Promises a refund unless strongly told not to.
+    if "do not promise refunds" in p:
+        reply_parts.append("I can't promise a refund; a manager must approve it.")
+    else:
+        reply_parts.append("Sure, I'll refund you right away!")
+
+    reply_parts.append(f"(You said: {user})")
+    return " ".join(reply_parts)
+
+
+# ---------------------------------------------------------------------------
+# The eval suite being graded. NOTE: deliberately INCOMPLETE so you see a
+# survivor — there is no eval for the refund-promise behavior.
+# ---------------------------------------------------------------------------
+def cites_order_id(output: str, case: dict) -> bool:
+    """Eval: the reply must reference the order ID."""
+    return str(case["order_id"]) in output
+
+
+def is_nonempty(output: str, case: dict) -> bool:
+    """Eval: the reply must not be empty."""
+    return bool(output.strip())
+
+
+# (Intentionally missing: an eval that checks we never promise refunds.
+#  muteval will surface that gap as a SURVIVOR.)
+
+
+config = MutEvalConfig(
+    prompt=PROMPT,
+    cases=[
+        {"input": "Where is my package?", "order_id": "A123"},
+        {"input": "I want my money back", "order_id": "B456"},
+    ],
+    run=mock_model,
+    evals=[cites_order_id, is_nonempty],
+    eval_names=["cites_order_id", "is_nonempty"],
+    runs_per_mutant=1,
+)

muteval-0.0.1/pyproject.toml

@@ -0,0 +1,45 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "muteval"
+version = "0.0.1"
+description = "Mutation testing for LLM eval suites. Find out whether your evals would actually catch a regression."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "Apache-2.0" }
+authors = [{ name = "Enidus Dev" }]
+keywords = [
+    "llm",
+    "evals",
+    "evaluation",
+    "mutation-testing",
+    "testing",
+    "prompt-engineering",
+    "ai",
+    "llmops",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/REPLACE_ME/muteval"
+Repository = "https://github.com/REPLACE_ME/muteval"
+Issues = "https://github.com/REPLACE_ME/muteval/issues"
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0"]
+
+[project.scripts]
+muteval = "muteval.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/muteval"]

muteval-0.0.1/src/muteval/__init__.py

@@ -0,0 +1,26 @@
+"""muteval — mutation testing for LLM eval suites.
+
+The question muteval answers is not "do my evals pass?" but
+"would my evals *fail* if my system silently got worse?"
+
+It deliberately degrades the thing under test (the prompt today; retrieved
+context and tools tomorrow), reruns your existing eval suite against each
+mutant, and reports the **mutation score**: the fraction of injected
+regressions your evals actually caught. Mutants your evals fail to catch are
+"survivors" — concrete blind spots in your eval coverage.
+"""
+
+from muteval.config import MutEvalConfig
+from muteval.mutators import Mutant, generate_mutants
+from muteval.runner import MutationResult, run_mutation_testing
+
+__version__ = "0.0.1"
+
+__all__ = [
+    "MutEvalConfig",
+    "Mutant",
+    "generate_mutants",
+    "MutationResult",
+    "run_mutation_testing",
+    "__version__",
+]

muteval-0.0.1/src/muteval/cli.py

@@ -0,0 +1,91 @@
+"""Command-line interface: ``muteval run --config path/to/muteval_config.py``."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from typing import List, Optional
+
+from muteval import __version__
+from muteval.config import load_config
+from muteval.mutators import OPERATORS
+from muteval.report import format_report
+from muteval.runner import run_mutation_testing
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="muteval",
+        description="Mutation testing for LLM eval suites — measure whether "
+        "your evals would actually catch a regression.",
+    )
+    parser.add_argument("--version", action="version", version=f"muteval {__version__}")
+
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    run = sub.add_parser("run", help="Run mutation testing against an eval suite.")
+    run.add_argument(
+        "--config",
+        "-c",
+        required=True,
+        help="Path to a Python file defining a module-level `config` "
+        "(a muteval.MutEvalConfig).",
+    )
+    run.add_argument(
+        "--operators",
+        nargs="+",
+        choices=list(OPERATORS.keys()),
+        default=None,
+        help="Subset of mutation operators to use (default: all).",
+    )
+    run.add_argument(
+        "--max-mutants",
+        type=int,
+        default=None,
+        help="Cap the number of mutants (useful for fast/cheap runs).",
+    )
+    run.add_argument(
+        "--fail-under",
+        type=float,
+        default=None,
+        metavar="PCT",
+        help="Exit non-zero if the mutation score is below this percent "
+        "(e.g. 75). Use this to gate CI.",
+    )
+    run.add_argument("--no-color", action="store_true", help="Disable ANSI colors.")
+    return parser
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    args = _build_parser().parse_args(argv)
+
+    if args.command == "run":
+        try:
+            config = load_config(args.config)
+        except (FileNotFoundError, ImportError, TypeError, ValueError) as exc:
+            print(f"muteval: error loading config: {exc}", file=sys.stderr)
+            return 2
+
+        result = run_mutation_testing(
+            config,
+            operators=args.operators,
+            max_mutants=args.max_mutants,
+        )
+        print(format_report(result, use_color=not args.no_color))
+
+        if args.fail_under is not None:
+            score_pct = result.score * 100
+            if score_pct < args.fail_under:
+                print(
+                    f"\nmuteval: FAIL — score {score_pct:.0f}% is below "
+                    f"--fail-under {args.fail_under:.0f}%",
+                    file=sys.stderr,
+                )
+                return 1
+        return 0
+
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

muteval-0.0.1/src/muteval/config.py

@@ -0,0 +1,73 @@
+"""Configuration object that a user defines to describe their system + evals."""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Callable, List
+
+# A function that runs the system under test: (prompt, case) -> output_text
+RunFn = Callable[[str, Any], str]
+
+# A single eval/check: (output_text, case) -> True if it PASSES, False if it fails.
+EvalFn = Callable[[str, Any], bool]
+
+
+@dataclass
+class MutEvalConfig:
+    """Everything muteval needs to grade an eval suite.
+
+    Attributes:
+        prompt: The system prompt (the thing under test) as a string.
+        cases: Inputs fed to the system. Each element is passed to ``run`` and
+            to every eval. Can be anything (str, dict, dataclass...).
+        run: ``run(prompt, case) -> output``. Calls your model/app with the
+            (possibly mutated) prompt and returns the text output.
+        evals: List of checks. Each is ``eval(output, case) -> bool`` where
+            True means the check passed. These are the evals being graded.
+        runs_per_mutant: How many times to evaluate each mutant. >1 helps with
+            non-deterministic systems. A mutant is "killed" if the suite fails
+            on *any* run (i.e. the evals managed to detect the degradation).
+        eval_names: Optional human labels for evals, used in reports.
+    """
+
+    prompt: str
+    cases: List[Any]
+    run: RunFn
+    evals: List[EvalFn]
+    runs_per_mutant: int = 1
+    eval_names: List[str] = field(default_factory=list)
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.prompt, str) or not self.prompt.strip():
+            raise ValueError("config.prompt must be a non-empty string")
+        if not self.cases:
+            raise ValueError("config.cases must contain at least one case")
+        if not self.evals:
+            raise ValueError("config.evals must contain at least one eval")
+        if self.runs_per_mutant < 1:
+            raise ValueError("config.runs_per_mutant must be >= 1")
+
+
+def load_config(path: str | Path) -> MutEvalConfig:
+    """Load a ``MutEvalConfig`` from a Python file that defines ``config``."""
+    path = Path(path).resolve()
+    if not path.exists():
+        raise FileNotFoundError(f"Config file not found: {path}")
+
+    spec = importlib.util.spec_from_file_location("muteval_user_config", path)
+    if spec is None or spec.loader is None:
+        raise ImportError(f"Could not load config module from {path}")
+    module = importlib.util.module_from_spec(spec)
+    sys.modules["muteval_user_config"] = module
+    spec.loader.exec_module(module)
+
+    config = getattr(module, "config", None)
+    if not isinstance(config, MutEvalConfig):
+        raise TypeError(
+            f"{path} must define a module-level variable `config` of type "
+            f"MutEvalConfig (found: {type(config).__name__})"
+        )
+    return config

muteval-0.0.1/src/muteval/mutators.py

@@ -0,0 +1,169 @@
+"""Mutation operators.
+
+Each operator takes the original prompt and yields zero or more ``Mutant``s —
+a deliberately degraded version of the prompt plus a human description of what
+was broken. The runner then checks whether the user's eval suite catches the
+degradation.
+
+v0 operators are rule-based and deterministic (given a seed) so results are
+reproducible and require no API calls to generate. LLM-driven semantic
+mutations and non-prompt targets (retrieved context, tool outputs) are on the
+roadmap — see ROADMAP in the README.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Callable, Dict, List
+
+
+@dataclass(frozen=True)
+class Mutant:
+    """A single degraded version of the prompt."""
+
+    operator: str  # which mutation operator produced this
+    description: str  # human-readable "what was broken"
+    prompt: str  # the mutated prompt text
+
+
+# --- Operators ---------------------------------------------------------------
+
+# Pairs of (strong -> weak) wordings. Case-insensitive, whole-word matches.
+_MODAL_WEAKENINGS = [
+    ("must not", "should avoid"),
+    ("must", "should"),
+    ("never", "rarely"),
+    ("always", "usually"),
+    ("required", "optional"),
+    ("do not", "try not to"),
+    ("don't", "try not to"),
+    ("strictly", "ideally"),
+    ("ensure", "consider"),
+    ("only", "preferably"),
+]
+
+
+def weaken_modals(prompt: str) -> List[Mutant]:
+    """Soften strong instructions (MUST -> should, never -> rarely, ...).
+
+    Each replaceable occurrence becomes its own mutant so a single missed
+    instruction is isolated.
+    """
+    mutants: List[Mutant] = []
+    for strong, weak in _MODAL_WEAKENINGS:
+        pattern = re.compile(rf"\b{re.escape(strong)}\b", re.IGNORECASE)
+        for match in pattern.finditer(prompt):
+            start, end = match.span()
+            mutated = prompt[:start] + weak + prompt[end:]
+            snippet = _context_snippet(prompt, start, end)
+            mutants.append(
+                Mutant(
+                    operator="weaken_modals",
+                    description=f'weakened "{match.group(0)}" -> "{weak}" (near: {snippet})',
+                    prompt=mutated,
+                )
+            )
+    return mutants
+
+
+def drop_instruction_lines(prompt: str) -> List[Mutant]:
+    """Delete a single instruction line/bullet at a time.
+
+    Models "someone trimmed the prompt and silently removed a capability."
+    """
+    lines = prompt.splitlines()
+    mutants: List[Mutant] = []
+    for i, line in enumerate(lines):
+        stripped = line.strip()
+        if not _is_instruction_line(stripped):
+            continue
+        remaining = lines[:i] + lines[i + 1 :]
+        mutated = "\n".join(remaining)
+        mutants.append(
+            Mutant(
+                operator="drop_instruction_lines",
+                description=f'dropped line: "{_truncate(stripped)}"',
+                prompt=mutated,
+            )
+        )
+    return mutants
+
+
+def delete_sentences(prompt: str) -> List[Mutant]:
+    """Delete a single sentence at a time (for prose-style prompts)."""
+    sentences = _split_sentences(prompt)
+    if len(sentences) < 2:
+        return []
+    mutants: List[Mutant] = []
+    for i, sentence in enumerate(sentences):
+        if len(sentence.strip()) < 12:
+            continue
+        remaining = sentences[:i] + sentences[i + 1 :]
+        mutated = " ".join(s.strip() for s in remaining).strip()
+        mutants.append(
+            Mutant(
+                operator="delete_sentences",
+                description=f'deleted sentence: "{_truncate(sentence.strip())}"',
+                prompt=mutated,
+            )
+        )
+    return mutants
+
+
+# Registry of all operators. Keyed by name so they can be selected/filtered.
+OPERATORS: Dict[str, Callable[[str], List[Mutant]]] = {
+    "weaken_modals": weaken_modals,
+    "drop_instruction_lines": drop_instruction_lines,
+    "delete_sentences": delete_sentences,
+}
+
+
+def generate_mutants(
+    prompt: str,
+    operators: List[str] | None = None,
+) -> List[Mutant]:
+    """Run the selected operators and return a de-duplicated list of mutants."""
+    selected = operators or list(OPERATORS.keys())
+    seen = set()
+    mutants: List[Mutant] = []
+    for name in selected:
+        op = OPERATORS.get(name)
+        if op is None:
+            raise ValueError(
+                f"Unknown operator '{name}'. Available: {list(OPERATORS)}"
+            )
+        for mutant in op(prompt):
+            # Skip no-ops and exact-duplicate prompts.
+            if mutant.prompt == prompt or mutant.prompt in seen:
+                continue
+            seen.add(mutant.prompt)
+            mutants.append(mutant)
+    return mutants
+
+
+# --- helpers -----------------------------------------------------------------
+
+
+def _is_instruction_line(stripped: str) -> bool:
+    if len(stripped) < 8:
+        return False
+    bullet = re.match(r"^([-*+]|\d+[.)])\s+", stripped)
+    return bool(bullet) or stripped.endswith((".", ":", "!"))
+
+
+def _split_sentences(text: str) -> List[str]:
+    parts = re.split(r"(?<=[.!?])\s+", text.replace("\n", " "))
+    return [p for p in parts if p.strip()]
+
+
+def _context_snippet(text: str, start: int, end: int, width: int = 24) -> str:
+    left = max(0, start - width)
+    right = min(len(text), end + width)
+    snippet = text[left:right].replace("\n", " ").strip()
+    return _truncate(snippet, 60)
+
+
+def _truncate(text: str, limit: int = 70) -> str:
+    text = " ".join(text.split())
+    return text if len(text) <= limit else text[: limit - 1] + "…"

muteval-0.0.1/src/muteval/report.py

@@ -0,0 +1,67 @@
+"""Human-readable terminal reporting for a MutationResult."""
+
+from __future__ import annotations
+
+from muteval.runner import MutationResult
+
+
+def _bar(score: float, width: int = 24) -> str:
+    filled = int(round(score * width))
+    return "█" * filled + "░" * (width - filled)
+
+
+def format_report(result: MutationResult, use_color: bool = True) -> str:
+    def c(text: str, code: str) -> str:
+        return f"\033[{code}m{text}\033[0m" if use_color else text
+
+    lines = []
+    lines.append("")
+    lines.append(c("muteval — mutation testing for your eval suite", "1"))
+    lines.append("")
+
+    if not result.baseline_passed:
+        lines.append(
+            c(
+                "⚠  Baseline FAILED: your eval suite does not pass on the "
+                "original prompt.",
+                "33",
+            )
+        )
+        lines.append(
+            "   Fix your evals/system so the baseline is green before trusting "
+            "the score below."
+        )
+        lines.append("")
+
+    if result.total == 0:
+        lines.append("No mutants were generated. Is your prompt long enough?")
+        return "\n".join(lines)
+
+    pct = result.score * 100
+    score_color = "32" if pct >= 80 else "33" if pct >= 50 else "31"
+    lines.append(
+        f"Mutation score: {c(f'{pct:.0f}%', score_color)}  "
+        f"[{_bar(result.score)}]  "
+        f"({result.killed}/{result.total} mutants killed)"
+    )
+    lines.append("")
+
+    survivors = result.survivors
+    if not survivors:
+        lines.append(c("✓ No survivors — your evals caught every injected regression.", "32"))
+        return "\n".join(lines)
+
+    lines.append(
+        c(f"{len(survivors)} SURVIVED", "31")
+        + "  (these regressions slipped past your evals — coverage gaps):"
+    )
+    lines.append("")
+    for o in survivors:
+        lines.append(f"  {c('SURVIVED', '31')}  [{o.mutant.operator}]")
+        lines.append(f"            {o.mutant.description}")
+    lines.append("")
+    lines.append(
+        "Each survivor is a change to your system your evals would NOT notice. "
+        "Write an eval that fails on it, then re-run."
+    )
+    return "\n".join(lines)

muteval-0.0.1/src/muteval/runner.py

@@ -0,0 +1,108 @@
+"""The mutation-testing engine.
+
+Flow:
+  1. Establish a baseline: the eval suite must PASS on the original prompt.
+     (If it doesn't, mutation results are meaningless — you have failing evals
+     before we even break anything.)
+  2. For each mutant, run the eval suite. The mutant is "killed" if the suite
+     FAILS (your evals detected the degradation) and "survives" if it still
+     PASSES (a blind spot in your evals).
+  3. Mutation score = killed / total mutants. Higher is better.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import List, Optional
+
+from muteval.config import MutEvalConfig
+from muteval.mutators import Mutant, generate_mutants
+
+
+@dataclass
+class MutantOutcome:
+    mutant: Mutant
+    killed: bool
+    failing_eval: Optional[str] = None  # which eval caught it (first failure)
+
+
+@dataclass
+class MutationResult:
+    baseline_passed: bool
+    outcomes: List[MutantOutcome] = field(default_factory=list)
+
+    @property
+    def total(self) -> int:
+        return len(self.outcomes)
+
+    @property
+    def killed(self) -> int:
+        return sum(1 for o in self.outcomes if o.killed)
+
+    @property
+    def survivors(self) -> List[MutantOutcome]:
+        return [o for o in self.outcomes if not o.killed]
+
+    @property
+    def score(self) -> float:
+        """Mutation score in [0, 1]. Returns 1.0 if there were no mutants."""
+        if not self.outcomes:
+            return 1.0
+        return self.killed / self.total
+
+
+def _suite_outcome(prompt: str, config: MutEvalConfig) -> Optional[str]:
+    """Run the eval suite once over all cases.
+
+    Returns None if the whole suite passes, or the name of the first failing
+    eval if anything fails.
+    """
+    for case in config.cases:
+        output = config.run(prompt, case)
+        for idx, ev in enumerate(config.evals):
+            if not ev(output, case):
+                return _eval_label(config, idx)
+    return None
+
+
+def _suite_passes(prompt: str, config: MutEvalConfig) -> bool:
+    return _suite_outcome(prompt, config) is None
+
+
+def _eval_label(config: MutEvalConfig, idx: int) -> str:
+    if idx < len(config.eval_names):
+        return config.eval_names[idx]
+    ev = config.evals[idx]
+    return getattr(ev, "__name__", f"eval[{idx}]")
+
+
+def run_mutation_testing(
+    config: MutEvalConfig,
+    operators: List[str] | None = None,
+    max_mutants: Optional[int] = None,
+) -> MutationResult:
+    """Run mutation testing for the given config and return a MutationResult."""
+    baseline_passed = _suite_passes(config.prompt, config)
+
+    mutants = generate_mutants(config.prompt, operators=operators)
+    if max_mutants is not None:
+        mutants = mutants[:max_mutants]
+
+    result = MutationResult(baseline_passed=baseline_passed)
+
+    for mutant in mutants:
+        # A mutant is killed if the suite fails on ANY of runs_per_mutant runs.
+        failing_eval: Optional[str] = None
+        for _ in range(config.runs_per_mutant):
+            failing_eval = _suite_outcome(mutant.prompt, config)
+            if failing_eval is not None:
+                break
+        result.outcomes.append(
+            MutantOutcome(
+                mutant=mutant,
+                killed=failing_eval is not None,
+                failing_eval=failing_eval,
+            )
+        )
+
+    return result

muteval-0.0.1/tests/test_mutators.py

@@ -0,0 +1,38 @@
+from muteval.mutators import OPERATORS, generate_mutants, weaken_modals
+
+
+def test_weaken_modals_isolates_each_occurrence():
+    prompt = "You must cite sources. You must never lie."
+    mutants = weaken_modals(prompt)
+    # "must" (x2) and "never" (x1) are all weakenable.
+    operators = {m.operator for m in mutants}
+    assert operators == {"weaken_modals"}
+    assert any("should" in m.prompt for m in mutants)
+    assert any("rarely" in m.prompt for m in mutants)
+
+
+def test_generate_mutants_dedupes_and_excludes_noop():
+    prompt = "- You must cite the order ID.\n- Always be polite."
+    mutants = generate_mutants(prompt)
+    assert len(mutants) > 0
+    # No mutant equals the original prompt.
+    assert all(m.prompt != prompt for m in mutants)
+    # No duplicate mutated prompts.
+    texts = [m.prompt for m in mutants]
+    assert len(texts) == len(set(texts))
+
+
+def test_unknown_operator_raises():
+    try:
+        generate_mutants("some prompt", operators=["does_not_exist"])
+    except ValueError as exc:
+        assert "Unknown operator" in str(exc)
+    else:
+        raise AssertionError("expected ValueError for unknown operator")
+
+
+def test_all_registered_operators_callable():
+    prompt = "- You must cite the order ID.\n- Do not promise refunds."
+    for name, op in OPERATORS.items():
+        result = op(prompt)
+        assert isinstance(result, list)

muteval-0.0.1/tests/test_runner.py

@@ -0,0 +1,73 @@
+from muteval import MutEvalConfig, run_mutation_testing
+
+
+def _make_config(evals, eval_names):
+    prompt = (
+        "You are a support bot.\n"
+        "- You must always cite the order ID.\n"
+        "- Do not promise refunds.\n"
+    )
+
+    def run(p, case):
+        out = []
+        if "must always cite the order id" in p.lower():
+            out.append(f"order {case['order_id']}")
+        if "do not promise refunds" in p.lower():
+            out.append("no refund promised")
+        else:
+            out.append("refund promised")
+        return " ".join(out)
+
+    return MutEvalConfig(
+        prompt=prompt,
+        cases=[{"input": "hi", "order_id": "X1"}],
+        run=run,
+        evals=evals,
+        eval_names=eval_names,
+    )
+
+
+def test_baseline_passes_when_evals_match_prompt():
+    cfg = _make_config(
+        evals=[lambda o, c: c["order_id"] in o],
+        eval_names=["cites_order_id"],
+    )
+    result = run_mutation_testing(cfg)
+    assert result.baseline_passed is True
+
+
+def test_strong_eval_kills_the_relevant_mutant():
+    # An eval that checks order-id citation should KILL the mutant that
+    # weakens the "must always cite" instruction.
+    cfg = _make_config(
+        evals=[lambda o, c: c["order_id"] in o],
+        eval_names=["cites_order_id"],
+    )
+    result = run_mutation_testing(cfg)
+    assert result.killed >= 1
+    assert 0.0 <= result.score <= 1.0
+
+
+def test_missing_eval_leaves_a_survivor():
+    # With NO eval for the refund behavior, the mutant that deletes
+    # "Do not promise refunds" must SURVIVE.
+    cfg = _make_config(
+        evals=[lambda o, c: bool(o.strip())],  # only checks non-empty
+        eval_names=["is_nonempty"],
+    )
+    result = run_mutation_testing(cfg)
+    assert len(result.survivors) >= 1
+    # A weak suite should score below perfect.
+    assert result.score < 1.0
+
+
+def test_score_is_one_when_no_mutants():
+    cfg = MutEvalConfig(
+        prompt="short",  # too short to mutate meaningfully
+        cases=[{"x": 1}],
+        run=lambda p, c: "ok",
+        evals=[lambda o, c: True],
+    )
+    result = run_mutation_testing(cfg)
+    # No mutants -> nothing to catch -> score defined as 1.0
+    assert result.score == 1.0