llm-mutation 0.1.0__tar.gz

llm_mutation-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 BuildWorld
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_mutation-0.1.0/PKG-INFO

@@ -0,0 +1,134 @@
+Metadata-Version: 2.4
+Name: llm-mutation
+Version: 0.1.0
+Summary: Mutation testing for LLM prompts. Find the gaps in your eval suite before production does.
+Project-URL: Homepage, https://github.com/Rowusuduah/llm-mutation
+Project-URL: Repository, https://github.com/Rowusuduah/llm-mutation
+Project-URL: Issues, https://github.com/Rowusuduah/llm-mutation/issues
+Author-email: Richmond Owusu Duah <Rowusuduah@users.noreply.github.com>
+License: MIT
+License-File: LICENSE
+Keywords: ai,anthropic,ci-cd,eval,evaluation,llm,llmops,mutation-testing,prompt-testing,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: black; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# llm-mutation
+
+**Mutation testing for LLM prompts. Find the gaps in your eval suite before production does.**
+
+```bash
+pip install llm-mutation
+mutate run --prompt prompts/customer_service.txt --eval evals/test_cs.py
+```
+
+## The Problem
+
+You have an eval suite. It passes. You ship. Production breaks.
+
+Your eval suite tested 50 specific cases you wrote. It was never tested itself. **llm-mutation tests whether your eval suite would notice if a key constraint was removed, a clause was dropped, or a scope was expanded.**
+
+## Quickstart
+
+```python
+from llm_mutation import MutationEngine, MutantRunner, MutationReport
+
+# 1. Generate semantic mutations of your prompt
+engine = MutationEngine()
+mutations = engine.generate("prompts/customer_service.txt")
+
+# 2. Run your eval suite against each mutant
+def my_eval_fn(prompt: str, test_cases: list) -> float:
+    # your existing eval logic — returns 0.0-1.0
+    ...
+
+runner = MutantRunner(eval_fn=my_eval_fn, test_cases=my_test_cases)
+results = runner.run(mutations)
+
+# 3. See your gaps
+report = MutationReport.from_results(results, prompt, original_score=0.91)
+print(report.summary())
+# MUTATION SCORE: 71% (5/7 mutations killed)
+# SURVIVING MUTATIONS:
+#   ✗ DropClause — "Direct pricing questions to sales@acmecorp.com." removed
+#     → ADD TEST CASE: "User asks 'What does the enterprise plan cost?'"
+```
+
+## Six Deterministic Mutation Operators
+
+| Operator | What it does |
+|----------|--------------|
+| `NegateConstraint` | Removes a prohibitive clause ("Never X") |
+| `DropClause` | Removes a requirement ("Always X", "You must X") |
+| `ScopeExpand` | Widens a scope restriction ("software only" → "products and services") |
+| `ScopeNarrow` | Narrows a permission ("any topic" → "general topics only") |
+| `ConditionInvert` | Removes a conditional behavior ("if A, then B") |
+| `PhraseSwap` | Swaps a style phrase ("concise" ↔ "comprehensive") |
+
+No LLM required for mutation generation — all operators are deterministic text transforms.
+
+## Mutation Score
+
+| Score | Verdict | Meaning |
+|-------|---------|---------|
+| >= 90% | STRONG | Eval suite is comprehensive |
+| 80-89% | ADEQUATE | Good for CI gate |
+| 70-79% | MARGINAL | Meaningful gaps |
+| 60-69% | WEAK | Significant gaps |
+| < 60% | DANGEROUS | Not fit for purpose |
+
+**Recommended minimum for production CI gate: 80%**
+
+## CLI
+
+```bash
+# Run mutation test
+mutate run --prompt prompts/cs.txt --eval evals/test_cs.py --output report.json
+
+# Generate report
+mutate report --input report.json --format markdown
+
+# CI gate (exit 1 if score < 80%)
+mutate ci --input report.json --min-score 0.80
+
+# Calibrate your eval suite
+mutate calibrate --prompt prompts/cs.txt --eval evals/test_cs.py
+```
+
+## GitHub Action
+
+```yaml
+- run: pip install llm-mutation
+- name: Run mutation tests
+  run: |
+    mutate run --prompt prompts/cs.txt --eval evals/test_cs.py --output report.json
+    mutate ci --input report.json --min-score 0.80
+```
+
+## Pattern Foundation
+
+Built on **PAT-045 — Judges 6:36-40 (The Gideon Fleece Inversion Pattern)**.
+
+Gideon designed a two-condition invertible test: fleece wet/ground dry, then fleece dry/ground wet. He wasn't testing God's power — he was testing whether his testing mechanism could discriminate signal from coincidence.
+
+**llm-mutation is the bowlful of water. Your mutation score is your measurement.**
+
+Supporting: PAT-046 (Acts 17:11 — Berean Null Test) → `mutate calibrate`
+Supporting: PAT-047 (Numbers 13:25-33 — Twelve Spies Divergence) → `mutate verify-judge`
+
+## License
+
+MIT

llm_mutation-0.1.0/README.md

@@ -0,0 +1,106 @@
+# llm-mutation
+
+**Mutation testing for LLM prompts. Find the gaps in your eval suite before production does.**
+
+```bash
+pip install llm-mutation
+mutate run --prompt prompts/customer_service.txt --eval evals/test_cs.py
+```
+
+## The Problem
+
+You have an eval suite. It passes. You ship. Production breaks.
+
+Your eval suite tested 50 specific cases you wrote. It was never tested itself. **llm-mutation tests whether your eval suite would notice if a key constraint was removed, a clause was dropped, or a scope was expanded.**
+
+## Quickstart
+
+```python
+from llm_mutation import MutationEngine, MutantRunner, MutationReport
+
+# 1. Generate semantic mutations of your prompt
+engine = MutationEngine()
+mutations = engine.generate("prompts/customer_service.txt")
+
+# 2. Run your eval suite against each mutant
+def my_eval_fn(prompt: str, test_cases: list) -> float:
+    # your existing eval logic — returns 0.0-1.0
+    ...
+
+runner = MutantRunner(eval_fn=my_eval_fn, test_cases=my_test_cases)
+results = runner.run(mutations)
+
+# 3. See your gaps
+report = MutationReport.from_results(results, prompt, original_score=0.91)
+print(report.summary())
+# MUTATION SCORE: 71% (5/7 mutations killed)
+# SURVIVING MUTATIONS:
+#   ✗ DropClause — "Direct pricing questions to sales@acmecorp.com." removed
+#     → ADD TEST CASE: "User asks 'What does the enterprise plan cost?'"
+```
+
+## Six Deterministic Mutation Operators
+
+| Operator | What it does |
+|----------|--------------|
+| `NegateConstraint` | Removes a prohibitive clause ("Never X") |
+| `DropClause` | Removes a requirement ("Always X", "You must X") |
+| `ScopeExpand` | Widens a scope restriction ("software only" → "products and services") |
+| `ScopeNarrow` | Narrows a permission ("any topic" → "general topics only") |
+| `ConditionInvert` | Removes a conditional behavior ("if A, then B") |
+| `PhraseSwap` | Swaps a style phrase ("concise" ↔ "comprehensive") |
+
+No LLM required for mutation generation — all operators are deterministic text transforms.
+
+## Mutation Score
+
+| Score | Verdict | Meaning |
+|-------|---------|---------|
+| >= 90% | STRONG | Eval suite is comprehensive |
+| 80-89% | ADEQUATE | Good for CI gate |
+| 70-79% | MARGINAL | Meaningful gaps |
+| 60-69% | WEAK | Significant gaps |
+| < 60% | DANGEROUS | Not fit for purpose |
+
+**Recommended minimum for production CI gate: 80%**
+
+## CLI
+
+```bash
+# Run mutation test
+mutate run --prompt prompts/cs.txt --eval evals/test_cs.py --output report.json
+
+# Generate report
+mutate report --input report.json --format markdown
+
+# CI gate (exit 1 if score < 80%)
+mutate ci --input report.json --min-score 0.80
+
+# Calibrate your eval suite
+mutate calibrate --prompt prompts/cs.txt --eval evals/test_cs.py
+```
+
+## GitHub Action
+
+```yaml
+- run: pip install llm-mutation
+- name: Run mutation tests
+  run: |
+    mutate run --prompt prompts/cs.txt --eval evals/test_cs.py --output report.json
+    mutate ci --input report.json --min-score 0.80
+```
+
+## Pattern Foundation
+
+Built on **PAT-045 — Judges 6:36-40 (The Gideon Fleece Inversion Pattern)**.
+
+Gideon designed a two-condition invertible test: fleece wet/ground dry, then fleece dry/ground wet. He wasn't testing God's power — he was testing whether his testing mechanism could discriminate signal from coincidence.
+
+**llm-mutation is the bowlful of water. Your mutation score is your measurement.**
+
+Supporting: PAT-046 (Acts 17:11 — Berean Null Test) → `mutate calibrate`
+Supporting: PAT-047 (Numbers 13:25-33 — Twelve Spies Divergence) → `mutate verify-judge`
+
+## License
+
+MIT

llm_mutation-0.1.0/llm_mutation/__init__.py

@@ -0,0 +1,48 @@
+"""
+llm-mutation — Mutation testing for LLM prompts.
+
+Find the gaps in your eval suite before production does.
+
+Quickstart:
+    pip install llm-mutation
+    mutate run --prompt prompts/cs.txt --eval evals/test_cs.py
+
+Pattern: PAT-045 (Judges 6:36-40 — The Gideon Fleece Inversion Pattern)
+"""
+from __future__ import annotations
+
+from ._models import (
+    Mutation,
+    MutantResult,
+    MutantVerdict,
+    MutationReport,
+    MutationOperator,
+    MutationScoreVerdict,
+)
+from ._engine import MutationEngine
+from ._runner import MutantRunner
+from ._store import MutationStore
+from ._calibrate import run_calibration, CalibrationReport
+
+__version__ = "0.1.0"
+__all__ = [
+    # Models
+    "Mutation",
+    "MutantResult",
+    "MutantVerdict",
+    "MutationReport",
+    "MutationOperator",
+    "MutationScoreVerdict",
+    # Core
+    "MutationEngine",
+    "MutantRunner",
+    "MutationStore",
+    # Calibration
+    "run_calibration",
+    "CalibrationReport",
+]
+
+
+def _cli_main() -> None:
+    from ._cli import main
+    main()

llm_mutation-0.1.0/llm_mutation/_calibrate.py

@@ -0,0 +1,210 @@
+"""
+Calibration module — Berean Null Test (PAT-046).
+
+Before trusting your mutation score, verify that your eval suite can
+actually detect known-severity mutations. If your eval suite can't catch
+a HIGH-severity mutation (complete system prompt removal), the mutation
+score is meaningless.
+
+The `mutate calibrate` command runs 5 known-severity mutations against
+the user's eval suite and reports a calibration score.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Callable, Literal
+
+from ._models import Mutation
+
+
+CalibrationSeverity = Literal["HIGH", "MEDIUM", "LOW"]
+
+
+@dataclass
+class CalibrationCase:
+    severity: CalibrationSeverity
+    description: str
+    build_mutant: Callable[[str], str]   # (original_text) -> mutated_text
+    recommendation: str
+
+
+def _remove_prohibitions(text: str) -> str:
+    import re
+    result = re.sub(
+        r"(?m)^[ \t]*(?:[Nn]ever|[Dd]o not|[Dd]on't|[Aa]void)\s+.+[.!]?\s*$",
+        "",
+        text,
+    )
+    return re.sub(r"\n{3,}", "\n\n", result).strip() or text
+
+
+def _remove_requirements(text: str) -> str:
+    import re
+    result = re.sub(
+        r"(?m)^[ \t]*(?:[Aa]lways|[Yy]ou must|[Yy]ou should|[Mm]ake sure|[Ee]nsure)\s+.+[.!]?\s*$",
+        "",
+        text,
+    )
+    return re.sub(r"\n{3,}", "\n\n", result).strip() or text
+
+
+def _remove_first_line(text: str) -> str:
+    lines = text.strip().split("\n")
+    if len(lines) <= 1:
+        return text
+    return "\n".join(lines[1:]).strip()
+
+
+def _remove_last_instruction(text: str) -> str:
+    import re
+    lines = text.strip().split("\n")
+    # Find last non-empty line
+    for i in range(len(lines) - 1, -1, -1):
+        if lines[i].strip():
+            new_lines = lines[:i] + lines[i + 1:]
+            result = "\n".join(new_lines).strip()
+            return result if result else text
+    return text
+
+
+# Five canonical calibration mutations (known severity) — defined after helper functions
+_CALIBRATION_CASES: list[CalibrationCase] = [
+    CalibrationCase(
+        severity="HIGH",
+        description="Complete system prompt removal (all instructions stripped)",
+        build_mutant=lambda text: "You are a helpful assistant.",
+        recommendation=(
+            "Your eval suite cannot detect complete prompt removal. "
+            "Add a test case that would fail on a blank/generic assistant."
+        ),
+    ),
+    CalibrationCase(
+        severity="HIGH",
+        description="All prohibitive clauses removed (all 'Never' / 'Do not' lines)",
+        build_mutant=_remove_prohibitions,
+        recommendation=(
+            "Your eval suite misses prohibition removal. "
+            "Add test cases that verify prohibited behaviors are actually refused."
+        ),
+    ),
+    CalibrationCase(
+        severity="MEDIUM",
+        description="All requirement clauses removed (all 'Always' / 'You must' lines)",
+        build_mutant=_remove_requirements,
+        recommendation=(
+            "Your eval suite misses requirement removal. "
+            "Add test cases verifying each required behavior appears in responses."
+        ),
+    ),
+    CalibrationCase(
+        severity="MEDIUM",
+        description="Role/persona instruction removed (first line of prompt)",
+        build_mutant=_remove_first_line,
+        recommendation=(
+            "Your eval suite misses persona loss. "
+            "Add a test case verifying the LLM adopts the specified role."
+        ),
+    ),
+    CalibrationCase(
+        severity="LOW",
+        description="Single instruction clause dropped (last instruction line)",
+        build_mutant=_remove_last_instruction,
+        recommendation=(
+            "Your eval suite misses single-clause removal. "
+            "Each important instruction should have at least one dedicated test case."
+        ),
+    ),
+]
+
+
+@dataclass
+class CalibrationResult:
+    case: CalibrationCase
+    original_score: float
+    mutant_score: float
+    delta: float
+    caught: bool   # True if delta >= delta_threshold
+
+
+@dataclass
+class CalibrationReport:
+    calibration_score: float     # fraction of calibration cases caught
+    total_cases: int
+    caught: int
+    missed: int
+    results: list[CalibrationResult]
+    warnings: list[str]
+
+    def summary(self) -> str:
+        pct = int(self.calibration_score * 100)
+        lines = [
+            f"\nCALIBRATION RESULTS:",
+            f"Tested {self.total_cases} known-severity mutations against your eval suite.",
+        ]
+        for r in self.results:
+            icon = "\u2713" if r.caught else "\u2717"
+            caught_str = "Caught" if r.caught else "Missed"
+            lines.append(
+                f"  {icon} Severity: {r.case.severity} \u2014 {r.case.description}"
+                f" \u2014 {caught_str} (score: {r.original_score:.2f} \u2192 {r.mutant_score:.2f})"
+            )
+        lines.append("")
+        lines.append(f"Calibration score: {pct}% ({self.caught}/{self.total_cases} known-severity mutations caught)")
+        for w in self.warnings:
+            lines.append(f"WARNING: {w}")
+        return "\n".join(lines)
+
+
+def run_calibration(
+    eval_fn: Callable[[str, list], float],
+    test_cases: list,
+    prompt: str,
+    delta_threshold: float = 0.15,
+    runs_per_case: int = 1,
+) -> CalibrationReport:
+    """Run calibration cases and return CalibrationReport."""
+    import statistics
+
+    def _score(text: str) -> float:
+        scores = [float(eval_fn(text, test_cases)) for _ in range(runs_per_case)]
+        return statistics.median(scores)
+
+    original_score = _score(prompt)
+    results: list[CalibrationResult] = []
+    warnings: list[str] = []
+
+    for case in _CALIBRATION_CASES:
+        try:
+            mutant_text = case.build_mutant(prompt)
+            if mutant_text == prompt:
+                # Skip if mutation didn't change anything (prompt may not have matching clauses)
+                continue
+            mutant_score = _score(mutant_text)
+            delta = original_score - mutant_score
+            caught = delta >= delta_threshold
+            results.append(
+                CalibrationResult(
+                    case=case,
+                    original_score=original_score,
+                    mutant_score=mutant_score,
+                    delta=delta,
+                    caught=caught,
+                )
+            )
+            if not caught and case.severity == "HIGH":
+                warnings.append(case.recommendation)
+        except Exception as exc:
+            warnings.append(f"Calibration case '{case.description}' failed: {exc}")
+
+    caught_count = sum(1 for r in results if r.caught)
+    total = len(results)
+    score = caught_count / total if total > 0 else 0.0
+
+    return CalibrationReport(
+        calibration_score=score,
+        total_cases=total,
+        caught=caught_count,
+        missed=total - caught_count,
+        results=results,
+        warnings=warnings,
+    )