llm-regtest 0.1.0__tar.gz

llm_regtest-0.1.0/PKG-INFO

@@ -0,0 +1,620 @@
+Metadata-Version: 2.4
+Name: llm-regtest
+Version: 0.1.0
+Summary: Lightweight regression testing for LLM prompts
+Author: Libin Samkutty
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.18; extra == "anthropic"
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0; extra == "yaml"
+Provides-Extra: semantic
+Requires-Dist: sentence-transformers>=2.2; extra == "semantic"
+Requires-Dist: numpy>=1.24; extra == "semantic"
+Provides-Extra: all
+Requires-Dist: openai>=1.0; extra == "all"
+Requires-Dist: anthropic>=0.18; extra == "all"
+Requires-Dist: pyyaml>=6.0; extra == "all"
+Requires-Dist: sentence-transformers>=2.2; extra == "all"
+Requires-Dist: numpy>=1.24; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+
+# llm-regtest
+
+A lightweight regression testing framework for LLM prompts. Catch semantic drift before it ships.
+
+---
+
+## What does this tool do?
+
+When you use an AI model, the exact wording of your instructions — called a **prompt** — has a big effect on the quality of responses. Small wording changes can cause unexpected regressions that are impossible to spot without running tests.
+
+This tool works like a test suite for prompts:
+
+1. You define test cases with prompts and optional inputs
+2. Run `update-baseline` — the AI's responses become your **golden dataset**
+3. Later, after changing a prompt or upgrading a model, run `llm-regtest run`
+4. The tool compares new responses to the baselines and flags anything that regressed
+
+Each comparison produces a **score from 0.0 to 1.0** using one or more methods:
+
+| Method | How it works | Best for |
+|--------|-------------|----------|
+| `exact` | Character-for-character match (0 or 1) | Classification labels, structured outputs |
+| `fuzzy` | Levenshtein edit-distance ratio | General text where minor wording drift is acceptable |
+| `semantic` | Cosine similarity of sentence embeddings (`all-MiniLM-L6-v2`) | Longer outputs where meaning matters more than wording |
+| `llm_judge` | An LLM rates quality similarity 0.0–1.0 | Conversational outputs where both fuzzy and semantic are too strict |
+
+Results are bucketed into **PASS / WARN / FAIL** based on configurable score thresholds.
+
+---
+
+## What you need before starting
+
+- **Python 3.10 or newer** ([download here](https://www.python.org/downloads/))
+- **pip** — comes with Python automatically
+- **An API key** — for OpenAI ([get one here](https://platform.openai.com/api-keys)) or Anthropic ([get one here](https://console.anthropic.com/))
+
+To check your Python version:
+
+```bash
+python --version
+```
+
+---
+
+## Installation
+
+```bash
+# OpenAI support
+pip install -e ".[openai]"
+
+# Anthropic (Claude) support
+pip install -e ".[anthropic]"
+
+# Semantic similarity scoring (sentence-transformers)
+pip install -e ".[semantic]"
+
+# Everything
+pip install -e ".[all]"
+```
+
+> The `[semantic]` extra installs `sentence-transformers` and `numpy`. The first time you use semantic scoring, the `all-MiniLM-L6-v2` model (~80 MB) is downloaded automatically.
+
+---
+
+## Quickstart
+
+### 1. Initialise the project
+
+```bash
+llm-regtest init
+```
+
+Creates `.promptregtest/config.json` and a `prompt_cases.json` file.
+
+### 2. Set your API key
+
+```bash
+# Mac / Linux
+export OPENAI_API_KEY="sk-..."
+
+# Windows PowerShell
+$env:OPENAI_API_KEY="sk-..."
+```
+
+### 3. Define test cases
+
+Edit `prompt_cases.json`:
+
+```json
+[
+  {
+    "id": "summarize-article",
+    "prompt": "Summarize the following text in one sentence:",
+    "input": "Scientists discovered that short 10-minute walks after meals reduce blood sugar spikes.",
+    "baseline_output": "",
+    "tags": ["summarization"]
+  },
+  {
+    "id": "sentiment-label",
+    "prompt": "Classify the sentiment. Reply with exactly one word: positive, negative, or neutral.",
+    "input": "I absolutely loved the product!",
+    "baseline_output": "",
+    "tags": ["classification"]
+  }
+]
+```
+
+### 4. Generate baselines
+
+```bash
+llm-regtest update-baseline
+```
+
+This calls the AI for every case and saves the responses as baselines inside `prompt_cases.json`.
+
+### 5. Run the regression tests
+
+```bash
+llm-regtest run
+```
+
+Sample output:
+
+```
+  [PASS] summarize-article (fuzzy: 0.94, semantic: 0.97, agg=0.96) [312ms, $0.00018]
+  [WARN] email-rewrite (fuzzy: 0.71, semantic: 0.79, agg=0.77) [289ms, $0.00014]
+  [FAIL] sentiment-label (exact: 0.00, fuzzy: 0.43, agg=0.21) [198ms, $0.00009]
+
+  ------------------------------------------------
+  Results (3 cases): 1 passed, 1 warned, 1 failed
+  Cost: $0.00041  |  Latency: 266.3ms avg
+  ------------------------------------------------
+```
+
+Each result line now shows **latency** (ms) and **cost** (USD) alongside the score. The summary line aggregates total cost and average latency across the run.
+
+### 6. View a saved report
+
+```bash
+llm-regtest report
+```
+
+---
+
+## How the Regression Check Works
+
+```
+prompt_cases.json            .promptregtest/baselines/
+┌──────────────────┐         ┌───────────────────────────┐
+│  id: "summarize" │         │  summarize.txt             │
+│  prompt: "..."   │──run──▶ │  "Short 10-min walks..."   │◀── baseline
+│  input:  "..."   │         └───────────────────────────┘
+└──────────────────┘                      │
+                                          │ compare
+         New run output ──────────────────┘
+         "Brief post-meal walks..."
+                  │
+                  ▼
+         scorer.semantic_similarity(new, baseline)
+              = cosine(embed(new), embed(baseline))
+              = 0.91  →  PASS (threshold: 0.85)
+```
+
+On each run, the tool:
+
+1. Loads every case from `prompt_cases.json`
+2. Sends the prompt (+ optional input) to the configured model
+3. Measures **latency** (wall-clock ms) and **token counts** for that call
+4. Computes **USD cost** from the provider's pricing table
+5. Runs each configured scorer against the stored baseline
+6. Computes a **weighted aggregate score**
+7. Compares it to the `pass` / `warn` thresholds
+8. Saves a JSON report to `.promptregtest/reports/`
+
+---
+
+## Cost and Latency Tracking
+
+Every run automatically captures:
+
+| Metric | Where it appears |
+|--------|-----------------|
+| Per-request latency (ms) | Beside each result: `[312ms, $0.00018]` |
+| Per-request USD cost | Beside each result: `[312ms, $0.00018]` |
+| Total run cost | Summary line: `Cost: $0.00041` |
+| Average latency | Summary line: `Latency: 266.3ms avg` |
+| Input / output token counts | Saved in the JSON report |
+
+Cost is calculated from built-in pricing tables for common OpenAI and Anthropic models. Unknown models report `$0.00000` (no crash). The JSON report stores `input_tokens`, `output_tokens`, `cost_usd`, and `latency_ms` per case for later analysis.
+
+---
+
+## Semantic Similarity Scoring
+
+The `semantic` scorer encodes both the new output and the baseline as sentence embeddings, then computes their **cosine similarity**. Two responses that express the same idea in different words score close to 1.0; responses with entirely different meaning score near 0.0.
+
+**Install:**
+
+```bash
+pip install -e ".[semantic]"
+```
+
+**Configure:**
+
+```json
+{
+  "scoring": {
+    "methods": ["fuzzy", "semantic"],
+    "weights": { "fuzzy": 0.3, "semantic": 0.7 },
+    "thresholds": { "pass": 0.85, "warn": 0.65 }
+  }
+}
+```
+
+The `all-MiniLM-L6-v2` model is downloaded once and cached in memory across cases in the same run. If `sentence-transformers` is not installed, the scorer is simply not registered — no crash on import. It will only fail if you explicitly request `"semantic"` in your config without the package installed.
+
+---
+
+## Comparing Two Versions of a Prompt
+
+The most common workflow: lock in version A, change the prompt, compare.
+
+**1. Generate the baseline (version A)**
+
+```bash
+llm-regtest update-baseline
+```
+
+**2. Modify your prompt in `prompt_cases.json`**
+
+```json
+"prompt": "Write a one-sentence summary focusing on the key finding:"
+```
+
+**3. Run the comparison**
+
+```bash
+llm-regtest run --verbose
+```
+
+`--verbose` also prints a line-by-line diff between the baseline and the new output for each case.
+
+**4. Compare two saved reports side-by-side**
+
+```bash
+llm-regtest compare \
+  --report-a .promptregtest/reports/report_before.json \
+  --report-b .promptregtest/reports/report_after.json
+```
+
+Output:
+
+```
+  Case                score-a  score-b    delta  change
+  ------------------------------------------------------
+  email-reply           0.73     0.95    +0.22  IMPROVED
+  summarize-article     0.91     0.84    -0.07  REGRESSED
+  sentiment-label       1.00     1.00    +0.00  unchanged
+
+  1 improved, 1 regressed, 1 unchanged
+```
+
+---
+
+## End-to-End OpenAI Demo
+
+A complete demo script is included that shows the full workflow with real OpenAI API calls — including baseline generation, clean regression run, drift simulation, and A/B comparison.
+
+**Prerequisites:**
+
+```bash
+pip install "llm-regtest[openai,semantic]"
+export OPENAI_API_KEY="sk-..."
+```
+
+**Run:**
+
+```bash
+python examples/demo_openai.py
+```
+
+The script runs four steps automatically:
+
+| Step | What happens |
+|------|-------------|
+| 1/4 Generate baselines | Calls `gpt-4o-mini` for all 8 cases and saves responses |
+| 2/4 Regression run | Reruns all cases — scores should be near 1.0 |
+| 3/4 Simulate drift | Rewrites two prompt wordings to mimic a real prompt change |
+| 4/4 Detect regression | Reruns with drifted prompts — expect WARN/FAIL on changed cases |
+| Bonus A/B compare | Side-by-side delta table between the two runs |
+
+The 8 demo cases cover summarization, tone rewriting, sentiment classification, and factual Q&A — a representative spread for testing how different task types respond to prompt drift.
+
+---
+
+## Configuration Reference
+
+Default config lives at `.promptregtest/config.json`:
+
+```json
+{
+  "model": {
+    "provider": "openai",
+    "model_name": "gpt-4o-mini",
+    "temperature": 0.0,
+    "max_tokens": 1024,
+    "system_prompt": ""
+  },
+  "scoring": {
+    "methods": ["fuzzy", "semantic"],
+    "weights": { "fuzzy": 0.3, "semantic": 0.7 },
+    "thresholds": {
+      "pass": 0.85,
+      "warn": 0.65
+    },
+    "llm_judge_model": null
+  },
+  "prompt_cases_path": "prompt_cases.json",
+  "reports_dir": ".promptregtest/reports",
+  "baselines_dir": ".promptregtest/baselines",
+  "concurrency": 1
+}
+```
+
+| Setting | What it does |
+|---------|-------------|
+| `provider` | `"openai"`, `"anthropic"`, or `"stub"` (no API key needed, for testing) |
+| `model_name` | Model ID, e.g. `"gpt-4o-mini"`, `"claude-sonnet-4-6"` |
+| `temperature` | Set to `0.0` for deterministic, repeatable outputs — strongly recommended for testing |
+| `max_tokens` | Maximum response length |
+| `system_prompt` | Global system-level instruction sent with every case |
+| `methods` | Scorers to use: any combination of `"exact"`, `"fuzzy"`, `"semantic"`, `"llm_judge"` |
+| `weights` | Per-method weights for the aggregate. Omit for equal weighting |
+| `thresholds.pass` | Score at or above this → PASS (default: `0.8`) |
+| `thresholds.warn` | Score at or above this → WARN (default: `0.5`); below → FAIL |
+| `llm_judge_model` | Model config for the LLM-as-judge scorer (same shape as `model`) |
+| `concurrency` | Cases to run in parallel (default: `1`) |
+
+---
+
+## Test Case Fields
+
+```json
+{
+  "id": "my-test",
+  "prompt": "Summarize in one sentence:",
+  "prompt_file": "prompts/summarize.md",
+  "system_prompt": "You are a concise assistant.",
+  "system_prompt_file": "prompts/system/concise.md",
+  "input": "Text to summarize goes here.",
+  "inputs": ["Input A", "Input B", "Input C"],
+  "inputs_file": "fixtures/reviews.json",
+  "variables": { "name": "Alice", "role": "engineer" },
+  "baseline_output": "",
+  "tags": ["smoke", "summarization"]
+}
+```
+
+| Field | Required | What it does |
+|-------|----------|--------------|
+| `id` | Yes | Unique identifier (no spaces) |
+| `prompt` | Yes* | The instruction text |
+| `prompt_file` | Yes* | Path to a `.txt` or `.md` file containing the prompt |
+| `system_prompt` | No | Per-case system prompt (overrides global config) |
+| `system_prompt_file` | No | Path to a file containing the system prompt |
+| `input` | No | Extra text appended to the prompt |
+| `inputs` | No | List of inputs — generates `id[0]`, `id[1]`, ... sub-cases |
+| `inputs_file` | No | JSON file with a list of input strings |
+| `variables` | No | Values for `{placeholder}` templates in the prompt |
+| `baseline_output` | No | Auto-filled by `update-baseline` — leave blank |
+| `tags` | No | Labels for filtering with `--tag` |
+
+*`prompt` or `prompt_file` is required, not both.
+
+---
+
+## CLI Reference
+
+| Command | What it does |
+|---------|-------------|
+| `llm-regtest init` | Create the project folder structure |
+| `llm-regtest init --ci` | Also create a GitHub Actions workflow file |
+| `llm-regtest update-baseline` | Run prompts and save responses as new baselines |
+| `llm-regtest run` | Run prompts and compare to existing baselines |
+| `llm-regtest report` | Display the most recent saved report |
+| `llm-regtest compare` | Compare two saved reports side-by-side |
+
+**Flags for `run` and `update-baseline`:**
+
+| Flag | What it does |
+|------|-------------|
+| `--config PATH` | Use a non-default config file |
+| `--case ID` | Run only this case ID (repeatable) |
+| `--tag TAG` | Run only cases with this tag (repeatable, OR logic) |
+| `--concurrency N` | Run N cases in parallel |
+
+**Flags for `run` only:**
+
+| Flag | What it does |
+|------|-------------|
+| `--verbose` / `-v` | Print a unified diff for each case |
+| `--format console` | Default coloured output |
+| `--format github` | GitHub Actions `::error::` / `::warning::` annotations |
+
+---
+
+## Advanced Features
+
+### Semantic similarity
+
+```json
+"scoring": {
+  "methods": ["fuzzy", "semantic"],
+  "weights": { "fuzzy": 0.3, "semantic": 0.7 }
+}
+```
+
+Requires `pip install -e ".[semantic]"`. Uses `all-MiniLM-L6-v2`.
+
+### LLM-as-judge
+
+```json
+"scoring": {
+  "methods": ["fuzzy", "llm_judge"],
+  "weights": { "fuzzy": 0.4, "llm_judge": 0.6 },
+  "llm_judge_model": {
+    "provider": "openai",
+    "model_name": "gpt-4o-mini"
+  }
+}
+```
+
+### Parallel execution
+
+```bash
+llm-regtest run --concurrency 10
+```
+
+### Tag filtering
+
+```bash
+llm-regtest run --tag smoke           # fast per-PR smoke suite
+llm-regtest run --tag smoke --tag critical  # OR logic
+llm-regtest update-baseline --tag customer-facing
+```
+
+### Parameterized inputs
+
+```json
+{
+  "id": "classify-sentiment",
+  "prompt": "Classify as positive, negative, or neutral:",
+  "inputs": [
+    "Love it!",
+    "Broke after one day.",
+    "It's fine."
+  ]
+}
+```
+
+Generates sub-cases `classify-sentiment[0]`, `classify-sentiment[1]`, `classify-sentiment[2]`.
+
+### Prompt files
+
+```json
+{
+  "id": "legal-analysis",
+  "prompt_file": "prompts/analyze_contract.md",
+  "system_prompt_file": "prompts/system/legal_analyst.md",
+  "input": "..."
+}
+```
+
+Paths are relative to the directory containing `prompt_cases.json`. Prompt files appear as clean diffs in pull requests.
+
+### Custom scorers
+
+```python
+from llm_regtest.scorer import register_scorer
+
+def keyword_overlap(output: str, baseline: str) -> float:
+    out_words = set(output.lower().split())
+    base_words = set(baseline.lower().split())
+    if not base_words:
+        return 1.0
+    return len(out_words & base_words) / len(base_words)
+
+register_scorer("keyword_overlap", keyword_overlap)
+```
+
+Then add `"keyword_overlap"` to `methods` in your config.
+
+---
+
+## CI / GitHub Actions
+
+### Generate a workflow file
+
+```bash
+llm-regtest init --ci
+```
+
+Creates `.github/workflows/prompt-regression.yml` — runs on every PR that touches prompt files.
+
+### Annotated PR output
+
+```bash
+llm-regtest run --format github
+```
+
+Emits `::error::` and `::warning::` lines that GitHub renders as inline annotations on the PR diff.
+
+### Example workflow
+
+```yaml
+name: Prompt Regression Tests
+on:
+  pull_request:
+    paths:
+      - 'prompts/**'
+      - 'prompt_cases.json'
+jobs:
+  regression:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - run: pip install -e ".[openai,semantic]"
+      - run: llm-regtest run --format github
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+```
+
+Add `OPENAI_API_KEY` as a repository secret under **Settings → Secrets → Actions**.
+
+---
+
+## Using Claude (Anthropic) as the Model
+
+```json
+{
+  "model": {
+    "provider": "anthropic",
+    "model_name": "claude-haiku-4-5-20251001",
+    "temperature": 0.0,
+    "max_tokens": 1024
+  }
+}
+```
+
+```bash
+pip install -e ".[anthropic]"
+export ANTHROPIC_API_KEY="sk-ant-..."
+```
+
+Available model names: `claude-haiku-4-5-20251001`, `claude-sonnet-4-6`, `claude-opus-4-6`.
+
+---
+
+## Understanding the Results
+
+| Status | Score range | Meaning |
+|--------|------------|---------|
+| **PASS** | ≥ `thresholds.pass` | Response is very similar to baseline |
+| **WARN** | ≥ `thresholds.warn` | Response has drifted noticeably — worth reviewing |
+| **FAIL** | < `thresholds.warn` | Significant regression detected |
+| **SKIP** | — | No baseline exists for this case yet |
+
+Default thresholds: `pass = 0.8`, `warn = 0.5`. Adjust in `config.json` under `scoring.thresholds`.
+
+---
+
+## Troubleshooting
+
+**"No module named llm_regtest"**
+Run `pip install -e .` from the project root directory.
+
+**"OPENAI_API_KEY not set" or authentication errors**
+Set the environment variable in the same terminal window you're running the tool from.
+
+**All tests show "skip"**
+No baselines yet. Run `llm-regtest update-baseline` first.
+
+**Semantic scorer not available**
+Install with `pip install -e ".[semantic]"`. The scorer is silently skipped if `sentence-transformers` is not installed.
+
+**Scores are lower than expected after a small change**
+Switch from `exact` to `fuzzy` or `semantic` scoring, which are more tolerant of minor wording differences.
+
+**Runs are slow with many cases**
+Use `--concurrency N` to run cases in parallel. Start with `5` and increase if your API rate limits allow.