promptlab-cli 0.1.0__tar.gz

promptlab_cli-0.1.0/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.4
+Name: promptlab-cli
+Version: 0.1.0
+Summary: Automated testing for LLM prompts. Like pytest, but for prompts.
+Project-URL: Homepage, https://github.com/vigp17/promptlab
+Project-URL: Repository, https://github.com/vigp17/promptlab
+Project-URL: Issues, https://github.com/vigp17/promptlab/issues
+Author: Vignesh Pai
+License-Expression: MIT
+Keywords: ai,claude,evaluation,llm,openai,prompts,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: httpx>=0.24.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# promptlab ⚡
+
+Automated testing for LLM prompts. Write test cases in YAML, run them against Claude or OpenAI, get pass/fail results in your terminal.
+
+**Like pytest, but for prompts.**
+
+```bash
+pip install promptlab-cli
+promptlab run tests/
+```
+
+```
+✅ summarize_article :: returns_short_summary        PASS (1.2s)
+✅ summarize_article :: mentions_key_points           PASS (1.1s)
+❌ translate_text :: preserves_tone                   FAIL (0.9s)
+   Expected: contains "formal"
+   Got: "Here is the translated text in a casual style..."
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Results: 2 passed, 1 failed, 3 total (3.2s)
+```
+
+## Why?
+
+You're building an app with Claude or GPT. Your prompt works today. Tomorrow you tweak it and something breaks. You don't notice until a user complains.
+
+**promptlab catches prompt regressions before they ship.** Define what good output looks like, run tests on every change, and know immediately if something broke.
+
+## Quickstart
+
+### Install
+
+```bash
+pip install promptlab-cli
+```
+
+### Set your API key
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+# or
+export OPENAI_API_KEY=sk-...
+```
+
+### Write a test file
+
+Create `tests/summarize.yaml`:
+
+```yaml
+prompt: |
+  Summarize this article in 2-3 sentences:
+  {{ article }}
+
+model: claude-sonnet-4-20250514
+
+tests:
+  - name: short_summary
+    vars:
+      article: |
+        The Federal Reserve held interest rates steady on Wednesday,
+        keeping the benchmark rate in the 5.25%-5.50% range. Chair
+        Jerome Powell said the committee needs more confidence that
+        inflation is moving toward the 2% target before cutting rates.
+    assert:
+      - type: max_tokens
+        value: 100
+      - type: contains
+        value: "Federal Reserve"
+      - type: contains
+        value: "interest rate"
+
+  - name: handles_empty_input
+    vars:
+      article: ""
+    assert:
+      - type: not_contains
+        value: "error"
+      - type: min_length
+        value: 10
+```
+
+### Run it
+
+```bash
+promptlab run tests/
+```
+
+## Test File Format
+
+Each `.yaml` file defines a prompt and its test cases:
+
+```yaml
+# The prompt template. Use {{ variable }} for inputs.
+prompt: |
+  You are a helpful assistant. {{ instruction }}
+
+# Which model to use
+model: claude-sonnet-4-20250514   # or gpt-4o, claude-haiku-4-5-20251001, etc.
+
+# Optional system prompt
+system: "You are a concise technical writer."
+
+# Optional model parameters
+temperature: 0
+max_tokens: 500
+
+# Test cases
+tests:
+  - name: test_name
+    vars:
+      instruction: "Explain what a CPU does in one sentence."
+    assert:
+      - type: contains
+        value: "processor"
+      - type: max_tokens
+        value: 50
+```
+
+## Assertion Types
+
+| Type | Description | Example |
+|---|---|---|
+| `contains` | Output must contain this string (case-insensitive) | `value: "machine learning"` |
+| `not_contains` | Output must NOT contain this string | `value: "I'm sorry"` |
+| `starts_with` | Output must start with this string | `value: "Sure"` |
+| `regex` | Output must match this regex pattern | `value: "\\d{4}"` |
+| `max_tokens` | Output must be at most N tokens | `value: 100` |
+| `min_length` | Output must be at least N characters | `value: 50` |
+| `max_length` | Output must be at most N characters | `value: 500` |
+| `equals` | Output must exactly equal this string | `value: "42"` |
+| `llm_judge` | Ask another LLM to evaluate the output | `value: "Is this response helpful and accurate?"` |
+
+## LLM-as-Judge
+
+The most powerful assertion type. Uses a second LLM call to evaluate output quality:
+
+```yaml
+tests:
+  - name: helpful_response
+    vars:
+      question: "How do I fix a memory leak in Python?"
+    assert:
+      - type: llm_judge
+        value: "Does this response provide specific, actionable debugging steps? Answer YES or NO."
+```
+
+## Supported Models
+
+**Anthropic (Claude):**
+- `claude-sonnet-4-20250514`
+- `claude-haiku-4-5-20251001`
+- Set `ANTHROPIC_API_KEY` environment variable
+
+**OpenAI:**
+- `gpt-4o`
+- `gpt-4o-mini`
+- Set `OPENAI_API_KEY` environment variable
+
+## CLI Commands
+
+```bash
+# Run all test files in a directory
+promptlab run tests/
+
+# Run a single test file
+promptlab run tests/summarize.yaml
+
+# Verbose output (show full LLM responses)
+promptlab run tests/ --verbose
+
+# Output results as JSON
+promptlab run tests/ --json
+
+# Dry run (show what would be tested without calling APIs)
+promptlab run tests/ --dry-run
+```
+
+## Use Cases
+
+- **Prompt regression testing** — Run tests in CI/CD to catch regressions
+- **Prompt comparison** — Test the same cases across different models
+- **Guard rails validation** — Verify your prompt rejects harmful inputs
+- **Output format checking** — Ensure structured output matches expectations
+
+## Development
+
+```bash
+git clone https://github.com/vigp17/promptlab.git
+cd promptlab
+pip install -e ".[dev]"
+pytest
+```
+
+## Roadmap
+
+- [x] YAML test definitions
+- [x] Claude and OpenAI support
+- [x] 9 assertion types including LLM-as-judge
+- [x] CLI with colored output
+- [ ] Cost tracking per test run
+- [ ] HTML report generation
+- [ ] Parallel test execution
+- [ ] GitHub Actions integration
+- [ ] Prompt versioning and diff
+- [ ] Custom scoring functions
+
+## License
+
+MIT

promptlab_cli-0.1.0/README.md

@@ -0,0 +1,209 @@
+# promptlab ⚡
+
+Automated testing for LLM prompts. Write test cases in YAML, run them against Claude or OpenAI, get pass/fail results in your terminal.
+
+**Like pytest, but for prompts.**
+
+```bash
+pip install promptlab-cli
+promptlab run tests/
+```
+
+```
+✅ summarize_article :: returns_short_summary        PASS (1.2s)
+✅ summarize_article :: mentions_key_points           PASS (1.1s)
+❌ translate_text :: preserves_tone                   FAIL (0.9s)
+   Expected: contains "formal"
+   Got: "Here is the translated text in a casual style..."
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Results: 2 passed, 1 failed, 3 total (3.2s)
+```
+
+## Why?
+
+You're building an app with Claude or GPT. Your prompt works today. Tomorrow you tweak it and something breaks. You don't notice until a user complains.
+
+**promptlab catches prompt regressions before they ship.** Define what good output looks like, run tests on every change, and know immediately if something broke.
+
+## Quickstart
+
+### Install
+
+```bash
+pip install promptlab-cli
+```
+
+### Set your API key
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+# or
+export OPENAI_API_KEY=sk-...
+```
+
+### Write a test file
+
+Create `tests/summarize.yaml`:
+
+```yaml
+prompt: |
+  Summarize this article in 2-3 sentences:
+  {{ article }}
+
+model: claude-sonnet-4-20250514
+
+tests:
+  - name: short_summary
+    vars:
+      article: |
+        The Federal Reserve held interest rates steady on Wednesday,
+        keeping the benchmark rate in the 5.25%-5.50% range. Chair
+        Jerome Powell said the committee needs more confidence that
+        inflation is moving toward the 2% target before cutting rates.
+    assert:
+      - type: max_tokens
+        value: 100
+      - type: contains
+        value: "Federal Reserve"
+      - type: contains
+        value: "interest rate"
+
+  - name: handles_empty_input
+    vars:
+      article: ""
+    assert:
+      - type: not_contains
+        value: "error"
+      - type: min_length
+        value: 10
+```
+
+### Run it
+
+```bash
+promptlab run tests/
+```
+
+## Test File Format
+
+Each `.yaml` file defines a prompt and its test cases:
+
+```yaml
+# The prompt template. Use {{ variable }} for inputs.
+prompt: |
+  You are a helpful assistant. {{ instruction }}
+
+# Which model to use
+model: claude-sonnet-4-20250514   # or gpt-4o, claude-haiku-4-5-20251001, etc.
+
+# Optional system prompt
+system: "You are a concise technical writer."
+
+# Optional model parameters
+temperature: 0
+max_tokens: 500
+
+# Test cases
+tests:
+  - name: test_name
+    vars:
+      instruction: "Explain what a CPU does in one sentence."
+    assert:
+      - type: contains
+        value: "processor"
+      - type: max_tokens
+        value: 50
+```
+
+## Assertion Types
+
+| Type | Description | Example |
+|---|---|---|
+| `contains` | Output must contain this string (case-insensitive) | `value: "machine learning"` |
+| `not_contains` | Output must NOT contain this string | `value: "I'm sorry"` |
+| `starts_with` | Output must start with this string | `value: "Sure"` |
+| `regex` | Output must match this regex pattern | `value: "\\d{4}"` |
+| `max_tokens` | Output must be at most N tokens | `value: 100` |
+| `min_length` | Output must be at least N characters | `value: 50` |
+| `max_length` | Output must be at most N characters | `value: 500` |
+| `equals` | Output must exactly equal this string | `value: "42"` |
+| `llm_judge` | Ask another LLM to evaluate the output | `value: "Is this response helpful and accurate?"` |
+
+## LLM-as-Judge
+
+The most powerful assertion type. Uses a second LLM call to evaluate output quality:
+
+```yaml
+tests:
+  - name: helpful_response
+    vars:
+      question: "How do I fix a memory leak in Python?"
+    assert:
+      - type: llm_judge
+        value: "Does this response provide specific, actionable debugging steps? Answer YES or NO."
+```
+
+## Supported Models
+
+**Anthropic (Claude):**
+- `claude-sonnet-4-20250514`
+- `claude-haiku-4-5-20251001`
+- Set `ANTHROPIC_API_KEY` environment variable
+
+**OpenAI:**
+- `gpt-4o`
+- `gpt-4o-mini`
+- Set `OPENAI_API_KEY` environment variable
+
+## CLI Commands
+
+```bash
+# Run all test files in a directory
+promptlab run tests/
+
+# Run a single test file
+promptlab run tests/summarize.yaml
+
+# Verbose output (show full LLM responses)
+promptlab run tests/ --verbose
+
+# Output results as JSON
+promptlab run tests/ --json
+
+# Dry run (show what would be tested without calling APIs)
+promptlab run tests/ --dry-run
+```
+
+## Use Cases
+
+- **Prompt regression testing** — Run tests in CI/CD to catch regressions
+- **Prompt comparison** — Test the same cases across different models
+- **Guard rails validation** — Verify your prompt rejects harmful inputs
+- **Output format checking** — Ensure structured output matches expectations
+
+## Development
+
+```bash
+git clone https://github.com/vigp17/promptlab.git
+cd promptlab
+pip install -e ".[dev]"
+pytest
+```
+
+## Roadmap
+
+- [x] YAML test definitions
+- [x] Claude and OpenAI support
+- [x] 9 assertion types including LLM-as-judge
+- [x] CLI with colored output
+- [ ] Cost tracking per test run
+- [ ] HTML report generation
+- [ ] Parallel test execution
+- [ ] GitHub Actions integration
+- [ ] Prompt versioning and diff
+- [ ] Custom scoring functions
+
+## License
+
+MIT

promptlab_cli-0.1.0/examples/summarize.yaml

@@ -0,0 +1,34 @@
+prompt: |
+  Summarize this text in 2-3 sentences:
+  {{ text }}
+
+model: claude-sonnet-4-20250514
+temperature: 0
+max_tokens: 200
+
+tests:
+  - name: short_summary
+    vars:
+      text: |
+        The Federal Reserve held interest rates steady on Wednesday,
+        keeping the benchmark rate in the 5.25%-5.50% range. Chair
+        Jerome Powell said the committee needs more confidence that
+        inflation is moving toward the 2% target before cutting rates.
+        Markets had been expecting a cut, but Powell's comments suggest
+        the Fed will wait for more data before making any changes.
+    assert:
+      - type: contains
+        value: "Federal Reserve"
+      - type: contains
+        value: "rate"
+      - type: max_tokens
+        value: 80
+
+  - name: handles_short_input
+    vars:
+      text: "The sky is blue."
+    assert:
+      - type: min_length
+        value: 10
+      - type: max_tokens
+        value: 50

promptlab_cli-0.1.0/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "promptlab-cli"
+version = "0.1.0"
+description = "Automated testing for LLM prompts. Like pytest, but for prompts."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [
+    { name = "Vignesh Pai" }
+]
+keywords = ["llm", "testing", "prompts", "claude", "openai", "evaluation", "ai"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "click>=8.0",
+    "pyyaml>=6.0",
+    "httpx>=0.24.0",
+    "rich>=13.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "ruff>=0.1.0",
+]
+
+[project.scripts]
+promptlab = "promptlab.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/vigp17/promptlab"
+Repository = "https://github.com/vigp17/promptlab"
+Issues = "https://github.com/vigp17/promptlab/issues"
+
+[tool.ruff]
+target-version = "py310"
+line-length = 100
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/promptlab"]

promptlab_cli-0.1.0/src/promptlab/__init__.py

@@ -0,0 +1,3 @@
+"""promptlab: Automated testing for LLM prompts."""
+
+__version__ = "0.1.0"

promptlab_cli-0.1.0/src/promptlab/assertions.py

@@ -0,0 +1,182 @@
+"""Assertion types for evaluating LLM outputs."""
+
+import re
+
+from promptlab.providers import call_llm
+
+
+def check_assertion(assertion: dict, output: str, model: str) -> dict:
+    """Check a single assertion against an LLM output.
+
+    Returns:
+        dict with keys: passed (bool), type, expected, got, message
+    """
+    atype = assertion["type"]
+    value = assertion.get("value")
+
+    checkers = {
+        "contains": _check_contains,
+        "not_contains": _check_not_contains,
+        "starts_with": _check_starts_with,
+        "regex": _check_regex,
+        "equals": _check_equals,
+        "max_tokens": _check_max_tokens,
+        "min_length": _check_min_length,
+        "max_length": _check_max_length,
+        "llm_judge": _check_llm_judge,
+    }
+
+    checker = checkers.get(atype)
+    if checker is None:
+        return {
+            "passed": False,
+            "type": atype,
+            "expected": value,
+            "got": None,
+            "message": f"Unknown assertion type: {atype}",
+        }
+
+    if atype == "llm_judge":
+        return checker(output, value, model)
+    return checker(output, value)
+
+
+def _check_contains(output: str, value: str) -> dict:
+    passed = value.lower() in output.lower()
+    return {
+        "passed": passed,
+        "type": "contains",
+        "expected": f'contains "{value}"',
+        "got": output[:200] + "..." if len(output) > 200 else output,
+        "message": "" if passed else f'Output does not contain "{value}"',
+    }
+
+
+def _check_not_contains(output: str, value: str) -> dict:
+    passed = value.lower() not in output.lower()
+    return {
+        "passed": passed,
+        "type": "not_contains",
+        "expected": f'does not contain "{value}"',
+        "got": output[:200] + "..." if len(output) > 200 else output,
+        "message": "" if passed else f'Output contains "{value}" (should not)',
+    }
+
+
+def _check_starts_with(output: str, value: str) -> dict:
+    passed = output.strip().lower().startswith(value.lower())
+    return {
+        "passed": passed,
+        "type": "starts_with",
+        "expected": f'starts with "{value}"',
+        "got": output[:100],
+        "message": "" if passed else f'Output does not start with "{value}"',
+    }
+
+
+def _check_regex(output: str, value: str) -> dict:
+    try:
+        passed = bool(re.search(value, output))
+    except re.error as e:
+        return {
+            "passed": False,
+            "type": "regex",
+            "expected": f"matches /{value}/",
+            "got": None,
+            "message": f"Invalid regex: {e}",
+        }
+    return {
+        "passed": passed,
+        "type": "regex",
+        "expected": f"matches /{value}/",
+        "got": output[:200] + "..." if len(output) > 200 else output,
+        "message": "" if passed else f"Output does not match regex /{value}/",
+    }
+
+
+def _check_equals(output: str, value: str) -> dict:
+    passed = output.strip() == value.strip()
+    return {
+        "passed": passed,
+        "type": "equals",
+        "expected": value,
+        "got": output.strip(),
+        "message": "" if passed else "Output does not exactly match expected value",
+    }
+
+
+def _check_max_tokens(output: str, value: int) -> dict:
+    # Rough token estimate: ~4 chars per token
+    token_estimate = len(output.split())
+    passed = token_estimate <= value
+    return {
+        "passed": passed,
+        "type": "max_tokens",
+        "expected": f"<= {value} tokens",
+        "got": f"~{token_estimate} tokens ({len(output)} chars)",
+        "message": "" if passed else f"Output has ~{token_estimate} tokens, max is {value}",
+    }
+
+
+def _check_min_length(output: str, value: int) -> dict:
+    length = len(output.strip())
+    passed = length >= value
+    return {
+        "passed": passed,
+        "type": "min_length",
+        "expected": f">= {value} chars",
+        "got": f"{length} chars",
+        "message": "" if passed else f"Output is {length} chars, minimum is {value}",
+    }
+
+
+def _check_max_length(output: str, value: int) -> dict:
+    length = len(output.strip())
+    passed = length <= value
+    return {
+        "passed": passed,
+        "type": "max_length",
+        "expected": f"<= {value} chars",
+        "got": f"{length} chars",
+        "message": "" if passed else f"Output is {length} chars, maximum is {value}",
+    }
+
+
+def _check_llm_judge(output: str, criteria: str, model: str) -> dict:
+    """Use an LLM to judge the output quality."""
+    judge_prompt = f"""You are evaluating an LLM output. Answer only YES or NO.
+
+Criteria: {criteria}
+
+Output to evaluate:
+---
+{output}
+---
+
+Does the output meet the criteria? Answer only YES or NO."""
+
+    try:
+        response = call_llm(
+            prompt=judge_prompt,
+            model=model,
+            temperature=0,
+            max_tokens=10,
+        )
+        answer = response["text"].strip().upper()
+        passed = answer.startswith("YES")
+    except Exception as e:
+        return {
+            "passed": False,
+            "type": "llm_judge",
+            "expected": criteria,
+            "got": f"Judge error: {e}",
+            "message": f"LLM judge failed: {e}",
+        }
+
+    return {
+        "passed": passed,
+        "type": "llm_judge",
+        "expected": criteria,
+        "got": f"Judge answered: {answer}",
+        "message": "" if passed else f"LLM judge said NO to: {criteria}",
+    }