promptbench-cli 0.1.0__tar.gz

promptbench_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shehriyar Ali Rustam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

promptbench_cli-0.1.0/PKG-INFO

@@ -0,0 +1,240 @@
+Metadata-Version: 2.4
+Name: promptbench-cli
+Version: 0.1.0
+Summary: A/B test prompts across LLM providers from your terminal
+Author-email: Shehriyar Ali Rustam <shehriyar@example.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Shehriyar-Ali-Rustam/promptbench
+Project-URL: Repository, https://github.com/Shehriyar-Ali-Rustam/promptbench
+Project-URL: Issues, https://github.com/Shehriyar-Ali-Rustam/promptbench/issues
+Keywords: llm,prompt,benchmark,openai,anthropic,gemini,cli,ai,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30; extra == "anthropic"
+Provides-Extra: google
+Requires-Dist: google-genai>=1.0; extra == "google"
+Provides-Extra: all
+Requires-Dist: openai>=1.0; extra == "all"
+Requires-Dist: anthropic>=0.30; extra == "all"
+Requires-Dist: google-genai>=1.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Dynamic: license-file
+
+# ⚡ PromptBench
+
+**A/B test prompts across LLM providers from your terminal.**
+
+[![CI](https://github.com/Shehriyar-Ali-Rustam/promptbench/actions/workflows/ci.yml/badge.svg)](https://github.com/Shehriyar-Ali-Rustam/promptbench/actions/workflows/ci.yml)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI](https://img.shields.io/pypi/v/promptbench-cli.svg)](https://pypi.org/project/promptbench-cli/)
+
+Compare how different LLMs respond to the same prompt — see latency, token usage, cost, and responses side by side in one command.
+
+```
+$ promptbench "Explain quantum computing in one sentence"
+
+  ⚡ PROMPTBENCH RESULTS
+  ────────────────────────────────────────────────────────────
+
+  Prompt: Explain quantum computing in one sentence
+
+  Model                               Latency     Tokens         Cost
+  ─────────────────────────────────── ────────── ────────── ────────────
+  claude-sonnet-4-20250514                 1.24s        142     $0.0006
+  gemini-2.0-flash                    ⚡💰 312ms         98     $0.0000
+  gpt-4o                                  845ms        127     $0.0010
+
+  ┌─ claude-sonnet-4-20250514
+  │ Quantum computing uses qubits that can exist in superpositions of
+  │ 0 and 1 simultaneously, enabling parallel computation that can
+  │ solve certain problems exponentially faster than classical computers.
+  └───────────────────────────────────────────────────────────
+
+  ┌─ gemini-2.0-flash
+  │ Quantum computing harnesses quantum mechanical phenomena like
+  │ superposition and entanglement to process information in ways
+  │ impossible for traditional computers.
+  └───────────────────────────────────────────────────────────
+
+  ┌─ gpt-4o
+  │ Quantum computing leverages the principles of quantum mechanics —
+  │ superposition and entanglement — to perform computations that
+  │ would be infeasible for classical computers.
+  └───────────────────────────────────────────────────────────
+
+  Total cost: $0.0016 · Avg latency: 799ms · 3 model(s)
+```
+
+## Why PromptBench?
+
+- **One command, multiple models** — no switching between playgrounds
+- **Side-by-side comparison** — latency, tokens, cost, and full responses
+- **Supports major providers** — OpenAI, Anthropic, Google Gemini
+- **Multiple output formats** — terminal table, JSON, CSV, Markdown
+- **Fast parallel execution** — all models run concurrently by default
+- **Batch mode** — test multiple prompts from a file
+- **Zero required dependencies** — install only the providers you need
+- **Pipe-friendly** — works with stdin for scripting workflows
+
+## Install
+
+```bash
+pip install promptbench-cli
+```
+
+Install with provider SDKs:
+
+```bash
+# Individual providers
+pip install "promptbench-cli[openai]"
+pip install "promptbench-cli[anthropic]"
+pip install "promptbench-cli[google]"
+
+# All providers at once
+pip install "promptbench-cli[all]"
+```
+
+## Setup
+
+Export your API keys:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+export ANTHROPIC_API_KEY="sk-ant-..."
+export GOOGLE_API_KEY="AI..."
+```
+
+You only need keys for the providers you want to use. PromptBench will warn you if a key is missing.
+
+## Usage
+
+### Basic comparison (default: GPT-4o, Claude Sonnet, Gemini Flash)
+
+```bash
+promptbench "Explain quantum computing in one sentence"
+```
+
+### Pick specific models
+
+```bash
+promptbench "Write a haiku about coding" -m gpt4o sonnet flash
+```
+
+### With a system prompt
+
+```bash
+promptbench "Summarize this text" -s "You are a concise technical writer" -m gpt4mini haiku
+```
+
+### Batch prompts from a file
+
+```bash
+# prompts.txt — one prompt per line
+promptbench -f prompts.txt -m gpt4o sonnet
+```
+
+### JSON output
+
+```bash
+promptbench "What is Python?" -o json
+```
+
+### CSV output
+
+```bash
+promptbench "What is Python?" -o csv > results.csv
+```
+
+### Save results to file
+
+```bash
+promptbench "Compare REST vs GraphQL" --save results.json
+```
+
+### Pipe from stdin
+
+```bash
+echo "What is the meaning of life?" | promptbench -m gpt4o sonnet
+```
+
+### List all supported models
+
+```bash
+promptbench --list-models
+```
+
+## Supported Models
+
+| Alias | Model | Provider |
+|-------|-------|----------|
+| `gpt4o` / `gpt4` | gpt-4o | OpenAI |
+| `gpt4mini` | gpt-4o-mini | OpenAI |
+| `gpt3.5` | gpt-3.5-turbo | OpenAI |
+| `sonnet` / `claude-sonnet` | claude-sonnet-4-20250514 | Anthropic |
+| `haiku` / `claude-haiku` | claude-haiku-4-5-20251001 | Anthropic |
+| `flash` / `gemini-flash` | gemini-2.0-flash | Google |
+| `gemini-pro` | gemini-1.5-pro | Google |
+
+You can also use full model names directly (e.g., `gpt-4-turbo`, `gemini-1.5-flash`).
+
+## Python Library Usage
+
+```python
+from promptbench.runner import run_bench
+from promptbench.display import display_comparison
+
+run = run_bench(
+    prompt="Explain recursion simply",
+    models=["gpt4o", "sonnet", "flash"],
+    temperature=0.5,
+)
+
+print(display_comparison(run))
+
+# Access individual results
+for result in run.results:
+    print(f"{result.model}: {result.latency_ms:.0f}ms, {result.cost_usd:.6f} USD")
+```
+
+## Configuration Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `-m, --models` | Models to test | `gpt4o sonnet flash` |
+| `-s, --system` | System prompt | None |
+| `-t, --temperature` | Sampling temperature | `0.7` |
+| `--max-tokens` | Max output tokens | `1024` |
+| `-f, --file` | Prompts file path | None |
+| `-o, --output` | Output format: table, json, csv, markdown | `table` |
+| `--full` | Show full responses (no truncation) | Off |
+| `--no-parallel` | Run models sequentially | Off |
+| `--save` | Save results to JSON file | None |
+| `--list-models` | List supported models | — |
+| `--version` | Show version | — |
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, how to add providers, and PR guidelines.
+
+## License
+
+MIT — see [LICENSE](LICENSE) for details.

promptbench_cli-0.1.0/README.md

@@ -0,0 +1,200 @@
+# ⚡ PromptBench
+
+**A/B test prompts across LLM providers from your terminal.**
+
+[![CI](https://github.com/Shehriyar-Ali-Rustam/promptbench/actions/workflows/ci.yml/badge.svg)](https://github.com/Shehriyar-Ali-Rustam/promptbench/actions/workflows/ci.yml)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI](https://img.shields.io/pypi/v/promptbench-cli.svg)](https://pypi.org/project/promptbench-cli/)
+
+Compare how different LLMs respond to the same prompt — see latency, token usage, cost, and responses side by side in one command.
+
+```
+$ promptbench "Explain quantum computing in one sentence"
+
+  ⚡ PROMPTBENCH RESULTS
+  ────────────────────────────────────────────────────────────
+
+  Prompt: Explain quantum computing in one sentence
+
+  Model                               Latency     Tokens         Cost
+  ─────────────────────────────────── ────────── ────────── ────────────
+  claude-sonnet-4-20250514                 1.24s        142     $0.0006
+  gemini-2.0-flash                    ⚡💰 312ms         98     $0.0000
+  gpt-4o                                  845ms        127     $0.0010
+
+  ┌─ claude-sonnet-4-20250514
+  │ Quantum computing uses qubits that can exist in superpositions of
+  │ 0 and 1 simultaneously, enabling parallel computation that can
+  │ solve certain problems exponentially faster than classical computers.
+  └───────────────────────────────────────────────────────────
+
+  ┌─ gemini-2.0-flash
+  │ Quantum computing harnesses quantum mechanical phenomena like
+  │ superposition and entanglement to process information in ways
+  │ impossible for traditional computers.
+  └───────────────────────────────────────────────────────────
+
+  ┌─ gpt-4o
+  │ Quantum computing leverages the principles of quantum mechanics —
+  │ superposition and entanglement — to perform computations that
+  │ would be infeasible for classical computers.
+  └───────────────────────────────────────────────────────────
+
+  Total cost: $0.0016 · Avg latency: 799ms · 3 model(s)
+```
+
+## Why PromptBench?
+
+- **One command, multiple models** — no switching between playgrounds
+- **Side-by-side comparison** — latency, tokens, cost, and full responses
+- **Supports major providers** — OpenAI, Anthropic, Google Gemini
+- **Multiple output formats** — terminal table, JSON, CSV, Markdown
+- **Fast parallel execution** — all models run concurrently by default
+- **Batch mode** — test multiple prompts from a file
+- **Zero required dependencies** — install only the providers you need
+- **Pipe-friendly** — works with stdin for scripting workflows
+
+## Install
+
+```bash
+pip install promptbench-cli
+```
+
+Install with provider SDKs:
+
+```bash
+# Individual providers
+pip install "promptbench-cli[openai]"
+pip install "promptbench-cli[anthropic]"
+pip install "promptbench-cli[google]"
+
+# All providers at once
+pip install "promptbench-cli[all]"
+```
+
+## Setup
+
+Export your API keys:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+export ANTHROPIC_API_KEY="sk-ant-..."
+export GOOGLE_API_KEY="AI..."
+```
+
+You only need keys for the providers you want to use. PromptBench will warn you if a key is missing.
+
+## Usage
+
+### Basic comparison (default: GPT-4o, Claude Sonnet, Gemini Flash)
+
+```bash
+promptbench "Explain quantum computing in one sentence"
+```
+
+### Pick specific models
+
+```bash
+promptbench "Write a haiku about coding" -m gpt4o sonnet flash
+```
+
+### With a system prompt
+
+```bash
+promptbench "Summarize this text" -s "You are a concise technical writer" -m gpt4mini haiku
+```
+
+### Batch prompts from a file
+
+```bash
+# prompts.txt — one prompt per line
+promptbench -f prompts.txt -m gpt4o sonnet
+```
+
+### JSON output
+
+```bash
+promptbench "What is Python?" -o json
+```
+
+### CSV output
+
+```bash
+promptbench "What is Python?" -o csv > results.csv
+```
+
+### Save results to file
+
+```bash
+promptbench "Compare REST vs GraphQL" --save results.json
+```
+
+### Pipe from stdin
+
+```bash
+echo "What is the meaning of life?" | promptbench -m gpt4o sonnet
+```
+
+### List all supported models
+
+```bash
+promptbench --list-models
+```
+
+## Supported Models
+
+| Alias | Model | Provider |
+|-------|-------|----------|
+| `gpt4o` / `gpt4` | gpt-4o | OpenAI |
+| `gpt4mini` | gpt-4o-mini | OpenAI |
+| `gpt3.5` | gpt-3.5-turbo | OpenAI |
+| `sonnet` / `claude-sonnet` | claude-sonnet-4-20250514 | Anthropic |
+| `haiku` / `claude-haiku` | claude-haiku-4-5-20251001 | Anthropic |
+| `flash` / `gemini-flash` | gemini-2.0-flash | Google |
+| `gemini-pro` | gemini-1.5-pro | Google |
+
+You can also use full model names directly (e.g., `gpt-4-turbo`, `gemini-1.5-flash`).
+
+## Python Library Usage
+
+```python
+from promptbench.runner import run_bench
+from promptbench.display import display_comparison
+
+run = run_bench(
+    prompt="Explain recursion simply",
+    models=["gpt4o", "sonnet", "flash"],
+    temperature=0.5,
+)
+
+print(display_comparison(run))
+
+# Access individual results
+for result in run.results:
+    print(f"{result.model}: {result.latency_ms:.0f}ms, {result.cost_usd:.6f} USD")
+```
+
+## Configuration Flags
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `-m, --models` | Models to test | `gpt4o sonnet flash` |
+| `-s, --system` | System prompt | None |
+| `-t, --temperature` | Sampling temperature | `0.7` |
+| `--max-tokens` | Max output tokens | `1024` |
+| `-f, --file` | Prompts file path | None |
+| `-o, --output` | Output format: table, json, csv, markdown | `table` |
+| `--full` | Show full responses (no truncation) | Off |
+| `--no-parallel` | Run models sequentially | Off |
+| `--save` | Save results to JSON file | None |
+| `--list-models` | List supported models | — |
+| `--version` | Show version | — |
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, how to add providers, and PR guidelines.
+
+## License
+
+MIT — see [LICENSE](LICENSE) for details.

promptbench_cli-0.1.0/promptbench/__init__.py

@@ -0,0 +1,4 @@
+"""PromptBench - A/B test prompts across LLM providers from your terminal."""
+
+__version__ = "0.1.0"
+__author__ = "Shehriyar Ali Rustam"

promptbench_cli-0.1.0/promptbench/__main__.py

@@ -0,0 +1,6 @@
+"""Allow running promptbench as a module: python -m promptbench."""
+
+from promptbench.cli import main
+
+if __name__ == "__main__":
+    main()

promptbench_cli-0.1.0/promptbench/cli.py

@@ -0,0 +1,228 @@
+"""CLI entry point for PromptBench."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+
+from promptbench import __version__
+from promptbench.display import display_comparison, display_csv, display_json, display_markdown
+from promptbench.providers import (
+    MODEL_ALIASES,
+    PRICING,
+    detect_provider,
+    resolve_model,
+)
+from promptbench.runner import run_bench, run_bench_from_file
+
+
+def _check_api_keys(models: list[str]) -> list[str]:
+    """Check which API keys are missing for the requested models and return warnings."""
+    warnings: list[str] = []
+    providers_needed: set[str] = set()
+    for model in models:
+        resolved = resolve_model(model)
+        providers_needed.add(detect_provider(resolved))
+
+    key_map = {
+        "openai": "OPENAI_API_KEY",
+        "anthropic": "ANTHROPIC_API_KEY",
+        "google": "GOOGLE_API_KEY",
+    }
+
+    for provider in sorted(providers_needed):
+        env_var = key_map.get(provider)
+        if env_var and not os.environ.get(env_var):
+            warnings.append(f"⚠ {env_var} not set — {provider} models will fail")
+
+    return warnings
+
+
+def _list_models() -> str:
+    """Format the list of supported models and aliases."""
+    lines: list[str] = []
+    lines.append("\n⚡ PromptBench — Supported Models\n")
+
+    lines.append("Models with pricing info:")
+    lines.append(f"  {'Model':<35} {'Input $/1M':>12} {'Output $/1M':>12} {'Provider':<10}")
+    lines.append(f"  {'─' * 35} {'─' * 12} {'─' * 12} {'─' * 10}")
+    for model, (inp, out) in sorted(PRICING.items()):
+        provider = detect_provider(model)
+        lines.append(f"  {model:<35} ${inp:<11.3f} ${out:<11.3f} {provider:<10}")
+
+    lines.append("\nAliases:")
+    lines.append(f"  {'Alias':<20} → {'Model':<35}")
+    lines.append(f"  {'─' * 20}   {'─' * 35}")
+    for alias, model in sorted(MODEL_ALIASES.items()):
+        lines.append(f"  {alias:<20} → {model:<35}")
+
+    lines.append("")
+    return "\n".join(lines)
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    """Build and return the argument parser."""
+    parser = argparse.ArgumentParser(
+        prog="promptbench",
+        description="⚡ PromptBench — A/B test prompts across LLM providers from your terminal.",
+        epilog="""Examples:
+  promptbench "Explain quantum computing in one sentence"
+  promptbench "Write a haiku about coding" -m gpt4o sonnet flash
+  promptbench "Summarize this" -s "You are a helpful assistant" -m gpt4mini haiku
+  promptbench -f prompts.txt -m gpt4o sonnet -o json
+  promptbench "Hello world" -o csv --save results.json
+  echo "What is Python?" | promptbench -m gpt4o
+  promptbench --list-models
+""",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+
+    parser.add_argument(
+        "prompt",
+        nargs="?",
+        default=None,
+        help="The prompt to test across models",
+    )
+    parser.add_argument(
+        "-m", "--models",
+        nargs="+",
+        default=["gpt4o", "sonnet", "flash"],
+        help="Models to test (names or aliases). Default: gpt4o sonnet flash",
+    )
+    parser.add_argument(
+        "-s", "--system",
+        default=None,
+        help="System prompt to use for all models",
+    )
+    parser.add_argument(
+        "-t", "--temperature",
+        type=float,
+        default=0.7,
+        help="Sampling temperature (default: 0.7)",
+    )
+    parser.add_argument(
+        "--max-tokens",
+        type=int,
+        default=1024,
+        help="Maximum output tokens (default: 1024)",
+    )
+    parser.add_argument(
+        "-f", "--file",
+        default=None,
+        help="Path to a file with prompts (one per line or JSON array)",
+    )
+    parser.add_argument(
+        "-o", "--output",
+        choices=["table", "json", "csv", "markdown"],
+        default="table",
+        help="Output format (default: table)",
+    )
+    parser.add_argument(
+        "--full",
+        action="store_true",
+        help="Show full responses (no truncation)",
+    )
+    parser.add_argument(
+        "--no-parallel",
+        action="store_true",
+        help="Run models sequentially instead of in parallel",
+    )
+    parser.add_argument(
+        "--save",
+        default=None,
+        metavar="FILE",
+        help="Save results to a JSON file",
+    )
+    parser.add_argument(
+        "--list-models",
+        action="store_true",
+        help="List all supported models and aliases",
+    )
+    parser.add_argument(
+        "--version",
+        action="version",
+        version=f"promptbench {__version__}",
+    )
+
+    return parser
+
+
+def _format_run(run, output: str, full: bool) -> str:
+    """Format a BenchRun according to the chosen output format."""
+    if output == "json":
+        return display_json(run)
+    if output == "csv":
+        return display_csv(run)
+    if output == "markdown":
+        return display_markdown(run)
+    return display_comparison(run, show_full=full)
+
+
+def main(argv: list[str] | None = None) -> None:
+    """Main CLI entry point."""
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+
+    # List models
+    if args.list_models:
+        print(_list_models())
+        return
+
+    # Determine prompt source
+    prompt = args.prompt
+    file_path = args.file
+
+    # Check for piped stdin
+    if not prompt and not file_path and not sys.stdin.isatty():
+        prompt = sys.stdin.read().strip()
+
+    if not prompt and not file_path:
+        parser.print_help()
+        sys.exit(1)
+
+    # Check API keys
+    warnings = _check_api_keys(args.models)
+    for w in warnings:
+        print(w, file=sys.stderr)
+
+    parallel = not args.no_parallel
+
+    # Run from file
+    if file_path:
+        runs = run_bench_from_file(
+            filepath=file_path,
+            models=args.models,
+            system=args.system,
+            temperature=args.temperature,
+            max_tokens=args.max_tokens,
+            parallel=parallel,
+            verbose=True,
+        )
+        for run in runs:
+            print(_format_run(run, args.output, args.full))
+
+        if args.save:
+            data = [r.to_dict() for r in runs]
+            with open(args.save, "w", encoding="utf-8") as f:
+                json.dump(data, f, indent=2)
+            print(f"\n✓ Results saved to {args.save}")
+        return
+
+    # Run single prompt
+    run = run_bench(
+        prompt=prompt,
+        models=args.models,
+        system=args.system,
+        temperature=args.temperature,
+        max_tokens=args.max_tokens,
+        parallel=parallel,
+        verbose=True,
+    )
+    print(_format_run(run, args.output, args.full))
+
+    if args.save:
+        with open(args.save, "w", encoding="utf-8") as f:
+            json.dump(run.to_dict(), f, indent=2)
+        print(f"\n✓ Results saved to {args.save}")