llm-model-diff 0.1.0__tar.gz

llm_model_diff-0.1.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: llm-model-diff
+Version: 0.1.0
+Summary: Compare LLM model outputs side-by-side with rich diff visualization
+Author: model-diff contributors
+License: MIT
+Keywords: llm,ai,diff,comparison,openai,anthropic,cli
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: click>=8.0
+Requires-Dist: rich>=13.0
+Requires-Dist: anthropic>=0.20.0
+Requires-Dist: openai>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: isort>=5.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: flake8>=6.0; extra == "dev"
+
+# model-diff
+
+Compare LLM model outputs side-by-side with rich diff visualization.
+
+Run the same prompt on multiple models simultaneously and see exactly what each model says differently.
+
+## Installation
+
+```bash
+pip install model-diff
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/yourname/model-diff
+cd model-diff
+pip install -e .
+```
+
+## Requirements
+
+Set the API keys for the providers you want to use:
+
+```bash
+export OPENAI_API_KEY=sk-...
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Missing keys are handled gracefully — models without a key are skipped with a warning.
+
+## Usage
+
+```bash
+# Default: compare GPT-4o vs Claude Sonnet
+model-diff "What is the best way to handle errors in Python?"
+
+# Specify models explicitly
+model-diff "Explain recursion" --models gpt-4o,claude-sonnet-4-6
+
+# Use a prompt file
+model-diff --prompt prompt.txt --models gpt-4o,claude-haiku-4-5-20251001,claude-sonnet-4-6
+
+# Word-level diff
+model-diff "Explain recursion" --diff words
+
+# Show only differences (hide matching sections)
+model-diff "Explain recursion" --only-diff
+
+# Deterministic outputs
+model-diff "Explain recursion" --temperature 0.0
+
+# Save results to JSON
+model-diff "Explain recursion" --output results.json
+```
+
+## Supported Models
+
+| Model ID | Provider | API Key |
+|---|---|---|
+| `gpt-4o` | OpenAI | `OPENAI_API_KEY` |
+| `gpt-4o-mini` | OpenAI | `OPENAI_API_KEY` |
+| `claude-opus-4-6` | Anthropic | `ANTHROPIC_API_KEY` |
+| `claude-sonnet-4-6` | Anthropic | `ANTHROPIC_API_KEY` |
+| `claude-haiku-4-5-20251001` | Anthropic | `ANTHROPIC_API_KEY` |
+
+## Architecture
+
+```
+src/model_diff/
+├── cli.py      # Click-based CLI entry point
+├── models.py   # Provider-specific API callers, run concurrently via threading
+└── differ.py   # difflib-based diff engine + Rich output formatter
+```
+
+Model calls are issued concurrently using `threading`, so wall time equals the slowest model rather than the sum of all models.
+
+## License
+
+MIT

llm_model_diff-0.1.0/README.md

@@ -0,0 +1,80 @@
+# model-diff
+
+Compare LLM model outputs side-by-side with rich diff visualization.
+
+Run the same prompt on multiple models simultaneously and see exactly what each model says differently.
+
+## Installation
+
+```bash
+pip install model-diff
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/yourname/model-diff
+cd model-diff
+pip install -e .
+```
+
+## Requirements
+
+Set the API keys for the providers you want to use:
+
+```bash
+export OPENAI_API_KEY=sk-...
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Missing keys are handled gracefully — models without a key are skipped with a warning.
+
+## Usage
+
+```bash
+# Default: compare GPT-4o vs Claude Sonnet
+model-diff "What is the best way to handle errors in Python?"
+
+# Specify models explicitly
+model-diff "Explain recursion" --models gpt-4o,claude-sonnet-4-6
+
+# Use a prompt file
+model-diff --prompt prompt.txt --models gpt-4o,claude-haiku-4-5-20251001,claude-sonnet-4-6
+
+# Word-level diff
+model-diff "Explain recursion" --diff words
+
+# Show only differences (hide matching sections)
+model-diff "Explain recursion" --only-diff
+
+# Deterministic outputs
+model-diff "Explain recursion" --temperature 0.0
+
+# Save results to JSON
+model-diff "Explain recursion" --output results.json
+```
+
+## Supported Models
+
+| Model ID | Provider | API Key |
+|---|---|---|
+| `gpt-4o` | OpenAI | `OPENAI_API_KEY` |
+| `gpt-4o-mini` | OpenAI | `OPENAI_API_KEY` |
+| `claude-opus-4-6` | Anthropic | `ANTHROPIC_API_KEY` |
+| `claude-sonnet-4-6` | Anthropic | `ANTHROPIC_API_KEY` |
+| `claude-haiku-4-5-20251001` | Anthropic | `ANTHROPIC_API_KEY` |
+
+## Architecture
+
+```
+src/model_diff/
+├── cli.py      # Click-based CLI entry point
+├── models.py   # Provider-specific API callers, run concurrently via threading
+└── differ.py   # difflib-based diff engine + Rich output formatter
+```
+
+Model calls are issued concurrently using `threading`, so wall time equals the slowest model rather than the sum of all models.
+
+## License
+
+MIT

llm_model_diff-0.1.0/pyproject.toml

@@ -0,0 +1,73 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "llm-model-diff"
+version = "0.1.0"
+description = "Compare LLM model outputs side-by-side with rich diff visualization"
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "model-diff contributors" }]
+requires-python = ">=3.8"
+keywords = ["llm", "ai", "diff", "comparison", "openai", "anthropic", "cli"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: Utilities",
+]
+dependencies = [
+    "click>=8.0",
+    "rich>=13.0",
+    "anthropic>=0.20.0",
+    "openai>=1.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-cov>=4.0",
+    "black>=23.0",
+    "isort>=5.0",
+    "mypy>=1.0",
+    "flake8>=6.0",
+]
+
+[project.scripts]
+llm-model-diff = "model_diff.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-dir]
+"" = "src"
+
+[tool.black]
+line-length = 88
+target-version = ["py38"]
+
+[tool.isort]
+profile = "black"
+line_length = 88
+
+[tool.mypy]
+python_version = "3.8"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = true
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py", "*_test.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]

llm_model_diff-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

llm_model_diff-0.1.0/src/llm_model_diff.egg-info/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.4
+Name: llm-model-diff
+Version: 0.1.0
+Summary: Compare LLM model outputs side-by-side with rich diff visualization
+Author: model-diff contributors
+License: MIT
+Keywords: llm,ai,diff,comparison,openai,anthropic,cli
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: click>=8.0
+Requires-Dist: rich>=13.0
+Requires-Dist: anthropic>=0.20.0
+Requires-Dist: openai>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: isort>=5.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: flake8>=6.0; extra == "dev"
+
+# model-diff
+
+Compare LLM model outputs side-by-side with rich diff visualization.
+
+Run the same prompt on multiple models simultaneously and see exactly what each model says differently.
+
+## Installation
+
+```bash
+pip install model-diff
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/yourname/model-diff
+cd model-diff
+pip install -e .
+```
+
+## Requirements
+
+Set the API keys for the providers you want to use:
+
+```bash
+export OPENAI_API_KEY=sk-...
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Missing keys are handled gracefully — models without a key are skipped with a warning.
+
+## Usage
+
+```bash
+# Default: compare GPT-4o vs Claude Sonnet
+model-diff "What is the best way to handle errors in Python?"
+
+# Specify models explicitly
+model-diff "Explain recursion" --models gpt-4o,claude-sonnet-4-6
+
+# Use a prompt file
+model-diff --prompt prompt.txt --models gpt-4o,claude-haiku-4-5-20251001,claude-sonnet-4-6
+
+# Word-level diff
+model-diff "Explain recursion" --diff words
+
+# Show only differences (hide matching sections)
+model-diff "Explain recursion" --only-diff
+
+# Deterministic outputs
+model-diff "Explain recursion" --temperature 0.0
+
+# Save results to JSON
+model-diff "Explain recursion" --output results.json
+```
+
+## Supported Models
+
+| Model ID | Provider | API Key |
+|---|---|---|
+| `gpt-4o` | OpenAI | `OPENAI_API_KEY` |
+| `gpt-4o-mini` | OpenAI | `OPENAI_API_KEY` |
+| `claude-opus-4-6` | Anthropic | `ANTHROPIC_API_KEY` |
+| `claude-sonnet-4-6` | Anthropic | `ANTHROPIC_API_KEY` |
+| `claude-haiku-4-5-20251001` | Anthropic | `ANTHROPIC_API_KEY` |
+
+## Architecture
+
+```
+src/model_diff/
+├── cli.py      # Click-based CLI entry point
+├── models.py   # Provider-specific API callers, run concurrently via threading
+└── differ.py   # difflib-based diff engine + Rich output formatter
+```
+
+Model calls are issued concurrently using `threading`, so wall time equals the slowest model rather than the sum of all models.
+
+## License
+
+MIT

llm_model_diff-0.1.0/src/llm_model_diff.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+README.md
+pyproject.toml
+src/llm_model_diff.egg-info/PKG-INFO
+src/llm_model_diff.egg-info/SOURCES.txt
+src/llm_model_diff.egg-info/dependency_links.txt
+src/llm_model_diff.egg-info/entry_points.txt
+src/llm_model_diff.egg-info/requires.txt
+src/llm_model_diff.egg-info/top_level.txt
+src/model_diff/__init__.py
+src/model_diff/cli.py
+src/model_diff/differ.py
+src/model_diff/models.py
+tests/test_cli.py
+tests/test_differ.py
+tests/test_models.py

llm_model_diff-0.1.0/src/llm_model_diff.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

llm_model_diff-0.1.0/src/llm_model_diff.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+llm-model-diff = model_diff.cli:main

llm_model_diff-0.1.0/src/llm_model_diff.egg-info/requires.txt

@@ -0,0 +1,12 @@
+click>=8.0
+rich>=13.0
+anthropic>=0.20.0
+openai>=1.0.0
+
+[dev]
+pytest>=7.0
+pytest-cov>=4.0
+black>=23.0
+isort>=5.0
+mypy>=1.0
+flake8>=6.0

llm_model_diff-0.1.0/src/llm_model_diff.egg-info/top_level.txt

@@ -0,0 +1 @@
+model_diff

llm_model_diff-0.1.0/src/model_diff/__init__.py

@@ -0,0 +1,7 @@
+"""model-diff: Compare LLM model outputs side-by-side with rich diff visualization."""
+
+__version__ = "0.1.0"
+__all__ = ["ModelRunner", "DiffEngine", "ModelResult"]
+
+from model_diff.models import ModelResult, ModelRunner
+from model_diff.differ import DiffEngine

llm_model_diff-0.1.0/src/model_diff/cli.py

@@ -0,0 +1,154 @@
+"""CLI entry point for model-diff."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+from typing import List, Optional
+
+import click
+from rich.console import Console
+
+from model_diff.models import DEFAULT_MODELS, ALL_SUPPORTED_MODELS, ModelRunner
+from model_diff.differ import DiffEngine
+
+console = Console()
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def _parse_models(models_str: str) -> List[str]:
+    """Split a comma-separated model list and strip whitespace."""
+    return [m.strip() for m in models_str.split(",") if m.strip()]
+
+
+def _load_prompt(prompt_text: Optional[str], prompt_file: Optional[str]) -> str:
+    """Return the prompt string from either the positional arg or a file."""
+    if prompt_file:
+        p = Path(prompt_file)
+        if not p.exists():
+            console.print(f"[red]Error:[/red] prompt file '{prompt_file}' not found.")
+            sys.exit(1)
+        return p.read_text(encoding="utf-8").strip()
+    if prompt_text:
+        return prompt_text.strip()
+    console.print(
+        "[red]Error:[/red] You must supply a prompt as an argument or via --prompt."
+    )
+    console.print("Usage: model-diff \"Your prompt here\"")
+    console.print("       model-diff --prompt prompt.txt")
+    sys.exit(1)
+
+
+# ---------------------------------------------------------------------------
+# Click command
+# ---------------------------------------------------------------------------
+
+@click.command(context_settings={"help_option_names": ["-h", "--help"]})
+@click.argument("prompt_text", required=False, metavar="PROMPT")
+@click.option(
+    "--prompt",
+    "prompt_file",
+    default=None,
+    metavar="FILE",
+    help="Path to a text file containing the prompt (alternative to inline PROMPT).",
+)
+@click.option(
+    "--models",
+    "models_str",
+    default=",".join(DEFAULT_MODELS),
+    show_default=True,
+    metavar="MODEL1,MODEL2,...",
+    help=(
+        "Comma-separated list of model IDs to compare.  "
+        f"Supported: {', '.join(sorted(ALL_SUPPORTED_MODELS))}"
+    ),
+)
+@click.option(
+    "--diff",
+    "diff_mode",
+    default="lines",
+    type=click.Choice(["lines", "words", "chars"], case_sensitive=False),
+    show_default=True,
+    help="Granularity of the diff: lines, words, or chars.",
+)
+@click.option(
+    "--only-diff",
+    is_flag=True,
+    default=False,
+    help="Hide matching sections and show only the differing parts.",
+)
+@click.option(
+    "--output",
+    "output_file",
+    default=None,
+    metavar="FILE",
+    help="Save the full results as JSON to this file.",
+)
+@click.option(
+    "--temperature",
+    default=0.7,
+    show_default=True,
+    type=click.FloatRange(0.0, 2.0),
+    help="Sampling temperature passed to each model (0.0 = deterministic).",
+)
+@click.version_option(package_name="model-diff")
+def main(
+    prompt_text: Optional[str],
+    prompt_file: Optional[str],
+    models_str: str,
+    diff_mode: str,
+    only_diff: bool,
+    output_file: Optional[str],
+    temperature: float,
+) -> None:
+    """
+    Run the same prompt on multiple LLM models and show a side-by-side diff.
+
+    \b
+    Examples:
+      model-diff "What is the best way to handle errors in Python?"
+      model-diff "Explain recursion" --models gpt-4o,claude-sonnet-4-6
+      model-diff --prompt prompt.txt --models gpt-4o-mini,claude-haiku-4-5-20251001
+      model-diff "Explain recursion" --diff words --only-diff
+      model-diff "Explain recursion" --output results.json --temperature 0.0
+    """
+    prompt = _load_prompt(prompt_text, prompt_file)
+
+    requested_models = _parse_models(models_str)
+
+    # Validate model names
+    valid_models, warnings = ModelRunner.validate_models(requested_models)
+    for w in warnings:
+        console.print(f"[yellow]Warning:[/yellow] {w}")
+
+    if not valid_models:
+        console.print(
+            "[red]Error:[/red] No valid models to run.  "
+            f"Supported: {', '.join(sorted(ALL_SUPPORTED_MODELS))}"
+        )
+        sys.exit(1)
+
+    # ── Run the models ────────────────────────────────────────────────
+    console.print(
+        f"\n[dim]Running prompt on {len(valid_models)} model(s): "
+        f"{', '.join(valid_models)}  (temperature={temperature})[/dim]\n"
+    )
+
+    runner = ModelRunner(models=valid_models, temperature=temperature)
+    results = runner.run(prompt)
+
+    # ── Render ────────────────────────────────────────────────────────
+    engine = DiffEngine(diff_mode=diff_mode, only_diff=only_diff, console=console)
+    engine.render(
+        prompt=prompt,
+        results=results,
+        temperature=temperature,
+        output_file=output_file,
+    )
+
+    # Exit with non-zero if every model failed
+    if all(r.error for r in results):
+        sys.exit(2)