llm2graph 0.3.0__tar.gz

llm2graph-0.3.0/PKG-INFO

@@ -0,0 +1,270 @@
+Metadata-Version: 2.4
+Name: llm2graph
+Version: 0.3.0
+Summary: LLM2Graph: Dynamic Knowledge Graph Construction via LLM-only elicitation
+Author: Raj Sanjay Shah
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic>=2.7
+Requires-Dist: tqdm>=4.66
+Requires-Dist: networkx>=3.2
+Requires-Dist: tenacity>=8.2
+Requires-Dist: typer>=0.12
+Requires-Dist: openai>=1.37
+Provides-Extra: gemini
+Requires-Dist: google-generativeai>=0.7; extra == "gemini"
+Provides-Extra: hf-local
+Requires-Dist: transformers>=4.44; extra == "hf-local"
+Requires-Dist: accelerate>=0.33; extra == "hf-local"
+Requires-Dist: sentencepiece>=0.2; extra == "hf-local"
+Requires-Dist: einops>=0.7; extra == "hf-local"
+
+
+# LLM2Graph - Dynamic Knowledge Graph Construction & Evaluation
+
+This package implements the graph-based methodology from the COLM 2025 paper:
+
+> **The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning**
+
+It provides an **LLM-only** pipeline (no heuristic fallbacks) for:
+1. **Graph construction** via entity-centric elicitation and triple extraction.
+2. **Query generation** with **multi-hop**, **alias-perturbed**, **paraphrased** questions, and optional **distractors**.
+3. **Evaluation** of **pre** vs **post** (unlearned) models, including a **residual knowledge** analysis.
+
+If any step returns an unexpected format, the package **raises** `LLMError`.
+
+
+---
+
+## Quick Start (End-to-End)
+
+```bash
+# 0) Install (choose providers you need)
+pip install -e .
+# Optionals:
+pip install -e '.[gemini]'      # Gemini support
+pip install -e '.[hf-local]'    # HuggingFace local LLMs
+
+# 1) Build a graph from an entity
+export OPENAI_API_KEY=sk-...
+llm2graph entity --seed "Stephen King" --max-depth 2 \
+  --provider openai --model gpt-4o-mini-2024-07-18 --out graph.json
+
+# 2) Generate multi-hop queries with alias/paraphrase perturbations + distractors
+llm2graph gen-queries --graph graph.json --target "Stephen King" \
+  --hops 2 --num-paths 50 --aliases 3 --paraphrases 2 --distractors 2 \
+  --provider openai --model gpt-4o-mini-2024-07-18 --out queries.json
+
+# 3) Evaluate pre vs post models (optionally use a judge model for equivalence)
+llm2graph eval --queries queries.json \
+  --pre-provider openai  --pre-model gpt-4o-mini-2024-07-18 \
+  --post-provider openai --post-model gpt-4o-mini-2024-07-18 \
+  --judge-provider openai --judge-model gpt-4o-mini-2024-07-18 \
+  --out eval_report.json
+```
+
+The **evaluation report** includes accuracies by bucket (single/multi-hop, alias, paraphrase) and a **residual_rate** capturing when gold phrasing fails but a perturbation still succeeds.
+
+
+---
+
+## Installation & Providers
+
+### Base
+```bash
+pip install -e .
+```
+
+### OpenAI (default)
+```bash
+export OPENAI_API_KEY=sk-...      # required for provider=openai
+```
+
+### Gemini
+```bash
+pip install -e '.[gemini]'
+export GEMINI_API_KEY=...         # required for provider=gemini
+```
+
+### Local HuggingFace
+```bash
+pip install -e '.[hf-local]'
+# Ensure PyTorch is installed and you have a compatible GPU (recommended).
+# Example model:
+llm2graph entity --seed "Ada Lovelace" --provider hf-local \
+  --model mistralai/Mistral-7B-Instruct-v0.3 --max-depth 1 --out graph.json
+```
+
+All providers share the same strict prompting/validation; non-conforming outputs raise `LLMError`.
+
+
+---
+
+## 1) Graph Construction (Entity --> Graph)
+
+**Command**
+```bash
+llm2graph entity \
+  --seed "Stephen King" \
+  --max-depth 2 \
+  --provider openai \
+  --model gpt-4o-mini-2024-07-18 \
+  --out graph.json
+```
+
+**What happens**
+- **Elicitation**: LLM writes a compact factual paragraph about the node.
+- **Triple extraction**: LLM returns strictly formatted triples: `(subject ; relation ; object)`.
+- **Strict checks**: subject must equal the current node; malformed lines raise.
+- **Expansion (BFS)**: Adds objects as next-depth nodes.
+
+**Advanced (programmatic kwargs in `GraphBuilder`)**
+- `use_relevance: bool` - LLM-scored 0-10; below threshold filtered.
+- `relevance_threshold: float` - default 3.0.
+- `decay: float in [0.1, 1.0]` - limits breadth as depth grows.
+- `max_nodes_per_depth: Optional[int]` - hard cap per depth.
+- `alias_merge: bool` - LLM-judged canonicalization of new nodes (YES/NO).
+
+**Output format (`graph.json`)**
+```jsonc
+{
+  "seed": "Stephen King",
+  "nodes": ["Stephen King", "The Shining", "Maine", "..."],
+  "edges": [
+    {"subject": "Stephen King", "relation": "wrote", "object": "The Shining"},
+    {"subject": "Stephen King", "relation": "lives in", "object": "Maine"}
+  ]
+}
+```
+
+
+---
+
+## 2) Query Generation (Multi-hop, Aliases, Paraphrases, Distractors)
+
+**Command**
+```bash
+llm2graph gen-queries \
+  --graph graph.json \
+  --target "Stephen King" \
+  --hops 2 \
+  --num-paths 50 \
+  --aliases 3 \
+  --paraphrases 2 \
+  --distractors 2 \
+  --provider openai \
+  --model gpt-4o-mini-2024-07-18 \
+  --out queries.json
+```
+
+**What happens**
+- Samples `--hops`-length paths from the graph.
+- Synthesizes a **single** question per path; the final node is the gold answer.
+- Generates **paraphrases** and **alias-perturbed** variants.
+- Optionally generates **distractors**.
+
+**Output (`queries.json`)**
+```jsonc
+{
+  "meta": {"hops": 2, "num_paths": 50, "aliases": 3, "paraphrases": 2, "distractors": 2},
+  "queries": [{
+    "path": [{"s": "A", "r": "rel1", "o": "B"}, {"s": "B", "r": "rel2", "o": "C"}],
+    "q_gold": "Which work by the 'King of Horror' features ...?",
+    "q_variants": ["... paraphrase1", "... paraphrase2"],
+    "q_alias_variants": ["... alias-perturbed phrasing ..."],
+    "answer": "C",
+    "distractors": ["X","Y"]
+  }]
+}
+```
+
+**Difficulty control**
+- **Hop length** (`--hops`) raises reasoning depth.
+- **Distractors** increase choice difficulty.
+- **Aliases/Paraphrases** stress alias-robustness and surface-form robustness.
+
+
+---
+
+## 3) Evaluation (Pre vs Post, with Residual Knowledge)
+
+**Command**
+```bash
+llm2graph eval \
+  --queries queries.json \
+  --pre-provider openai  --pre-model gpt-4o-mini-2024-07-18 \
+  --post-provider openai --post-model gpt-4o-mini-2024-07-18 \
+  --judge-provider openai --judge-model gpt-4o-mini-2024-07-18 \
+  --out eval_report.json
+```
+
+**What happens**
+- Asks **pre** and **post** models the gold question.
+- Asks the **post** model every variant (paraphrase/alias).
+- If `judge` is provided, equivalence is decided by strict `"YES"/"NO"` judgments; otherwise exact string equality is used.
+
+**Residual Knowledge (paper-aligned)**
+- An item is marked **residual** if **gold** is incorrect **post**, **but** any alias/paraphrase variant is correct.
+- Summarized via `residual_rate` and `residual_count`.
+
+**Output (`eval_report.json`)**
+```jsonc
+{
+  "summary": {
+    "all":         {"total": N, "correct": k, "accuracy": 0.xx},
+    "single_hop":  {"total": ..., ...},
+    "multi_hop":   {"total": ..., ...},
+    "alias":       {"total": ..., ...},
+    "paraphrase":  {"total": ..., ...},
+    "residual_rate": 0.xx,
+    "residual_count": M,
+    "num_items": N_items
+  },
+  "items": [
+    {
+      "path": [...],
+      "predictions": [
+        {"variant": "gold", "type": "gold", "pre": "…", "post": "…", "correct": true/false},
+        {"variant": "paraphrase", "type": "paraphrase", "pre": null, "post": "…", "correct": ...},
+        {"variant": "alias", "type": "alias", "pre": null, "post": "…", "correct": ...}
+      ],
+      "residual_flags": {
+        "residual": true/false,
+        "gold_correct": false,
+        "alias_any": true/false,
+        "para_any": true/false
+      }
+    }
+  ]
+}
+```
+
+
+---
+
+## Implementation Notes
+
+- **Strict parsing**: Triple lines must be exactly `(subject ; relation ; object)`; subject must equal the current node.
+- **Alias canonicalization**: Node merging uses `canonical_same(a,b)` --> strict `"YES"/"NO"` from an LLM.
+- **Relevance scoring**: 0-10 numeric, LLM-only; thresholded filtering (optional).
+- **HF local chat templates**: If available, we use `.apply_chat_template`; else a minimal structured prompt is used.
+- **No heuristic fallbacks**: Any format drift raises `LLMError`.
+
+
+---
+
+## Troubleshooting
+
+- **LLMError**: The model did not follow the strict format. Retry with a different model or lower temperature.
+- **Model access**: Ensure `OPENAI_API_KEY`/`GEMINI_API_KEY` is set; confirm the `--model` exists for that provider.
+- **HF OOM**: Choose a smaller HF repo; reduce generation tokens; consider 4/8-bit loading (extend loader as needed).
+
+
+---
+
+## Citation
+
+If you use this package, please cite:
+
+Shah, Raj Sanjay, Jing Huang, Keerthiram Murugesan, Nathalie Baracaldo, and Diyi Yang. *The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning.* Second Conference on Language Modeling. 2025.

llm2graph-0.3.0/README.md

@@ -0,0 +1,248 @@
+
+# LLM2Graph - Dynamic Knowledge Graph Construction & Evaluation
+
+This package implements the graph-based methodology from the COLM 2025 paper:
+
+> **The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning**
+
+It provides an **LLM-only** pipeline (no heuristic fallbacks) for:
+1. **Graph construction** via entity-centric elicitation and triple extraction.
+2. **Query generation** with **multi-hop**, **alias-perturbed**, **paraphrased** questions, and optional **distractors**.
+3. **Evaluation** of **pre** vs **post** (unlearned) models, including a **residual knowledge** analysis.
+
+If any step returns an unexpected format, the package **raises** `LLMError`.
+
+
+---
+
+## Quick Start (End-to-End)
+
+```bash
+# 0) Install (choose providers you need)
+pip install -e .
+# Optionals:
+pip install -e '.[gemini]'      # Gemini support
+pip install -e '.[hf-local]'    # HuggingFace local LLMs
+
+# 1) Build a graph from an entity
+export OPENAI_API_KEY=sk-...
+llm2graph entity --seed "Stephen King" --max-depth 2 \
+  --provider openai --model gpt-4o-mini-2024-07-18 --out graph.json
+
+# 2) Generate multi-hop queries with alias/paraphrase perturbations + distractors
+llm2graph gen-queries --graph graph.json --target "Stephen King" \
+  --hops 2 --num-paths 50 --aliases 3 --paraphrases 2 --distractors 2 \
+  --provider openai --model gpt-4o-mini-2024-07-18 --out queries.json
+
+# 3) Evaluate pre vs post models (optionally use a judge model for equivalence)
+llm2graph eval --queries queries.json \
+  --pre-provider openai  --pre-model gpt-4o-mini-2024-07-18 \
+  --post-provider openai --post-model gpt-4o-mini-2024-07-18 \
+  --judge-provider openai --judge-model gpt-4o-mini-2024-07-18 \
+  --out eval_report.json
+```
+
+The **evaluation report** includes accuracies by bucket (single/multi-hop, alias, paraphrase) and a **residual_rate** capturing when gold phrasing fails but a perturbation still succeeds.
+
+
+---
+
+## Installation & Providers
+
+### Base
+```bash
+pip install -e .
+```
+
+### OpenAI (default)
+```bash
+export OPENAI_API_KEY=sk-...      # required for provider=openai
+```
+
+### Gemini
+```bash
+pip install -e '.[gemini]'
+export GEMINI_API_KEY=...         # required for provider=gemini
+```
+
+### Local HuggingFace
+```bash
+pip install -e '.[hf-local]'
+# Ensure PyTorch is installed and you have a compatible GPU (recommended).
+# Example model:
+llm2graph entity --seed "Ada Lovelace" --provider hf-local \
+  --model mistralai/Mistral-7B-Instruct-v0.3 --max-depth 1 --out graph.json
+```
+
+All providers share the same strict prompting/validation; non-conforming outputs raise `LLMError`.
+
+
+---
+
+## 1) Graph Construction (Entity --> Graph)
+
+**Command**
+```bash
+llm2graph entity \
+  --seed "Stephen King" \
+  --max-depth 2 \
+  --provider openai \
+  --model gpt-4o-mini-2024-07-18 \
+  --out graph.json
+```
+
+**What happens**
+- **Elicitation**: LLM writes a compact factual paragraph about the node.
+- **Triple extraction**: LLM returns strictly formatted triples: `(subject ; relation ; object)`.
+- **Strict checks**: subject must equal the current node; malformed lines raise.
+- **Expansion (BFS)**: Adds objects as next-depth nodes.
+
+**Advanced (programmatic kwargs in `GraphBuilder`)**
+- `use_relevance: bool` - LLM-scored 0-10; below threshold filtered.
+- `relevance_threshold: float` - default 3.0.
+- `decay: float in [0.1, 1.0]` - limits breadth as depth grows.
+- `max_nodes_per_depth: Optional[int]` - hard cap per depth.
+- `alias_merge: bool` - LLM-judged canonicalization of new nodes (YES/NO).
+
+**Output format (`graph.json`)**
+```jsonc
+{
+  "seed": "Stephen King",
+  "nodes": ["Stephen King", "The Shining", "Maine", "..."],
+  "edges": [
+    {"subject": "Stephen King", "relation": "wrote", "object": "The Shining"},
+    {"subject": "Stephen King", "relation": "lives in", "object": "Maine"}
+  ]
+}
+```
+
+
+---
+
+## 2) Query Generation (Multi-hop, Aliases, Paraphrases, Distractors)
+
+**Command**
+```bash
+llm2graph gen-queries \
+  --graph graph.json \
+  --target "Stephen King" \
+  --hops 2 \
+  --num-paths 50 \
+  --aliases 3 \
+  --paraphrases 2 \
+  --distractors 2 \
+  --provider openai \
+  --model gpt-4o-mini-2024-07-18 \
+  --out queries.json
+```
+
+**What happens**
+- Samples `--hops`-length paths from the graph.
+- Synthesizes a **single** question per path; the final node is the gold answer.
+- Generates **paraphrases** and **alias-perturbed** variants.
+- Optionally generates **distractors**.
+
+**Output (`queries.json`)**
+```jsonc
+{
+  "meta": {"hops": 2, "num_paths": 50, "aliases": 3, "paraphrases": 2, "distractors": 2},
+  "queries": [{
+    "path": [{"s": "A", "r": "rel1", "o": "B"}, {"s": "B", "r": "rel2", "o": "C"}],
+    "q_gold": "Which work by the 'King of Horror' features ...?",
+    "q_variants": ["... paraphrase1", "... paraphrase2"],
+    "q_alias_variants": ["... alias-perturbed phrasing ..."],
+    "answer": "C",
+    "distractors": ["X","Y"]
+  }]
+}
+```
+
+**Difficulty control**
+- **Hop length** (`--hops`) raises reasoning depth.
+- **Distractors** increase choice difficulty.
+- **Aliases/Paraphrases** stress alias-robustness and surface-form robustness.
+
+
+---
+
+## 3) Evaluation (Pre vs Post, with Residual Knowledge)
+
+**Command**
+```bash
+llm2graph eval \
+  --queries queries.json \
+  --pre-provider openai  --pre-model gpt-4o-mini-2024-07-18 \
+  --post-provider openai --post-model gpt-4o-mini-2024-07-18 \
+  --judge-provider openai --judge-model gpt-4o-mini-2024-07-18 \
+  --out eval_report.json
+```
+
+**What happens**
+- Asks **pre** and **post** models the gold question.
+- Asks the **post** model every variant (paraphrase/alias).
+- If `judge` is provided, equivalence is decided by strict `"YES"/"NO"` judgments; otherwise exact string equality is used.
+
+**Residual Knowledge (paper-aligned)**
+- An item is marked **residual** if **gold** is incorrect **post**, **but** any alias/paraphrase variant is correct.
+- Summarized via `residual_rate` and `residual_count`.
+
+**Output (`eval_report.json`)**
+```jsonc
+{
+  "summary": {
+    "all":         {"total": N, "correct": k, "accuracy": 0.xx},
+    "single_hop":  {"total": ..., ...},
+    "multi_hop":   {"total": ..., ...},
+    "alias":       {"total": ..., ...},
+    "paraphrase":  {"total": ..., ...},
+    "residual_rate": 0.xx,
+    "residual_count": M,
+    "num_items": N_items
+  },
+  "items": [
+    {
+      "path": [...],
+      "predictions": [
+        {"variant": "gold", "type": "gold", "pre": "…", "post": "…", "correct": true/false},
+        {"variant": "paraphrase", "type": "paraphrase", "pre": null, "post": "…", "correct": ...},
+        {"variant": "alias", "type": "alias", "pre": null, "post": "…", "correct": ...}
+      ],
+      "residual_flags": {
+        "residual": true/false,
+        "gold_correct": false,
+        "alias_any": true/false,
+        "para_any": true/false
+      }
+    }
+  ]
+}
+```
+
+
+---
+
+## Implementation Notes
+
+- **Strict parsing**: Triple lines must be exactly `(subject ; relation ; object)`; subject must equal the current node.
+- **Alias canonicalization**: Node merging uses `canonical_same(a,b)` --> strict `"YES"/"NO"` from an LLM.
+- **Relevance scoring**: 0-10 numeric, LLM-only; thresholded filtering (optional).
+- **HF local chat templates**: If available, we use `.apply_chat_template`; else a minimal structured prompt is used.
+- **No heuristic fallbacks**: Any format drift raises `LLMError`.
+
+
+---
+
+## Troubleshooting
+
+- **LLMError**: The model did not follow the strict format. Retry with a different model or lower temperature.
+- **Model access**: Ensure `OPENAI_API_KEY`/`GEMINI_API_KEY` is set; confirm the `--model` exists for that provider.
+- **HF OOM**: Choose a smaller HF repo; reduce generation tokens; consider 4/8-bit loading (extend loader as needed).
+
+
+---
+
+## Citation
+
+If you use this package, please cite:
+
+Shah, Raj Sanjay, Jing Huang, Keerthiram Murugesan, Nathalie Baracaldo, and Diyi Yang. *The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning.* Second Conference on Language Modeling. 2025.

llm2graph-0.3.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "llm2graph"
+version = "0.3.0"
+description = "LLM2Graph: Dynamic Knowledge Graph Construction via LLM-only elicitation"
+readme = "README.md"
+requires-python = ">=3.9"
+authors = [{name = "Raj Sanjay Shah"}]
+license = {text = "MIT"}
+dependencies = [
+  "pydantic>=2.7",
+  "tqdm>=4.66",
+  "networkx>=3.2",
+  "tenacity>=8.2",
+  "typer>=0.12",
+  "openai>=1.37",
+]
+
+[project.optional-dependencies]
+gemini = ["google-generativeai>=0.7"]
+hf-local = ["transformers>=4.44", "accelerate>=0.33", "sentencepiece>=0.2", "einops>=0.7"]
+
+[project.scripts]
+llm2graph = "llm2graph.cli:app"
+
+[tool.setuptools.packages.find]
+where = ["src"]

llm2graph-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

llm2graph-0.3.0/src/llm2graph/__init__.py

@@ -0,0 +1,5 @@
+__all__ = ["GraphBuilder", "LLMClient", "Config", "KG", "Triple"]
+
+from .graph_builder import GraphBuilder, KG, Triple
+from .llm_client import LLMClient
+from .settings import Config

llm2graph-0.3.0/src/llm2graph/alias.py

@@ -0,0 +1,22 @@
+from .llm_client import LLMClient, LLMError
+
+class AliasTools:
+    def __init__(self, llm: LLMClient):
+        self.llm = llm
+
+    def canonical_same(self, a: str, b: str) -> bool:
+        system = "Answer strictly 'YES' or 'NO'."
+        user = f'Are these two names the **same canonical entity**?\nA = "{a}"\nB = "{b}"\nAnswer:'
+        out = self.llm.chat(system, user).strip().upper()
+        if out not in {"YES", "NO"}:
+            raise LLMError(f"Alias check returned invalid token: {out!r}")
+        return out == "YES"
+
+    def generate_aliases(self, name: str, k: int) -> list[str]:
+        system = "Output exactly k alternative surface forms (aliases) for the given entity; one per line; no numbering."
+        user = f'Entity: "{name}"\nk = {k}\nConstraints: common nicknames/epithets/abbreviations; no hallucinated entities.'
+        text = self.llm.chat(system, user)
+        aliases = [line.strip() for line in text.splitlines() if line.strip()]
+        if len(aliases) < 1:
+            raise LLMError("No aliases produced by LLM")
+        return aliases[:k]

llm2graph-0.3.0/src/llm2graph/cli.py

@@ -0,0 +1,70 @@
+import json
+import typer
+from .settings import Config
+from .graph_builder import GraphBuilder
+
+app = typer.Typer(help="LLM2Graph: LLM-only knowledge graph construction")
+
+@app.command("entity")
+def entity_to_graph(seed: str = typer.Option(..., "--seed", help="Seed entity name"),
+                    max_depth: int = typer.Option(2, "--max-depth", help="Max BFS depth (0..5)"),
+                    model: str = typer.Option("gpt-4o-mini-2024-07-18", "--model", help="Model id or HF repo id"),
+                    provider: str = typer.Option("openai", "--provider", help="Provider (openai | gemini | hf-local)"),
+                    temperature: float = typer.Option(0.2, "--temperature"),
+                    timeout_s: int = typer.Option(60, "--timeout-s"),
+                    out: str = typer.Option("graph.json", "--out", help="Output JSON file path")):
+    cfg = Config(model=model, max_depth=max_depth, temperature=temperature, timeout_s=timeout_s, provider=provider)
+    builder = GraphBuilder(cfg)
+    kg = builder.build(seed)
+    with open(out, "w", encoding="utf-8") as f:
+        json.dump(kg.to_json(), f, ensure_ascii=False, indent=2)
+    typer.echo(f"Wrote {out}")
+
+if __name__ == "__main__":
+    app()
+
+from .query_generator import QueryGenerator
+from .evaluation import Evaluator
+from .llm_client import LLMClient
+
+@app.command("gen-queries")
+def gen_queries(graph: str = typer.Option(..., "--graph", help="Graph JSON from entity builder"),
+                target: str = typer.Option(..., "--target", help="Seed/target entity to start paths"),
+                hops: int = typer.Option(1, "--hops"),
+                num_paths: int = typer.Option(50, "--num-paths"),
+                aliases: int = typer.Option(0, "--aliases"),
+                paraphrases: int = typer.Option(0, "--paraphrases"),
+                distractors: int = typer.Option(0, "--distractors"),
+                provider: str = typer.Option("openai", "--provider"),
+                model: str = typer.Option("gpt-4o-mini-2024-07-18", "--model"),
+                out: str = typer.Option("queries.json", "--out")):
+    import json as _json
+    with open(graph, "r", encoding="utf-8") as f:
+        gj = _json.load(f)
+    llm = LLMClient(provider=provider, model=model)
+    gen = QueryGenerator(llm)
+    q = gen.generate(gj, target=target, hops=hops, num_paths=num_paths, aliases=aliases, paraphrases=paraphrases, distractors=distractors)
+    with open(out, "w", encoding="utf-8") as f:
+        _json.dump(q, f, ensure_ascii=False, indent=2)
+    typer.echo(f"Wrote {out}")
+
+@app.command("eval")
+def eval_models(queries: str = typer.Option(..., "--queries"),
+                pre_provider: str = typer.Option("openai", "--pre-provider"),
+                pre_model: str = typer.Option("gpt-4o-mini-2024-07-18", "--pre-model"),
+                post_provider: str = typer.Option("openai", "--post-provider"),
+                post_model: str = typer.Option("gpt-4o-mini-2024-07-18", "--post-model"),
+                judge_provider: str = typer.Option(None, "--judge-provider"),
+                judge_model: str = typer.Option(None, "--judge-model"),
+                out: str = typer.Option("eval_report.json", "--out")):
+    import json as _json
+    with open(queries, "r", encoding="utf-8") as f:
+        qj = _json.load(f)
+    pre = LLMClient(provider=pre_provider, model=pre_model)
+    post = LLMClient(provider=post_provider, model=post_model)
+    judge = LLMClient(provider=judge_provider, model=judge_model) if judge_provider and judge_model else None
+    ev = Evaluator(pre, post, judge)
+    rep = ev.run(qj)
+    with open(out, "w", encoding="utf-8") as f:
+        _json.dump(rep, f, ensure_ascii=False, indent=2)
+    typer.echo(f"Wrote {out}")